United Kingdom's HM Land Registry Price Paid Data

HM Land Registry began a phased release of its data on property transactions—the Price Paid Dataset—in March 2012, and by November 2013 the entire historic record dating back to 1995 was released. The data provides much-needed transparency in a historically “murky” business, and is already being used extensively by some traditional players in the property market. Additionally, new players are consolidating around the field of proptech, developing digital tools to bring buying and selling property “out of the Stone Age”. Proptech startups attracted an estimated $1.4 billion in investment globally in 2014. PI Labs, an incubator for proptech startups, opened in London in late 2014.

• HM Land Registry is a trading fund, a type of government agency that meets its outgoing costs from the money it charges for services. Along with the UK’s other trading funds, its future has been a sticking point in the UK’s journey towards open data.
• In contrast to some of the other datasets studied in this report (e.g., TfL), HM Land Registry’s data is delivering value most where it is cross-referenced with other, often proprietary, datasets.
• Releasing Price Paid data as open data has had a positive effect on data quality. HM Land Registry report an increase in the number of notifications about inaccurate data by members of the public since the Price Paid data was released.
• HM Land Registry is a potential candidate for privatisation. The government set a poor precedent when it privatised the Postcode Address File alongside Royal Mail. Selling off HM Land Registry data would likely lead to a slowdown in the positive impacts described in this case: market transparency and proptech innovation.

HM Land Registry (HMLR) was created in 1862 to register the ownership of land and property in England and Wales. It is responsible for maintaining the Land Register, where more than 24 million titles (evidence of land and property ownership) are documented.

In 1993 it was established as a trading fund—a type of government agency that has the authority (under the Government Trading funds Act 1973) to meet its outgoing costs from the money it charges for services. Other trading funds include Companies House, The Met Office and Ordnance Survey (OS). HMLR’s receipts come mainly from fees it charges individuals and organisations to register land, and to register changes in title to land and properties.

The Land Register is the world’s largest property database1. At the height of the UK property boom in 2007, it processed around £1 million worth of property every minute in England and Wales2. However, the credit crunch saw a slowdown in the UK market. HMLR lost £220m over three years to January 2011 and was forced to increase fees to users by 30%3. Those fees have since decreased, as part of HM Land Registry’s strategy to encourage more people to register transactions online.

The trading fund model has played a controversial role in the UK’s journey to open data, since at least in the cases where agencies are funded by receipts from reselling data (for example OS) the two approaches are in conflict. Indeed, a previous study by the author of the UK’s journey to open data, records “strong resistance” from trading funds and in particular OS, to a future of open government data:

The Power of Information Review had recommended a detailed cost-benefit analysis of the Trading fund model, which was commissioned jointly by HM Treasury and the then Department for Business, Enterprise and Regulatory Reform, and published in February 2008. Despite that study finding strongly in favour of abandoning the Trading fund model, and despite the Power of Information Taskforce recommending reform on the back of its findings, positions against re-examining the Trading fund model within government remained entrenched.4

In November 2010, the government announced plans to create a Public Data Corporation (PDC) to coordinate the release of data from the trading funds that deal in data and information (OS, The Met Office, Companies House, and HMLR). The plans represented “a wholesale reorganisation of the largest trading funds”, but did not go into detail on how much of the data released by the PDC would be open, and also included the possibility of the PDC being part-privatised5.

In August 2011, the Department for Business, Innovation and Skills (BIS) consulted with the public on the PDC’s data policy6. In November 2011, Chancellor George Osborne made commitments on open data in his Autumn Statement that included data releases from Companies House, HMLR, and The Met Office, a limited data release from OS, and the establishment of a Public Data Group to share best practice that would supersede the PDC7. In March 2012 BIS published a response that included terms of reference for a Data Strategy Board that would “seek to maximise the value of data from the Public Data Group … for long-term economic and social benefit, including through the release of data free of charge”8.

In July 2014 the UK’s coalition government abandoned plans to privatise or part-privatise HMLR, after Liberal Democrat business secretary Vince Cable blocked the move9. However, after the May 2015 elections returned a majority Conservative government privatisation may be back on the table: Chancellor George Osborne has outlined plans to privatise £23bn worth of government assets as an attempt to bolster public finances10, and campaigners against privatisation identify HMLR as one of the targets11.

In April 2015, the functions of the Public Data Group were integrated into a new BIS-run board, the Digital Culture, Services Platforms, and Data Board, and the PDG met for the last time12.

HMLR’s Price Paid Dataset contains over 20 million transactions going back to 1995 for residential properties sold for full market value. It was this dataset that Chancellor George Osborne committed the Land Registry to releasing as open data in his 2011 Autumn Statement. In March 2012, HMLR began releasing monthly files of transactions. 2013 saw the release of historic data in two phases: Records of transactions between 2009 and 2012 were released in June 2013; and in November 2013 the full historical record from January 1995 to the current month was released. That year also saw the release of the price paid database as linked data. In 2014, HMLR worked with a third-party supplier to create an “easy to use” front end, the Price Paid Data Report Builder tool13.

The data is not a full picture of land ownership in England and Wales—it only includes details of properties bought and sold between January 1995 and the present. It is released under an Open Government Licence and as such it conforms with the open definition. It is updated every month with the new transactions that have been recorded, and corrections to older entries.

The quality—and timeliness—of the data relies in part on those people (homeowners and their solicitors) who register it with HMLR. A sample monthly update file showed approximately 2,500 entries against transactions older than a year, which might be assumed to be corrections to existing transactions. Of the (roughly) remaining 79,000 transactions assumed to be new transactions, 3,345 (4.2%) were more than six months old.

Before its release as open data, Price Paid data had been made available in bulk to paying customers as part of HMLR’s commercial activities. As such, HMLR had sought the view of the Information Commissioner’s Office (ICO) in 2004 as to whether Price Paid data (which includes addresses) constituted biographical information, and the ICO had indicated that it did not. Before its release in 2012, HMLR carried out a Privacy Impact Assessment14, and reviewed the assessment again in 201315. Both reviews concluded that Price Paid data was deemed not to be biographical or personal in nature. The pre-release review flagged an increase in companies using the data for direct marketing purposes as a potential issue; the 2013 review concluded that there had been no increase in direct marketing as a result of the release.

Who was calling for HMLR Price Paid data to be released as open data? The fact that HMLR is a trading fund means it was a general target for open data enthusiasts. In responses to the government’s PDC consultation, HMLR data was rated as the third most requested dataset to be “free for use and reuse” after Ordnance Survey data and Address/Postcode information22. Lynne Nicholson, Head of Data Products and Services at HMLR, says the data release responded to “a general call for government to release more data23”, and did not mention any particular organisations who were calling for the data to be released, beyond the Open Data Institute, which in any case was only created at the same time as the announcement that the Price Paid Dataset was to be made available as open data.

Data based on HM Land Registry data, including Property Sales24 and Median House Prices25 had been published intermittently on data.gov.uk by the Department for Communities and Local Government since 2010, and had fed into a small number of apps, notably the parking app JustPark. Before the open data release, HMLR Price Paid data had been available as downloadable .txt files (and also as bespoke reports) to a small group of organisations on a commercial basis. A business case for releasing the data, drafted before the Chancellor’s autumn Statement and released under FOI in February 2012, reveals that HMLR were attracting a modest £600,000 of income from this activity per year. It notes that:

Existing customers for this data are, in the main, companies that publish the data on their own websites. Land Registry has 30 existing subscribers who are website based businesses and 7 others which they class as consultant/research users. In a typical month they also receive ad hoc requests for specific areas and time periods of data from a small group of customers, most of which are Local Authorities—on average 5 requests a month. They also receive ad hoc requests from Estate agents and solicitors, usually 1-2 a month from each sector26.

During interview, HMLR did not wish to reveal the identities of its paying customers prior to the data release27. A source in the industry speculated such customers might include property portals (such as Rightmove, Prime Location, and MousePrice), business-to-business publications (such as Hometrack) and nationwide property consultancies (such as Savilles). Attempts by the author to interview one such potential customer (Rightmove) were unsuccessful.

The same business case document asserts that, across all the trading funds considered, it would be existing customers of the data that would capture a significant proportion of the initial benefit of releasing the data for free. This notwithstanding, when asked to identify stakeholders who were against the release of the Price Paid Dataset as open data, Lynne Nicholson, Head of Data Products and Services at HMLR, pointed to concerns from existing commercial customers “that opening up the data would allow more competition into the market”28.

Indeed, the response to a consultation on government open data strategy put forward by the Cabinet Office in 2011, includes one from Landmark Information Group (who own Mouseprice.com). It argues strongly in favour of a presumption for open data, but notes that:

Due consideration should be made of the private sector organisations who may already offer services around any data sets which currently attract a charge from the public sector data providers which may be candidates for being re-designated as ‘open’. Thus avoiding any potential negative affects to the existing markets.29

According to the second privacy impact assessment, between January 2012 and March 2013, the Price Paid data was downloaded a total of nearly 78,000 times31.

One data re-user I spoke to described how portals listing Price Paid data have proliferated on the web:

Now you can download a .csv and have your own [HM Land Registry] property price portal up in a couple of hours …. If you Google your own house address you will see there are dozens of sites showing sold prices for your house.

Nicholson also notes that government departments were attracted by the open data offer, then entered into confidential data-sharing agreements involving richer HMLR datasets as a result. Nicholson mentions “significant use by the media and researchers,” making an example of the data and what could be achieved with it. The cartogram in Figure 3 (which appears uncredited on the London Datastore and was picked up by bloggers at UCL’s Centre for Advanced Spatial Analysis32), shows the median house prices in different London boroughs.

Figure 3: Cartogram of median house prices in London. Released uncredited on the London Datastore under OGL.

HMLR hosted an event in July 2014 for users of its data33. At the event, Manuel Timita from Illustreets—a platform that takes open data from a variety of sources (including the Office for National Statistics, OS, Police.uk, Environmental Agency, Department for Transport, as well as HMLR) and displays it on a map to help people decide where they want to live—showed a timeline of London house prices, matched against key events surrounding the credit crunch (see Figure 4)34.

Figure 4: Screenshot of Illustreets’ video timeline of rise in London house prices. Copyright Manuel Timita. Reproduced with permission.

Also at the event was Johnny Morris from estate agents Hamptons International, who, according to the Land Registry’s report of the event, “divulged how [Hamptons International] are enriching their market view of the ‘downsizer’, through mixing their own proprietary data with Land Registry Price Paid Data and Ordnance Survey open data”35. However, it is not clear whether Hamptons were an original customer of the HMLR Price Paid data, or only began using the data once it was released openly.

One estate agent who is using the data now it’s been released by HMLR is Adrian Black, founder and director of YOUhome. He moved to estate agency after 10 years working in technology for Goldman Sachs, drawn to a business that looked ripe for technological change:

In every agency business you have diminishing returns and you have to become more productive and efficient and also pick up revenue from other sources. That’s not happening in estate agency. Fee rates are coming down, but they’re coming down slowly and the increase in property prices means agents are actually earning more. [With online portals] reaching buyers is getting easier, and yet revenues per sale are increasing.36

YOUhome is using Land Registry data and other proprietary sources to make their estate agency business more efficient, and more attractive to potential clients.

Sellers typically look to engage agents who offer the highest valuation and the lowest fees. But that leads to low conversion rates.37

Black says seeing actual data can persuade sellers to go for a more realistic asking price, can get buyers comfortable more quickly, and can achieve a quicker sale at a fair price. And it’s working for their business. They report that typical central London agents charge 2-3% rates, whereas their central London rate is 0.8%. And in Bournemouth their brokers transact on average 50 properties per year per agent, against an industry average of around 25 (15 in London). “We’re the Land Registry’s number one fan,” says Black:

This is what the government should be doing to make efficient businesses. They should open up data.38

Property expert Henry Pryor anticipates that more and more groups will find ways to make use of HMLR’s data:

Retailers are already making use of the value of property as part of the matrix that they look at when trying to segment their market. Clearly, they’re trying to target different individuals with different products. To know that somebody lives in a million pound house as opposed to a hundred thousand pound house, you can make assumptions based on that information.39

The release of HMLR’s Price Paid Data has coincided with the rise of the new field of proptech: businesses using data and technology to innovate in the property sector. One report quotes figures from CrunchBase showing investors worldwide put a record $1.4bn into proptech startups in 2014, although most of the money went to US-based firms40. In late 2014 Pi Labs, the UK’s first proptech incubator, was set up by estate agents Cushman & Wakefield and venture capitalists Spire Ventures.

“It’s kind of buzzy at the moment and so a lot of different things fall under [proptech],” explains one HMLR data re-user who does not wish to be identified. A video filmed to promote a roundtable on the future of the property market in May 201541 showcases companies including Splittable (which helps tenants of shared houses split their bills), Hubble (a web platform for making renting office space easier), Homeshift (a platform through which agents, tenants, and housebuyers can communicate during a house move), Fixflo (a tool for tenants and property managers), Land Technologies (a data aggregator that identifies land that’s suitable for building on), MoveBubble (an online agent that acts on behalf of tenants), and We Are Pop Up (an agency for short-term retail lets).

Peter Thum-Bonanno sold his first proptech startup, the property portal Find Properly, to search engine Nestoria in November 2014. Although he is unwilling to ascribe the rise of proptech entirely to the release of the HM Land Registry dataset, he says there is a link:

I don’t know if it’s grown or if I’ve just become more aware of it, but there’s definitely a rise in property technology companies who call themselves proptech companies. I think a lot of it is because property seems to still be stuck in the Stone Age. The way things are done still seems to be very antiquated. People think technology could help make that more efficient. And I think, compared to a lot of other industries, there’s a huge amount of data out there and the Land Registry is mainly to thank for that.42

Thum-Bonanno’s new venture is a tool to assess the efficiency of agents by using data including asking prices, sold prices, and conveyancing times. GetAgent uses a mix of public and private data sources, but HMLR data “plays a big part in that”.43 As well as using HMLR Price Paid data, it uses a non-open data API supplied by property portal Zoopla44. This is not uncommon, and the developers and entrepreneurs I spoke to all agreed with Vasanth Subramanian of Splittable, that HMLR’s Price Paid Dataset was just “one tool in the toolbox”:

It’s when it’s combined with other data, expertise or products you or your organisation might possess that the real value of the dataset is realised.45

A developer who did not wish to be identified feared that because Price Paid data was creating value only in conjunction with other datasets, arguments for it to remain open during any privatisation process would be weakened. This would be a mistake, according to Vasanth Subramanian:

It would be a big problem if the Price Paid got sold off with HMLR. Even if the dataset [is not providing amazing value in and of itself], it’s a vital building block and you’d see a slow down in proptech innovation.46

How can we measure the impact of HMLR releasing their Price Paid data? Henry Pryor believes one of the main benefits will be in bringing transparency to the housing market, and likens the change to the introduction of screen-based trading on the London Stock Exchange in the 1980s:

This data shines a light into a murky world and provides transparency which in turn provides trust and confidence in transaction values associated with the purchase and sale of residential property. We don’t have it in the rental market and the market suffers as a result. But just as we saw with Big Bang in the city and this idea of effectively real-time pricing, buyers and sellers can get far more confidence and as a result you get a far firmer, more mature market where confidence is high.47

This echoes Adrian Black’s experience on the ground: that homeowners use data to feel comfortable about the price of the property they are buying or selling, helping the market move more quickly.

But the state of the UK housing market is subject to many more forces than the state of the estate agency business, however “murky” or “Stone Age” those involved in it believe it to be. Prior to the UK’s 2007 credit crunch, precipitated by the run on mortgage lender Northern Rock, housing transaction volumes had peaked at around 150,000 a year. 2015 figures (see Figure 5) show they have now stabilised at around 100,000 a year48. Although the steady upward climb towards this figure from a low of around 50,000 a year in December 2008 appears to accelerate after March 2012, given the complexity of the market and the availability of the data to market participants through HMLR’s paying customers prior to this date, it seems wise to regard HMLR’s open data release as a correlation, and not a cause.

Figure 5: Total UK Residential Property Transactions. Reproduced from (HMRC, 2015).

The OECD ranks the UK in the top 10 countries with over-valued housing markets, one of six in this group where house prices are still rising49. The affordability of homes in the UK is a significant political issue. Transaction volumes are also distorted by the attempts of successive governments to both maintain an economy that is back-stopped for many voters by the (inflated) value of their property, and also help first-time buyers afford a home50.

Compared to ascertaining the release of open data’s effect on the housing market, its effect on innovation in the property sector looks slightly more straightforward. Here, the rise of the proptech sector, and the opening of Pi Labs, look like good indicators for innovation, and many of the developers and entrepreneurs involved are quick to recognise the contribution of HMLR. But all of them qualify this with statements that indicate that HMLR data works best in conjunction with other datasets, be they other open datasets (as with Illustreets) or, more often, closed data sets (as with Peter Thum-Bonanno’s GetAgent). This stands in contrast to TfL, where the end products created by innovators are far more likely to rely on TfL data alone. Any contribution that proptech makes or goes on to make to the UK economy will be hard to pick apart: It will be hard to say how much of this is thanks to HMLR’s open data policy.

One area where it is easiest to say that the open data policy has had impact is data quality. Lynne Nicholson states that increased exposure has also helped HMLR to maintain data quality and that they are reacting to an increased number of queries since release:

Now we’re publishing information, it’s more accessible, more people are viewing it, so you’re bound to get more queries and questions, and requests too. I’ll give you one example. Say we’ve captured a property as an end terrace. The owner might ring up and say ‘I regard my property as semi-detached, not an end terrace’. So those are some of the types of questions we get.51

HMLR report an increase in the number of notifications about inaccurate data by members of the public since the Price Paid data was released, saying they received 2,256 issues related to incorrect or missing Price Paid information from the release of the data up to the publication of their second privacy impact assessment in May 2013. However, no figure for complaints about inaccurate data prior to the Price Paid release is given for comparison52.

HMLR is committed to making the data it releases easier to digest, and has invested in a reporter tool for Price Paid data that it believes is driving more non-expert users to its data.

As for the proptech sector, for the moment innovation rather than consolidation looks set to rule. “You’ve got lots of startups like ours that are nibbling away at small bits of the value chain,” says Peter Thum-Bonanno:

So you have companies that are trying to make getting a mortgage easier and more straightforward. We’re trying to help you find an estate agent .... There’s no one yet who is doing the whole process for you from finding an estate agent, finding you a buyer, helping you move, helping you manage your home. It seems like that’s the aspiration of all these small proptech startups and it will be interesting to see if there will be a consolidation in the industry, allowing you to manage your entire home online, or if it’s going to continue to be piecemeal in the future. I’m not sure.54

HMLR’s data is delivering value to existing stakeholders in property price information (estate agents and homeowners) and stimulating innovation that looks set to contribute to UK economic growth. Putting an exact figure on the impact of the data, however, may never be possible.

In the absence of quantitative methods, it is likely that future data release policies will be governed by other factors. Attitudes to HMLR’s current data release policy, and those of other trading funds such as Ordnance Survey, varied wildly between the data users interviewed. Innovators ranged from being “very grateful” to HMLR for the data release, to feeling frustrated that more data was not being either collected (e.g., number of bedrooms, square footage) or released (e.g., full land ownership records). Meanwhile, at the more traditional end of the spectrum, Henry Pryor did not see why businesses who were able to monetise HMLR data were not required to pay for it.

This contrast in attitudes persists to the highest levels of government. Claudia Arney, chair of the Public Data Group between June 2012 and its final meeting in April 2015, describes trading funds navigating their open data strategies in an environment of competing pressures: strong pressure to open up data; and equal and opposite pressure to provide evidence that any change to the status quo represented true value for money and maintained the integrity of the data in question55. She believes the integration of the Public Data Group into the Digital Culture, Services Platforms, and Data Board, which will include representatives from the Cabinet Office and HM Treasury, will allow trading funds to plot their future path with greater certainty. But the election of a Conservative government and the broader focus on the health of public finances in UK politics may yet shift the balance.

Any sell-off of HMLR may not be as swift as anti-privatisation campaigners fear. The replacement as Chief Land Registrar of former Student Loans Company Chief Executive Ed Lester with the former Chief Executive of Thurrock and Brentwood councils, Graham Farrant, could be read as a subtle shift away from immediate privatisation towards a strategy of delivering efficiency savings before any sell-off. Confirming Farrant’s appointment in a written statement to the House of Commons, Matt Hancock (then a minister at BIS and now Minister for the Cabinet Office) wrote “Graham has the skills and knowledge to manage the organisation through its transformation into a modern, digital organisation.56”

What will be of consequence is less if or when the sell-off comes, than whether HMLR data is included in the package. Regrettably, the last government set a precedent here when it included the Postcode Address File in the sell-off of Royal Mail, a move that attracted strong criticism from the open data community57 and from Parliament’s Public Administration Select Committee58.

For open data advocates

• HM Land Registry may be a target for privatisation. The open data community needs to pressure the UK government to prevent the bundling of HM Land Registry data with any sale. In particular, the open data community should ensure that government is aware of all the stakeholders in HM Land Registry data as open data, including estate agents and proptech innovators such as those interviewed for this study.

For funders

• Funders could encourage open data advocates to network more with innovators in the proptech sector, in order to cement alliances against data privatisation at HM Land Registry.
• Any campaign against data privatisation will be greatly supported by economic analysis that speaks the language of those making the decisions at HM Treasury. However, it is also important to recognise that open data reforms are in some sense a leap of faith (see TfL study).