Every few months I get a message from a developer or technical founder who’s already built a Companies House scraper and wants to know if they should switch to the API. The answer is always yes, and usually the conversation that follows reveals they’d have saved themselves two or three weeks of work if they’d started with the API in the first place.
This post is for anyone who’s considering scraping the Companies House website as an alternative to using the official API, or who has already built a scraper and is wondering whether it’s worth rebuilding. I’ll cover legality, reliability, data completeness, and maintenance burden — then you can make the call.
First: why people consider scraping
The reasons are usually one of three:
They didn’t know the API existed. Companies House has had a public REST API since 2015, but discovery is poor — the developer hub isn’t prominently linked from the main website and it doesn’t rank well for casual searches. Plenty of developers stumble on the website first and reach for a scraper before looking for an API.
They tried the API and hit a friction point. The API requires registration and an API key, which takes five minutes but feels like a barrier. Scraping can start immediately with no registration. For a quick prototype this is tempting.
They need data the API doesn’t provide. This is the only technically legitimate reason to consider scraping, and it’s rarer than people think. More on this below.
The legal position
Companies House publishes its data under the Open Government Licence. The data itself is free to use. The question is whether scraping the website to access it is permitted.
The Companies House terms of use are clear: the website is provided for human users. Automated access to the website — scraping — is not permitted without explicit agreement from Companies House. The API exists precisely to provide legitimate programmatic access to the data that scraping would otherwise require.
In practice, scraping a government website for publicly available data is unlikely to result in legal action for a small-scale use. But it’s a terms of service violation, and Companies House actively blocks scraping attempts through rate limiting and bot detection. Building on a foundation that violates the terms of service of your data source is not a sensible technical or legal position for a production system.
The API is the legitimate route. It’s free. Use it.
Reliability
This is where the practical difference between scraping and the API becomes most stark.
The Companies House API is a versioned REST API with published uptime SLAs and advance notice of breaking changes. When something changes, you get notice and a deprecation timeline. The response format is stable and documented.
Scraping the Companies House website means your code depends on the HTML structure of pages that Companies House can change at any time without notice. A design update, a template change, a class name refactor — any of these can silently break your scraper. Not with an error you can catch, but by returning wrong data that looks correct until you notice something is off.
I’ve seen scrapers that worked perfectly for six months fail silently after a Companies House website update, returning empty director lists for every company. The system continued to function — it just stopped verifying anything. That kind of failure is worse than an obvious error because it’s invisible.
Rate limits
The Companies House API rate limit is 600 requests per five minutes per API key. That’s generous for most use cases. If you need more, you can apply for an increased limit. The limit is documented, predictable, and enforced with a standard 429 response you can catch and handle.
Scraping rate limits are enforced by bot detection, IP blocking, and CAPTCHAs — none of which give you a clean error to handle. Your scraper either gets through or it doesn’t, and managing around bot detection is an ongoing maintenance task as Companies House updates its defences.
Data completeness
The Companies House API provides access to the full register: company profiles, officer details, PSC data, filing history with document links, charges, insolvency cases, and more. The data is structured, consistently formatted (within the limits I’ll get to in the gotchas post), and documented.
Scraping the website gives you whatever is visible on the page at the time. Some data that’s available via the API isn’t surfaced prominently on the website. Filing document metadata, charge details, and certain PSC fields are easier to access via the API than by scraping the corresponding pages.
The one area where scraping might seem to have an advantage is the document content itself — the actual PDF filings. The API provides links to filed documents and their metadata, but doesn’t parse the content of accounts documents into structured financial data. If you need profit figures from filed accounts, neither the API nor the website gives you that as structured data — you’re looking at PDF parsing either way, which is a different problem entirely.
Maintenance burden
This is the biggest practical argument against scraping and the one that tends to land hardest with technical founders who’ve already built a scraper.
An API integration, once built and tested, is essentially maintenance-free until the API changes — and API changes come with notice. The integration I build comes with documentation on how to handle the update when it comes.
A scraper requires ongoing maintenance regardless of whether you want it to. Every time the Companies House website updates, there’s a risk your scraper breaks. You need monitoring to catch when it fails. You need time to fix it when it does. If you’re running this in production for a business-critical process, that maintenance obligation is real and recurring.
The time cost of maintaining a scraper over 12 months typically exceeds the time cost of building the API integration correctly in the first place.
When might scraping make sense?
Honestly, almost never for Companies House specifically. The API covers everything the website covers for programmatic use cases, it’s free, it’s legal, and it’s stable. The friction of getting an API key is genuinely minimal.
The only scenario I can construct where scraping is justifiable is a one-off data extraction for a research project where the data needed is genuinely not available via the API and you’re not going to run the scraper in production. That’s a narrow edge case and even then you’d be operating outside the terms of service.
Making the switch
If you’ve already built a scraper and it’s running in production, the switch to the API is worth doing. The API responses are more reliable and more consistently structured than scraped HTML. The maintenance burden drops. And you’re no longer operating outside the terms of service of your data source.
The rebuild isn’t starting from scratch — the business logic of what you do with the data stays the same. What changes is the data source: instead of parsing HTML, you’re parsing a documented JSON response. The data field names are different but the values are the same.
You can see what the API returns by trying the live Companies House demo tool — the same endpoints your integration would use, returning live data from the register. If you need help with the rebuild, the Companies House API integration service covers it at a fixed price.
For the practical side of working with the API — the edge cases and inconsistencies that aren’t obvious from the documentation — see the Companies House API gotchas post. And for pricing, the cost breakdown post covers what drives the price of a properly built integration.
Related posts
Need a custom integration built?
I build custom API integrations — Stripe, Companies House and bespoke data pipelines. Reliable, well-documented, no agency overhead.
Discuss your project →