Two of seven city permit datasets we tested had quietly died
Open-data portals abandon datasets without removing them. Two of seven candidate city permit feeds were still serving hundreds of thousands of rows years after they stopped updating. Here is how to check before you build on one.
A permit dataset that has stopped updating is worse than no dataset, because it looks like it is working.
They do not switch them off
When a city migrates permitting systems, the old dataset usually stays published. It keeps its URL, keeps answering queries, and keeps returning a large row count. Nothing about the response says the last row arrived in 2019.
Both of the failures found while building this product looked completely healthy from the outside. New York's DOB Permit Issuance dataset returns nearly a million rows and stops in June 2020. Dallas's Building Permits dataset returns thousands and stops in December 2019.
Candidate city permit datasets that had silently stopped updating while still serving data
Source: EarlyIntent dataset freshness audit
Why row count fools you
The instinct when evaluating a feed is to check that it returns data and that the fields you need are populated. Both of those pass on a dead dataset. Historical rows are complete, well-formed and plentiful, that is precisely what makes them convincing.
For a product whose value is timeliness, this is the worst possible failure mode: it degrades silently into confidently serving six-year-old leads.
The check is one query
Socrata supports aggregate queries, so the newest record date is a single request against the date column you intend to filter on. Anything older than a few weeks means the feed is not being maintained, whatever the row count says.
Two details matter. Check the column you will actually filter on, not any date column, some datasets keep updating a system timestamp long after the source data stops. And check the type: several datasets store dates as text in MM/DD/YYYY, where a lexical comparison against an ISO string silently matches nothing rather than erroring.
- ·Query max() on the date column you filter by, not on a modified timestamp.
- ·Treat anything older than about three weeks as abandoned.
- ·Confirm the column is a real date type, not text, a text date breaks range filters without any error.
- ·Re-run it continuously. A live dataset can die at any time, usually during a system migration nobody announced.
- ·Surface the result in the product, so the failure is visible rather than inferred from an empty feed.
Prefer the replacement, not the familiar name
In New York's case a live replacement existed and was better than the dataset it superseded: DOB NOW carries real job descriptions, borough-block-lot parcel identifiers and estimated job costs, none of which the older feed had. The stale dataset was simply the one with the more obvious name.
Dallas had no live equivalent, so the honest answer was to drop the metro rather than ship a city whose leads would all be from 2019.
How often should freshness be checked?
Continuously, and the result should be visible in the product. A dataset that is healthy today can be migrated tomorrow, and the failure gives no signal other than the newest record date ceasing to move.