Skip to content
PTEN

Web scraping for AI agents: permits from 74 towns

How I built a data pipeline for AI agents: six permit systems, raw data kept for reprocessing, and an MCP server that teaches the agent the caveats.

Massachusetts building permits are public, but they live in six different municipal systems with no shared API. I built a data pipeline for AI agents that collects them from 74 towns into one PostgreSQL database and serves it through a read-only MCP server. This post covers the parts that matter when the consumer is an agent rather than a person: keeping the collection running, and keeping the agent from drawing wrong conclusions.

The project page is Cape Cod Permits. The numbers below come from the code and from the production MCP server.

The problem: one dataset, six platforms

Each town picks its own permit vendor. In this project that means OpenGov, PermitEyes, Patriot PermitPro, PermitLink, MapsOnline, and, for one town, monthly XLSX reports. They don't agree on anything:

  • PermitEyes serves a DataTables listing plus JSON detail endpoints.
  • Patriot PermitPro has a guest dashboard that returns JSON, newest first.
  • PermitLink is an InterSystems Zen page, read in headless Chromium because paging is an encrypted request.
  • MapsOnline only offers a "Permits Listing" PDF with no date column, so the collector asks for one day at a time and parses the PDF word boxes.

Today the database holds 147,639 records from 74 towns: 15 on Cape Cod and 59 elsewhere in Massachusetts. That count includes administrative records, so it is not a count of construction jobs. I'll come back to that, because it is exactly the kind of detail an agent gets wrong.

Web scraping for AI agents: one registry, one collector per platform

The first lesson is to normalize many sources behind one registry. Every town is an entry in a single config file with its platform, tenant and portal URL. The enumerator maps each platform to a pass function:

const PASS = {
  opengov: passOpenGov, permiteyes: passPermitEyes, accela: passHarwich,
  patriot: passPatriot, permitlink: passPermitLink, mapsonline: passMapsOnline,
};

for (const item of sources) {
  await loadTenant(item);          // resumable state for this town
  if (!due()) continue;
  const run = PASS[item.platform];
  if (!run) { state.paused_reason = `unsupported_platform:${item.platform}`; await save(); continue; }
  try {
    await run();
  } catch (error) {
    state.last_error = error.message;  // this town fails, the loop moves on
    await save();
  }
}

Adding a town on a platform that is already supported is a config change. Adding a platform means writing one collector and one line in PASS. Patriot, PermitLink and MapsOnline all produce one listing row per record, so they share a single row builder in the normalizer.

I also mapped 1,262 towns within a 200-mile radius to the permit system each one uses. That file decides which collector to write next.

Keep the raw data so you can reprocess

Vendors change their frontends, and my parser changes too. So the pipeline never treats the parsed row as the source of truth. Every raw portal response is written as a gzipped, content-addressed snapshot, with a SHA-256 hash. The normalizer has a version number:

export const PARSER_VERSION = 6; // Bumping reprocesses every record.

Each normalized snapshot is stored with the parser version that produced it. When I bump the version, the normalizer reprocesses every record from the raw files on disk. Nothing gets downloaded again, and the portals see no extra traffic. Fields that don't fit the common model stay in dynamic fields and in the raw snapshot, so they aren't lost either.

Isolate failures and save progress per town

Public portals fail all the time: rate limits, expired credentials, layout changes, outages. A data pipeline for AI agents has to treat that as normal.

Two rules handle it. First, a failure stays inside its platform. If a portal rejects its credentials, only that platform's towns pause with a paused_reason, and the other platforms keep collecting. Second, progress is saved per town in the database. The enumerator writes its state every 20 IDs, per listing page and at every pass boundary. A redeploy or a crash resumes where it stopped instead of restarting a crawl that can take hours.

Monitor progress, not completed cycles

My first instinct was to alert when no collection pass had finished recently. That is wrong for this workload. A backfill over a year-wide window can run for hours for a single town, so "no pass finished" would fire on every backfill and every weekly refresh.

The health check uses progress instead. Since the enumerator saves state constantly, the newest updated_at across all towns works as a heartbeat:

const lastBeat = Math.max(0, ...states.map((s) => Date.parse(s.updated_at || 0) || 0));
if (states.length && lastBeat && Date.now() - lastBeat > STALE_BEAT_MS) {
  reasons.push({ code: 'stale', text: `Enumerator has made no progress for ${minutes} min` });
}

A long backfill raises no false alarm, and a real stall shows up within 20 minutes. Every reason has a stable code for monitors and a readable text for people.

An MCP server that teaches the agent the caveats

Collecting the data is half the job. The other half is handing it to an agent without letting it produce confident nonsense. This dataset has traps that a person learns over weeks and an agent will hit in its first query:

  • Record types and statuses are not standardized across towns. There are hundreds of spellings.
  • One project generates several permits (building, electrical, plumbing, gas), plus administrative records. Counting permits is not counting jobs.
  • estimated_cost is self-reported and only present in 44% of records.
  • Coverage is uneven. Each town has its own date range, and towns outside the Cape are added one at a time.

The MCP server has 7 read-only tools and a dataset guide the agent can read as a resource or through a dataset_docs tool. The caveats aren't only in the guide. They are in the tool descriptions, where the agent actually looks when it decides what to call. For example, list_facets returns each town's real from_date and to_date, and its description tells the agent to call it before comparing towns or periods. permit_stats says:

Use this for any "how many", "how much" or "which are the largest"
question instead of paging search_permits. Each row has `permits`,
`with_cost`, `total_cost` and `median_cost`. Since only 44% of records
have a cost, compare `with_cost` to `permits` before quoting the sum,
and prefer the median to the mean.

So the server pushes the agent toward aggregates. The database does the counting in one query, and the agent doesn't page through 200-row results and add them up in its context window. get_permit says to use it for one case, never in a loop over a list. The server instructions also ask the agent to always state the period and the filters it used.

A tool schema tells the agent what it can ask. The descriptions and the dataset guide tell it what the answer means.

Access control per key

The server sits behind API keys. Each key has its own name in the audit log and its own rate limit, set per credential rather than per IP. The reason is in the code: an agent asked for "everything" will paginate until the pages run out, and one agent shouldn't be able to pin the database. Every read of contact data is logged with the filters that were used.

Lessons

  1. Normalize many sources behind one registry. One config entry per town, one collector per platform. A new town is configuration.
  2. Keep the raw data. Store raw responses compressed, with a hash and the parser version. Then a parser fix is a reprocess, not a re-crawl.
  3. Isolate failures. One platform breaking should pause only its own towns, and progress should survive a redeploy.
  4. Monitor progress. For long-running collection, a heartbeat beats "did the cycle finish."
  5. Write the MCP server for the agent's mistakes. Put coverage and data-quality caveats in the tool descriptions, ship a dataset guide, and steer the agent toward aggregates.

I built this with Claude Code, which is how I work on every project. If you need public data collected and served to agents in a way they can trust, get in touch.

All articles