GovCon API The federal contracting data layer

Federal Grants Data: How to Build Your Own Ingest Pipeline

We ran a federal grants pipeline in production from June 2026 until we retired it in July 2026. It held 1,182,063 assistance transactions, 83,002 grant opportunities, and 604,078 FFATA sub-grants. This is the complete write-up of how it worked, what the upstream sources actually do, and the specific things that cost us time, so you can build it yourself without repeating them.

Three datasets people confuse

Almost every mistake here starts with treating these as one thing. They come from different systems, key on different identifiers, and answer different questions.

Grant awards key on CFDA (now Assistance Listing) numbers and FAIN. Contract data keys on PIID and NAICS. They do not join cleanly, and any design that assumes they do will produce confident nonsense. We kept grant dollars out of every contract lens for exactly this reason.

Grant awards: the USAspending bulk_download API

There is no paginated JSON endpoint that will give you the full assistance corpus at a sane speed. The working path is the asynchronous bulk download: you submit a job, poll a status URL, then fetch a ZIP.

Submitting the job

POST https://api.usaspending.gov/api/v2/bulk_download/awards/
Content-Type: application/json

{
  "file_format": "csv",
  "filters": {
    "prime_award_types": ["02", "03", "04", "05"],
    "date_type": "action_date",
    "date_range": {"start_date": "2025-10-01", "end_date": "2026-09-30"}
  }
}

Those four award types are block grants (02), formula grants (03), project grants (04) and cooperative agreements (05). That is the definition of "grants" most people mean. Leave them out and you will also pull direct payments, loans and insurance, which behave nothing like grants and will distort every aggregate you compute.

The response gives you a status_url. Poll it until the job reports finished, then download the file_url. The file you want inside the ZIP is Assistance_PrimeTransactions: one row per action, so a single award with six modifications is six rows, not one.

The date_type decision, with numbers

This is the choice that determines whether your daily job takes thirty seconds or twelve minutes. We measured both on 2026-06-17 against the same corpus:

Use last_modified_date for the daily delta. It catches every grant touched recently regardless of how old the action date is, which is the only way to pick up late filings and retroactive corrections. An action_date window silently misses those once the record ages past your window.

But note the third row. last_modified_date cliffs badly as the window widens, because USAspending re-stamps records in monthly batches. Fourteen days is already 38 times the volume of seven. Keep the delta window narrow (we used 10 days) and run it daily so the overlap heals short outages. Then run a separate periodic reconcile on action_date (we used a trailing 60 days) to heal anything a longer outage missed.

Chunk anything longer than a year

USAspending returns HTTP 400 on a date_range wider than roughly one year. For a historical backfill, split into 365-day chunks and submit them. If you are loading several years, submit every chunk up front and poll them together rather than serially, since the server-side compile is the slow part and it parallelises.

The truncated download that will bite you

On 2026-06-16 an 89.5 MB chunk arrived short. The HTTP request returned 200, the file wrote without error, and then zipfile threw BadZipFile at open time. There is no error at the transport layer to catch.

The fix is to treat the download as unverified until proven otherwise:

with urllib.request.urlopen(file_url, timeout=600) as r, open(zip_path, "wb") as f:
    expected = r.headers.get("Content-Length")
    written = 0
    while chunk := r.read(1 << 20):
        f.write(chunk)
        written += len(chunk)

if expected and written != int(expected):
    raise IOError(f"short read: {written} of {expected} bytes")

# and prove it opens, do not assume
with zipfile.ZipFile(zip_path) as z:
    z.testzip()

Then retry the whole download. A partial file that opens fine but is missing its tail is worse than a failure, because it loads silently and you get a gap you will not notice for weeks.

Dedup and idempotency

Key on assistance_transaction_unique_key. It is genuinely row-unique, so it works as a primary key and makes the load idempotent. Re-running yesterday's window is then free rather than dangerous, which matters because your delta windows will overlap by design.

One refinement worth the effort: make the UPSERT skip rows whose last_modified_date has not changed.

INSERT INTO grants (...) VALUES %s
ON CONFLICT (assistance_transaction_unique_key) DO UPDATE
SET ...
WHERE grants.last_modified_date IS DISTINCT FROM EXCLUDED.last_modified_date;

Without that WHERE, every overlapping daily run rewrites tens of thousands of unchanged rows, generating WAL, bloating the table and forcing vacuum work for no information gain. With it, a steady-state daily run writes almost nothing.

Grant opportunities: the Grants.gov daily XML extract

Grants.gov publishes the entire opportunity corpus once a day as a single XML file. Not a delta, not an API. The whole thing, every day.

https://prod-grants-gov-chatbot.s3.amazonaws.com/extracts/GrantsDBExtract{YYYYMMDD}v2.zip

That is a 77 MB ZIP expanding to roughly 318 MB of XML, containing every opportunity in every status with all fields. Because it is a full corpus every time, the sane model is accumulate-all: UPSERT the whole file keyed on opportunity_id, never delete anything, and derive status at query time. An opportunity that closes does not vanish from your table, it just stops matching an open filter. That also means you accumulate history the source does not keep.

Five parsing gotchas, all of which we hit

• Do not load 318 MB into a DOM. Use iterparse and call elem.clear() after each record, or you will hold the entire tree in memory.
• Two record types with different children. The file mixes synopsis records and forecast records. They do not share a field set. Code that assumes one shape will silently drop fields from the other.
• Multi-valued elements repeat. Things like eligibility categories appear as several sibling elements rather than one delimited string. Collect them into an array column (text[]) rather than overwriting on each pass and keeping only the last.
• HTML entities arrive raw. The feed serves ' and & literally inside text nodes. Run html.unescape() or your titles will carry visible entity codes into every downstream surface.
• Dates are MMDDYYYY. No separators. 09152026 is 15 September 2026. Naive parsers will either fail or, worse, silently produce a valid wrong date.

Namespaced tags are the sixth annoyance. Strip the namespace or match on the local name, otherwise every find() quietly returns None.

Sub-grants: FFATA assistance sub-awards

Same bulk_download API, different filter:

{
  "file_format": "csv",
  "filters": {
    "sub_award_types": ["grant"],
    "date_type": "action_date",
    "date_range": {"start_date": "...", "end_date": "..."}
  }
}

The important design decision: keep this in its own table with its own job, separate from contract sub-awards. They key differently (FAIN and CFDA versus PIID), and the contract sub-award feed has its own reconciliation behaviour that does not transfer. We deliberately duplicated the code rather than sharing it, and that was the right call both times we revisited it.

Use action_date here rather than last_modified_date. FFATA sub-award reporting is late and irregular by nature, and the action-date lane is the one that proved reliable in practice.

What it costs to run

Honest numbers from our deployment, so you can decide whether this is worth owning:

Storage is cheap and the daily jobs are small once backfilled. The real cost is neither. It is that three upstream feeds change shape without warning, and every one of the gotchas above was discovered in production rather than in documentation.

If you would rather not run it

The data is public and free at source, and this guide is everything we know about getting it. We are not selling a grants API and do not plan to. If you are building on federal contracting data, that is what we maintain: SAM.gov contract opportunities with full descriptions and searchable attachment text, FPDS contract awards, contract vehicles, the SAM entity registry, exclusions and FFATA contract sub-awards, all cross-linked on the same identifiers.

Official sources

• USAspending API documentation, including the bulk_download endpoints
• Grants.gov XML extract, the daily full-corpus download
• Assistance Listings, the CFDA program catalogue
• FSRS, where FFATA sub-award reports originate

Related guides

• The SAM.gov API: a complete guide
• SAM.gov API rate limits explained
• The USAspending API for contract data
• Federal contract awards (FPDS)