Grants.gov XML Extract: Parsing the Daily Opportunity Feed

Grants.gov does not offer a practical API for bulk opportunity data. What it offers instead is better once you know how to handle it: the entire corpus, every field, every status, published as one XML file every single day. We ran this feed in production and held 83,002 opportunities from it. This is how it works and what breaks.

Why this is a guide and not a product. We retired our grant-opportunities endpoints in July 2026 to focus on federal contracting, which is a different domain with a different buyer. The feed is free and public. The parsing knowledge below is the part that took us time.

The source

https://prod-grants-gov-chatbot.s3.amazonaws.com/extracts/GrantsDBExtract{YYYYMMDD}v2.zip

Substitute today's date, for example GrantsDBExtract20260718v2.zip. That is a 77 MB ZIP containing roughly 318 MB of XML. It is not a delta. It is the complete corpus, republished daily, with every opportunity in every status and every field.

Two consequences follow from that, and both are good news. You never need to reconcile deletions or track a cursor, because tomorrow's file is authoritative. And because you keep what you ingest, you accumulate history that Grants.gov itself does not expose: an opportunity that closes disappears from their search, but stays in your table forever.

The model: accumulate all, derive status at query time

Do not mirror the feed by deleting rows that vanish. UPSERT the whole file on opportunity_id, never delete, and compute status when you query rather than when you load.

INSERT INTO grant_opportunities (...) VALUES %s
ON CONFLICT (opportunity_id) DO UPDATE
SET ...
WHERE grant_opportunities.last_updated_date IS DISTINCT FROM EXCLUDED.last_updated_date;

That WHERE clause matters more here than almost anywhere else. You are re-loading the full corpus every day, so without it you rewrite 83,000 rows daily to change nothing, generating WAL and vacuum work forever. With it, a steady-state day touches only what actually moved.

Five things that break naive parsers

1. Do not load 318 MB into a DOM

ET.parse() will hold the entire tree in memory. Stream it and release each element as you finish with it:

import xml.etree.ElementTree as ET

for _ev, elem in ET.iterparse(xml_path, events=("end",)):
    tag = _strip(elem.tag)
    if tag in ("OpportunitySynopsisDetail_1_0", "OpportunityForecastDetail_1_0"):
        d = _collect(elem)
        oid = _one(d, "OpportunityID")
        if oid and oid not in seen:
            seen.add(oid)
            batch.append(_row(tag, d))
        elem.clear()          # without this, memory grows to the full 318 MB

2. There are two record types, with different children

OpportunitySynopsisDetail_1_0 is a posted opportunity. OpportunityForecastDetail_1_0 is a forecast of one that has not opened yet. They do not share a field set. Code written against one will silently drop fields from the other, and forecasts are roughly where next quarter's pipeline lives, so dropping them is expensive.

3. Namespaced tags

Every tag arrives namespaced, so find("OpportunityID") returns None forever and you will assume the field is missing. Strip the namespace and match the local name:

_strip = lambda t: t.split('}')[-1]

4. Entities are double-encoded

This one cost us real time. The feed does not just serve raw entities, it sometimes serves double-encoded ones: &apos; which unescapes to ' which unescapes to an apostrophe. A single html.unescape() leaves visible entity codes in your titles. Unescape until the string stops changing, with a bound so a pathological input cannot loop:

import html

def _unescape(s):
    """grants.gov sometimes double-encodes (&amp;apos; -> &apos; -> ');
    unescape until stable, bounded."""
    for _ in range(3):
        new = html.unescape(s)
        if new == s:
            return s
        s = new
    return s

5. Dates are MMDDYYYY with no separators

09152026 is 15 September 2026. The danger is not that a naive parser fails, it is that %d%m%Y succeeds on ambiguous dates and gives you a valid, wrong answer. Parse explicitly and reject anything that is not eight digits:

def _pdate(s):
    s = (s or "").strip()
    if len(s) == 8 and s.isdigit():
        try:
            return datetime.datetime.strptime(s, "%m%d%Y").date()
        except ValueError:
            return None
    return None

Bonus: multi-valued elements repeat

Fields like eligibility categories appear as several sibling elements, not one delimited string. If you assign rather than append you keep only the last one. Collect into a list and store as text[]:

def _collect(elem):
    d = {}
    for c in elem:
        d.setdefault(_strip(c.tag), []).append((c.text or "").strip())
    return d

What it costs to run

Metric	Value
Download	77 MB ZIP, daily
Parsed	~318 MB XML
Rows held	83,002 opportunities
Storage	344 MB with indexes
Schedule	One full re-sync per day

The job is not expensive. The cost is that the feed changes shape without announcement, and every gotcha above was found in production rather than in documentation.

We still hold this data. We ran this feed through July 2026 and kept the corpus: 83,002 opportunities including closed and forecast records, which Grants.gov itself does not expose once they age out. If a one-time export would save you building the parser, get in touch. Sold as a dated snapshot, not a feed.