Parse Google AI Overviews without scraping the SERP

To parse a Google AI Overview reliably, don't parse the SERP — request the surface directly. POST your query to https://api.agentgeo.org/v1/fetches with "surfaces": ["google_ai_overview"] plus a country and language, and read back the overview's answer text and a structured sources[] as JSON. Scraping the AI Overview block out of Google's search HTML is doubly brittle: the markup is obfuscated and rebuilt on deploy, and the overview often isn't rendered at all — for many queries Google shows no AI Overview, and a SERP scraper can't tell 'didn't parse' from 'wasn't there.' This page shows the structured call, the sources[] shape, and — the part that matters most — how to treat absence as data by tracking a presence rate.

The short version: requests.post to the fetch endpoint with {"query": "...", "surfaces": ["google_ai_overview"], "country": "US", "language": "en"}, then read answers[0]. When the overview exists you get its text and a parsed sources[]; when Google shows no overview for that query, the record comes back status: "failed" at zero credits — and that absence is the first signal, not an error to swallow. The country and language params matter here specifically: they map to Google's gl and hl, and the overview's presence and content shift by locale.

Why parsing the SERP HTML fails

The instinct is to fetch the Google results page and pull the AI Overview block out of the DOM. It works in a notebook once and falls over in production, for two independent reasons:

• The markup is obfuscated and unstable. Google's SERP HTML uses hashed, rebuilt-on-deploy class names and deeply nested containers. The selector that isolated the overview last week returns nothing this week, with no error — your scrape 'succeeds' and yields an empty string. You discover it when the data looks wrong, not when the code runs.
• The overview often isn't there. Google renders an AI Overview for some queries and not others, and the same query can show one today and none tomorrow. A SERP scraper that finds no overview block can't distinguish 'my selector broke,' 'Google didn't render one,' and 'the block loaded after my snapshot' — three very different situations that all look like an empty result.
• Locale changes the outcome. Presence and content depend on gl (country) and hl (language). Scrape from one datacentre IP and you get one locale's view, silently — you're measuring a geography you didn't choose and can't easily vary.
• Bot defenses apply. Automated SERP fetching trips rate limits and challenges like any scraping does, so you inherit a proxy-and-detection maintenance burden on top of the parsing one.

The structured surface removes all four at once. Google's overview is fetched server-side and returned as parsed data, absence is reported explicitly rather than looking like a parse failure, and locale is a parameter you set rather than an IP you hope for.

The structured call

google_ai_overview is the one surface that honours both language (Google's hl) and country (Google's gl). Set them deliberately — the overview you get for US/en is not the one you get for GB/en or DE/de.

import requests

resp = requests.post(
    "https://api.agentgeo.org/v1/fetches",
    json={
        "query": "how to reduce cloud storage costs",
        "surfaces": ["google_ai_overview"],
        "country": "US",   # maps to Google's gl
        "language": "en",  # maps to Google's hl
    },
    headers={"Authorization": "Bearer ag_live_your_key_here"},
    timeout=200,  # the API holds slow scrapes up to 180s - stay above that
)
resp.raise_for_status()

answer = resp.json()["answers"][0]
if answer.get("status") == "failed":
    # Absence is the signal: Google showed no AI Overview for this query/locale.
    print("no AI Overview present (0 credits)")
else:
    print(answer["answerText"][:300], "...\n")
    for src in answer.get("sources") or []:
        print(f"{src['position']:>2}. {src['url']}")

When the overview is present, the response is one JSON object with the answer text and an ordered sources[]:

{
  "id": "run_a41c92f7b0de",
  "query": "how to reduce cloud storage costs",
  "surfaces": ["google_ai_overview"],
  "status": "completed",
  "recordsDelivered": 1,
  "creditsCharged": 1,
  "answers": [
    {
      "surfaceKey": "google_ai_overview",
      "answerText": "To reduce cloud storage costs, review storage classes, set lifecycle rules to tier or delete stale objects, and remove orphaned snapshots ...",
      "sources": [
        { "title": "Cloud Storage Cost Guide", "url": "https://example.com/cloud-storage-costs", "position": 1 },
        { "title": "Lifecycle Rules Explained", "url": "https://example.org/lifecycle-rules", "position": 2 }
      ],
      "fetchedAt": "2026-09-29T09:11:40Z"
    }
  ]
}

When Google shows no overview, the record comes back failed at zero credits — and this is the shape to design around, not an exception to hide:

{
  "id": "run_b73e18a2c5fd",
  "query": "acme corp headquarters address",
  "surfaces": ["google_ai_overview"],
  "status": "completed",
  "recordsDelivered": 0,
  "creditsCharged": 0,
  "answers": [
    {
      "surfaceKey": "google_ai_overview",
      "status": "failed",
      "answerText": "",
      "sources": [],
      "fetchedAt": "2026-09-29T09:12:03Z"
    }
  ]
}

recordsDelivered is 0 and creditsCharged is 0. You paid nothing, and you learned something concrete: for this query, in this locale, Google decided an AI Overview wasn't warranted. That is a real, trackable fact about your topic — and a SERP scraper reports it as the same empty string it returns when its selector breaks.

Track presence rate as a metric

Because absence is data, the first metric worth computing for Google AI Overviews isn't your citation share — it's how often an overview appears for your query set at all. Presence rate is the denominator everything else divides by: a domain that's cited in every overview it could appear in still has near-zero reach if overviews only render for 10% of the topic. This script fetches a query set and reports presence rate alongside the citations, spending zero credits on the absent ones.

"""Fetch Google AI Overviews for a query set; track presence rate and citations."""
import csv
from collections import Counter
from datetime import date
from pathlib import Path
from urllib.parse import urlparse, urlunparse

import requests

API = "https://api.agentgeo.org/v1/fetches"
KEY = "ag_live_your_key_here"
COUNTRY, LANGUAGE = "US", "en"

QUERIES = [
    "how to reduce cloud storage costs",
    "best object storage for backups",
    "s3 lifecycle rules explained",
    "cloud storage vs on-prem cost",
]


def normalize(url):
    p = urlparse(url)
    host = p.netloc.lower().removeprefix("www.")
    clean = urlunparse((p.scheme, host, p.path.rstrip("/"), "", "", ""))
    return host, clean


def fetch(query):
    resp = requests.post(
        API,
        json={
            "query": query,
            "surfaces": ["google_ai_overview"],
            "country": COUNTRY,
            "language": LANGUAGE,
        },
        headers={"Authorization": f"Bearer {KEY}"},
        timeout=200,  # the API holds slow scrapes up to 180s
    )
    resp.raise_for_status()
    return resp.json()


present = 0
hosts = Counter()
rows = []

for query in QUERIES:
    run = fetch(query)
    answer = run["answers"][0]
    if answer.get("status") == "failed":
        # No overview for this query/locale - the signal, at zero credits.
        rows.append([query, "absent", "", "", "", answer.get("fetchedAt")])
        print(f"absent   {query}")
        continue
    present += 1
    for src in answer.get("sources") or []:
        host, clean = normalize(src["url"])
        hosts[host] += 1
        rows.append([query, "present", src["position"], host, clean,
                     answer.get("fetchedAt")])
    print(f"present  {len(answer.get('sources') or [])} sources  {query}")

rate = present / len(QUERIES) if QUERIES else 0
out = Path(f"ai-overview-{date.today():%Y-%m-%d}.csv")
with out.open("w", newline="", encoding="utf-8") as f:
    w = csv.writer(f)
    w.writerow(["query", "presence", "position", "host", "url", "fetchedAt"])
    w.writerows(rows)

print(f"\npresence rate: {present}/{len(QUERIES)} = {rate:.0%}")
print("most-cited hosts (within present overviews):")
for host, n in hosts.most_common(10):
    print(f"{n:>3}  {host}")

The present/absent column is the whole point: absent rows carry no credits and no citations, but they carry the query and the timestamp, so presence rate becomes trendable. Watch it move across months and you learn whether Google is expanding AI Overviews into your topic or pulling back — a shift that reframes every citation number underneath it.

Surface behaviour and failure modes

• Absence is the default failure, and it's free. When Google shows no AI Overview, the record is status: "failed" with recordsDelivered and creditsCharged at 0. Don't treat it as an error — treat it as the presence signal. raise_for_status() won't flag it, so check answer["status"] explicitly.
• country and language are load-bearing here. This is the surface that reads both — country is gl, language is hl. They change whether an overview appears and what it says. Set them on every request; on other surfaces language is ignored entirely and country is read only by copilot.
• A slow scrape can exceed the 180-second budget. That's distinct from absence: it's a scrape that ran long, comes back failed at zero credits, and carries providerFields.snapshot_id. Redeem it with a follow-up POST carrying a top-level "snapshot_id" and the same single surface — don't re-pay for the scrape.
• A client timeout under 180 seconds cancels healthy requests. The API holds the connection up to 180 seconds; timeout=30 aborts overviews that were about to arrive, and no timeout hangs forever. Use timeout=200.
• position ranks within one overview only. First of three sources in a sparse overview is not the same outcome as first of ten in a dense one. Keep the query and the source count beside the position if you intend to trend it.

Summary

Parsing a Google AI Overview out of SERP HTML is brittle twice over — obfuscated markup and an overview that frequently isn't there — and a scraper can't tell those two failures apart. Requesting the google_ai_overview surface returns the overview as structured JSON with locale you control via country and language, and it reports absence explicitly at zero credits. Treat that absence as the first signal: track presence rate as the denominator under every citation number, and watch it by locale over time. The same request shape covers the other five engines — the Python API page shows the multi-surface version, and pull citations from Google AI Overviews covers the task-level view.