Get Google AI Overviews citations in cURL

Strip it to parts and the whole job is one pipe: curl -s POSTs the query with "surfaces": ["google_ai_overview"], the response carries the overview's reference list at answers[0].sources[], and jq prints it ranked. No SDK exists and none is missing — curl and jq carry everything from first call to cron job. What this surface adds is a bookkeeping duty the chat engines don't have: Google doesn't put an AI Overview on every results page, so a scheduled script has to count the overviews that weren't there as carefully as the citations that were. The task-level story — manual collection versus the API — is covered separately; this page stays in the terminal.

The one-liner and what it returns

The minimal call, assuming an ag_live_ key from onboarding — or watch a keyless fetch first in the playground. country and language ride along because this surface reads both, and --max-time 200 sits above the API's 180-second hold — the contract allows the full hold even though AI Overview fetches usually return well under it:

curl -s --max-time 200 https://api.agentgeo.org/v1/fetches \
  -H "Authorization: Bearer ag_live_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"query": "best payroll software for restaurants", "surfaces": ["google_ai_overview"], "country": "US", "language": "en"}' \
  | jq -r '.answers[0].sources | sort_by(.position)[] | [.position, .url] | @tsv'

Two columns come out — position, URL — in the order Google attached the references. Behind the filter, the full response:

{
  "id": "run_5b02d7e3c1a9",
  "query": "best payroll software for restaurants",
  "surfaces": ["google_ai_overview"],
  "status": "completed",
  "recordsDelivered": 1,
  "creditsCharged": 1,
  "answers": [
    {
      "surfaceKey": "google_ai_overview",
      "answerText": "Popular payroll options for restaurants include ...",
      "sources": [
        { "title": "Best Restaurant Payroll Software Compared", "url": "https://example.com/restaurant-payroll", "position": 1 },
        { "title": "Payroll for Hourly and Tipped Employees", "url": "https://guides.example.org/tipped-payroll", "position": 2 }
      ],
      "fetchedAt": "2026-07-28T06:41:17Z"
    }
  ]
}

Read three things off that before piping it away. position is the ranking this surface hands you — Google attached the references in an order, and flattening them into a set of links deletes it. fetchedAt is the anchor for every future diff. And creditsCharged counts delivered records only: a query whose SERP carried no overview comes back status: "failed" at zero credits, which is the behaviour the production script below is built around.

A queries file, a loop, a dated ledger

One list answers this morning's question. The recurring one — which hosts keep supplying Google's answers across a topic, and whether that set moved — needs several queries, one normalization convention, and rows that outlive the run. Queries go in a flat file:

best payroll software for restaurants
payroll rules for tipped employees
how much does restaurant payroll software cost
restaurant payroll and tip pooling compliance

#!/usr/bin/env bash
# aio-citations.sh — AI Overview citations for every query in queries.txt,
# appended as dated rows to aio-citations.csv. Two counters, never blended:
#   ASKED    every query sent
#   PRESENT  queries where Google actually showed an overview
set -euo pipefail

KEY="ag_live_your_key_here"
CSV="aio-citations.csv"
GL="US"   # country -> Google gl=
HL="en"   # language -> Google hl=
TODAY=$(date -u +%F)

[ -f "$CSV" ] || echo "date,host,position,url,query,title" > "$CSV"

ASKED=0; PRESENT=0; CITED=0

while IFS= read -r query; do
  [ -n "$query" ] || continue
  ASKED=$((ASKED + 1))

  # jq -cn builds the body — an apostrophe in a query can't break the JSON
  body=$(jq -cn --arg q "$query" --arg gl "$GL" --arg hl "$HL" \
    '{query: $q, surfaces: ["google_ai_overview"], country: $gl, language: $hl}')

  resp=$(curl -s --fail --max-time 200 https://api.agentgeo.org/v1/fetches \
    -H "Authorization: Bearer $KEY" \
    -H "Content-Type: application/json" \
    -d "$body") || { echo "curl failed: $query" >&2; continue; }

  if [ "$(jq -r '.status' <<<"$resp")" != "completed" ]; then
    # No AI Overview on this SERP — failed record, zero credits. The absence
    # lives in the counters and the log, never as a row in the CSV.
    echo "no overview: $query" >&2
    continue
  fi

  PRESENT=$((PRESENT + 1))

  if [ "$(jq '.answers[0].sources | length' <<<"$resp")" -eq 0 ]; then
    continue  # counted as present, not as cited
  fi
  CITED=$((CITED + 1))

  jq -r --arg d "$TODAY" --arg q "$query" '
    .answers[0].sources | sort_by(.position)[]
    | [$d,
       (.url | sub("^https?://(www\\.)?"; "") | split("/")[0] | ascii_downcase),
       .position,
       (.url | split("?")[0]),
       $q, .title]
    | @csv' <<<"$resp" >> "$CSV"

done < queries.txt

echo "AI Overview presence: $PRESENT/$ASKED queries"
echo "cited among present:  $CITED/$PRESENT overviews"

Three decisions in there deserve a sentence each. Bodies are built with jq -cn --arg, because the first apostrophe in a real query ends a single-quoted -d string and hands the API broken JSON. A missing overview never becomes a CSV row — it goes to stderr and into the ASKED/PRESENT counters, because an empty row would read as "an overview that cited nobody", which is a different and much rarer fact. And the comma-free columns — date, host, position — lead the row order, so cut -d, keeps telling the truth no matter what punctuation a page title ships with.

# Hosts Google keeps reaching for — host is column 2, comma-free by construction
tail -n +2 aio-citations.csv | cut -d, -f2 | tr -d '"' | sort | uniq -c | sort -rn | head -15

# crontab -e — Mondays 06:40 UTC; each run appends one dated layer
40 6 * * 1 cd /home/you/geo && ./aio-citations.sh >> aio.log 2>&1

tr -d '"' strips the quoting @csv adds. After a few cron cycles, the same cut over two date ranges gives you hosts that entered and hosts that dropped — with the position column still there to show a slide from second reference to sixth, which a present/absent flag records as no change at all. More shell idioms against the same endpoint, including multi-engine calls, live on the cURL API page.

gl, hl, and the overview that isn't there

Two optional fields do real work on google_ai_overview and a third does none. country sets the search gl= — which market's results page gets asked — and is also read by copilot, so a per-market setup can share the variable. language sets hl=, and this is the only surface of the six that reads it; country: "CH" with language: "de" is a legitimate combination that produces a different overview than either default. web_search belongs to chatgpt alone — sending it here does nothing except mislead whoever inherits the script.

The other engine-specific fact is absence. Google decides per SERP whether an AI Overview appears, so any ratio you publish needs two denominators kept strictly apart:

Because the fetch rides the results page rather than a chat session, this is one of the faster surfaces — most calls return well before the 180-second ceiling the API is allowed to use. Keep --max-time 200 anyway; a shorter client limit copied from another script is the classic self-inflicted failure. And run one ledger per gl/hl pair: a French SERP cites different pages than a US one, sometimes has an overview where the other has none, and rows from two markets in one CSV can't be pulled apart later.

Common problems

• Blending the denominators is the classic corruption. Log a missing overview as "cited nobody" and every host in the ledger looks like it's losing ground the week Google trims overview coverage on your topic. Absence goes in the counters; only delivered references go in the CSV.
• Single-quoted -d bodies meet real queries. The first "what's" ends the shell string and the API receives invalid JSON. Build every body with jq -cn --arg — the script does, and one-off commands should too.
• A failed record on this surface is not a scrape to nurse. It almost always means the SERP carried no AI Overview, at zero credits. An immediate retry tends to reproduce the same absence — let the next scheduled run ask again, and read the run's stderr for which queries went missing.
• Un-normalized URLs split one page into five rows. Tracking parameters, www., mixed-case hosts. The script lowercases hosts and drops query strings before writing; if you ever change the convention, restate the whole CSV — rows normalized two different ways can't be diffed.
• One CSV, two markets. Different gl/hl pairs change the reference lists and the presence pattern. Keep a file per market, or add the market as a leading comma-free column on day one; retrofitting means guessing which rows came from where.

Summary

One POST and one jq filter turn a Google AI Overview's references into shell data — title, url and position at answers[0].sources[]. The one-liner answers today's question; the loop, the counters and the dated CSV answer the durable one: how much of the topic Google answers inline, and who it reaches for when it does. Build bodies with jq -cn, keep --max-time above the 180-second hold, count absence apart from citation loss, and let cron accumulate the layers. The same endpoint and shape back the Python and Node.js versions, and the docs hold the full contract.