Measure AI share of voice without scraping

You don't need a browser fleet to measure AI share of voice. Point one query set at a managed API, get back the answer text and the structured sources for each engine, and compute the share in code you own — brand against competitors, appended row by row so next month's run has something to diff against. "Without scraping" is the literal claim: no Playwright, no rotating proxies, no headless Chrome quietly breaking every time an answer engine ships a layout change. The API runs the fetch; you keep the formula. This is the runnable pipeline. For the math behind it — the two denominators, normalization, position-weighting — see the companion post on the share-of-voice formula. This page is the code.

The shortest honest version: POST each query in your set to https://api.agentgeo.org/v1/fetches with the surfaces you care about, read answerText and sources[] off each answer, tally your brand and each competitor, and append the counts to a CSV with the fetchedAt timestamp. Share of voice is a division you do at the end. No part of that touches a browser — the managed API already returned the answer as data, which is the entire point of not scraping.

What "without scraping" actually buys you

Scraping AI answers means running a browser that logs into each engine, types the query, waits for the stream to finish, and parses the rendered DOM into something countable. It works until it doesn't: the engines gate behind auth, reshape their markup on a whim, and rate-limit anything that looks automated. A share-of-voice number built on that stack is one CSS change away from silently going blank.

• No browser fleet to run. The API holds the scrape connection server-side and hands back JSON. Your code is requests.post and a loop — nothing to keep patched against layout changes.
• Structured sources, not parsed HTML. sources[] arrives as {title, url, position} per answer, already ordered. Citation share of voice becomes counting hosts, not writing DOM selectors that rot.
• One contract across six engines. ChatGPT, Perplexity, Google AI Overviews, Google AI Mode, Copilot and Gemini share an identical request and response shape, so a single tally loop covers all of them — you change the surfaces array, not the parser.
• The formula stays yours. The API never computes a share-of-voice score. It returns the raw records; you write the division, which means you can audit it, change the denominator, and defend the number to a client without waiting on a vendor.

That last point matters more than it looks. "Share of voice" has no standard definition for AI answers — mention share and citation share are different numbers computed from different fields, and conflating them is the most common way these reports go wrong. Because you own the counting code, you get to pick, state which one you computed, and keep it consistent. The formula post is where that decision lives; here we compute citation share of voice, and note where mention share would differ.

One call, one engine

Start with a single fetch so the response shape is concrete. query and surfaces are the whole request for most surfaces; the optional parameters (web_search for chatgpt, country for google_ai_overview and copilot, language for google_ai_overview) are honoured only by the surfaces that name them and ignored everywhere else.

import requests

resp = requests.post(
    "https://api.agentgeo.org/v1/fetches",
    json={"query": "best time tracking software", "surfaces": ["perplexity"]},
    headers={"Authorization": "Bearer ag_live_your_key_here"},
    timeout=200,  # the API holds slow scrapes up to 180s — stay above that
)
resp.raise_for_status()

answer = resp.json()["answers"][0]
print(answer["answerText"][:200])
for src in answer["sources"]:
    print(f"{src['position']:>2}. {src['url']}")

The response is one JSON object per run — answerText for mention counting, sources[] for citation counting, both in the same record:

{
  "id": "run_a41f0c9d2b7e",
  "query": "best time tracking software",
  "surfaces": ["perplexity"],
  "status": "completed",
  "recordsDelivered": 1,
  "creditsCharged": 1,
  "answers": [
    {
      "surfaceKey": "perplexity",
      "answerText": "The tools mentioned most often are Acme, Toggl and Northwind ...",
      "sources": [
        { "title": "Best Time Tracking Software 2026", "url": "https://acme.com/time-tracking", "position": 1 },
        { "title": "Toggl vs the field", "url": "https://www.toggl.com/compare", "position": 2 }
      ],
      "fetchedAt": "2026-09-28T09:12:44Z"
    }
  ]
}

creditsCharged counts delivered records, one here. position ranks within this one answer — first of two sources, not first across the run — so it only means something next to its own answer's source count. Keep that in mind before you trend positions across engines. The full field reference is in the docs.

The pipeline: query set to appended rows

Now the whole thing. Loop the query set across every surface you care about, normalize each source URL before counting — this is not optional, it's the difference between a right number and a wrong one — tally your brand against competitors, and append per-host rows to a CSV. Appending is deliberate: the file becomes a run history, and next month's diff is a filter on the fetchedAt column, not a fresh scrape.

#!/usr/bin/env python3
"""Citation share of voice across engines, from one query set. Append-and-diff."""
import csv
from collections import Counter, defaultdict
from datetime import datetime, timezone
from pathlib import Path
from urllib.parse import urlparse, urlunparse

import requests

API = "https://api.agentgeo.org/v1/fetches"
KEY = "ag_live_your_key_here"
OUT = Path("sov_rows.csv")
FIELDS = ["runAt", "surface", "query", "host", "citations", "answerSources"]

SURFACES = ["chatgpt", "perplexity", "google_ai_overview"]

# The query set IS the metric. Freeze it, version it, ship it with the number.
QUERIES = [
    "best time tracking software",
    "time tracking for agencies",
    "toggl alternatives",
    "software to track billable hours",
]

# Host -> the brand it belongs to. Everything else is "other" and still counted.
BRAND_HOSTS = {
    "acme.com": "Acme",
    "northwind.io": "Northwind",
    "toggl.com": "Toggl",
    "contoso.com": "Contoso",
}


def normalize(url):
    """Lowercase host, drop www., strip query string and trailing slash."""
    parts = urlparse(url)
    host = parts.netloc.lower().removeprefix("www.")  # Python 3.9+
    clean = urlunparse((parts.scheme, host, parts.path.rstrip("/"), "", "", ""))
    return host, clean


def fetch(query, surfaces):
    resp = requests.post(
        API,
        json={"query": query, "surfaces": surfaces},
        headers={"Authorization": f"Bearer {KEY}"},
        timeout=200,  # the API holds the request up to 180s — sit above it
    )
    resp.raise_for_status()
    return resp.json()


run_at = datetime.now(timezone.utc).isoformat(timespec="seconds")
rows = []
# citations[surface][host] = count of cited pages for that host on that surface
citations = defaultdict(Counter)
total = defaultdict(int)  # all cited pages per surface = the denominator

for query in QUERIES:
    run = fetch(query, SURFACES)
    for answer in run["answers"]:
        surface = answer["surfaceKey"]
        if answer.get("status") == "failed":
            print(f"failed record, skipping: {surface} / {query}")
            continue
        sources = answer.get("sources") or []
        n_sources = len(sources)
        per_query = Counter()
        for src in sources:
            host, _clean = normalize(src["url"])
            citations[surface][host] += 1
            total[surface] += 1
            per_query[host] += 1
        for host, count in per_query.items():
            rows.append({
                "runAt": run_at,
                "surface": surface,
                "query": query,
                "host": host,
                "citations": count,
                "answerSources": n_sources,
            })

new_file = not OUT.exists()
with OUT.open("a", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=FIELDS)
    if new_file:
        writer.writeheader()
    writer.writerows(rows)

print(f"appended {len(rows)} rows to {OUT} for run {run_at}\n")
for surface in SURFACES:
    denom = total[surface]
    if not denom:
        print(f"{surface}: no cited sources this run")
        continue
    print(f"citation share of voice — {surface} ({denom} cited pages)")
    for host, n in citations[surface].most_common(8):
        brand = BRAND_HOSTS.get(host, "other")
        print(f"  {n / denom:6.1%}  {host:<22} {n:>2}x  [{brand}]")
    print()

Read the shape of it. The denominator per surface is total[surface] — every cited page across the query set on that engine — and each host's share is its citations over that total. That's citation share of voice: your cited domains divided by all cited domains. Swap to mention share and only two things change — you count answers whose answerText names the brand, and the denominator becomes answers delivered, not cited pages. Same loop, different field. Never average the two together and call it one number.

One run is a sample, not a measurement

The single most important thing the pipeline gets right is that it appends. A single run of a query set is one draw from a distribution — the engines re-search on every ask and drift between them, so two runs an hour apart can cite different pages with nobody's content having changed. Perplexity drifts the most; ChatGPT sometimes doesn't browse at all, in which case its sources[] comes back empty, which is a real signal (it answered from parametric memory) and not a failure to record.

• Run the set repeatedly, then diff. Because rows carry runAt, next month's comparison is GROUP BY host filtered to two run windows. You never re-fetch history; you filter it.
• Require movement across several runs before calling it a trend. A host going from 12% to 15% in one run pair is noise until it holds across three or four. Treat single-run swings with suspicion, especially on Perplexity.
• Record the per-answer source count. The answerSources column is there so a position or a share means something — being one of four cited pages is a different outcome from being one of twelve, and only that column can tell them apart later.
• Missing AI Overviews aren't zero. Google AI Overview sometimes doesn't render for a query; that record comes back failed at zero credits. It's absent, not a brand scoring nothing — excluding it from the denominator is correct, counting it as a miss is not.

What breaks in production

• A failed record returns HTTP 200. status: "failed" rides inside a 200 response, so raise_for_status() sails right past it — the script checks answer.get("status") and skips, because a failed record is not "zero citations". When the failure was a scrape exceeding the 180-second budget, the record costs zero credits and carries providerFields.snapshot_id: a follow-up POST with a top-level "snapshot_id" and the same single surface redeems the finished answer instead of paying to re-scrape.
• A client timeout under 180 seconds cancels healthy work. The API holds the connection up to 180 seconds while a slow surface finishes, so timeout=30 gives up on answers that were about to land — and no timeout at all hangs forever on a dead socket. timeout=200 sits above the server's budget and below infinity.
• Wall-clock adds up before credits do. Each call can run to three minutes and the loop is sequential, so a 30-query set across three surfaces is a long job. Run it from cron or a worker, not inside a request handler, and make sure nothing upstream imposes a shorter timeout.
• Extra parameters imply behaviour you're not getting. web_search on a perplexity call, country on chatgpt — these don't error, they sit in the payload doing nothing. Only pass a parameter to the surface that honours it, or you'll debug a feature that was never on.
• A hardcoded brand map measures only the race you believe you're in. Every so often, read a handful of raw answers and look at the hosts landing in "other" — a challenger shows up there long before it shows up in your number. Add its domain to BRAND_HOSTS and note the date, because the shape of the table just changed.

Summary

Measuring AI share of voice without scraping is a managed fetch plus a tally you own: POST the query set, read answerText and sources[], normalize, count, append. No browser fleet, no rotting selectors, and a run history you can diff instead of re-collect. The pipeline here computes citation share; the share-of-voice formula covers the math — both denominators, position-weighting, and the sampling caveats — and the per-engine how-tos for Perplexity and ChatGPT go deep on each surface's quirks. Set it on a schedule and the number reports itself.