The alternative to scraping ChatGPT answers

Scraping ChatGPT answers with a headless browser works in the demo and fails as a system. The alternative is one authenticated POST: AgentGEO's answer-access layer returns the raw answer text, citations, sources, and provider metadata from ChatGPT, Perplexity, Google AI Overviews, Google AI Mode, Copilot, and Gemini — six engines that would otherwise mean five separate bots to babysit.

Credit where due: Playwright and Selenium are superb tools, and driving a chat UI with them really does work — once, on your laptop, this week. The first run was never the problem. The problem is that a consumer chat frontend is an adversarial, constantly shifting surface that was never meant to be an API, and a scraper inherits every one of its moods in production.

Why the scraping demo dies in production

Here is the life of a ChatGPT scraper after the demo ships. Sessions expire and logins start getting challenged. Anti-bot systems fingerprint the headless browser and serve CAPTCHAs. You add rotating proxies; accounts get banned anyway. The DOM selectors you targeted break on silent frontend deploys — no changelog, no warning, just empty strings arriving in your pipeline at 3am. The citations you actually came for have no stable structure to parse, because the UI renders them for human eyes. And the whole operation sits in a ToS gray area that one enforcement email can end.

Concretely, this sketch is the artifact you sign up to babysit — shown here as the burden it is, not as code to copy:

// A sketch of what you would maintain — every line is a failure mode
const browser = await chromium.launch({ proxy: nextProxy() }); // proxy pool: monthly bill
const page = await browser.newPage();
await page.goto("https://chatgpt.com/"); // session cookies expire without warning
await solveCaptcha(page); // third-party solver: per-solve fee, sometimes fails
await page.fill('[contenteditable="true"]', query); // selector broke twice this year
await page.keyboard.press("Enter");
await page.waitForSelector('[data-testid*="turn"]', { timeout: 60000 });
const answer = await page.innerText(".markdown"); // breaks on silent frontend deploys
const sources = parseCitationsSomehow(page); // there is no stable citation DOM
// …now multiply by six engines, plus retries, bans, and storage

Every commented line above is a real failure mode with a real on-call page attached. And it covers exactly one engine.

What the scraper actually costs

The scraper's sticker price is zero, which is exactly how it gets into the codebase. The real bill shows up later, as line items nobody put in the sprint:

• Proxy bills. Residential or ISP proxies to keep looking human — a recurring cost that scales with query volume.
• CAPTCHA solving. Per-solve fees, added latency, and a failure rate you do not control.
• Selector rewrites. Every silent UI deploy is potential parser surgery — unplanned work that preempts planned work.
• On-call. Chat frontends ship whenever they ship. Your pipeline breaks at 3am, and someone owns that page.
• Account churn. Bans mean fresh accounts, fresh sessions, fresh fingerprints — a treadmill, not a task.
• One engine is not six. Multiply all of the above by ChatGPT, Perplexity, Google AI Overviews, Google AI Mode, Copilot, and Gemini — each engine is its own bot with its own failure modes.

None of this appears in the estimate that got the scraper approved. All of it appears in the quarter after.

One authenticated POST instead

The replacement is deliberately boring: one endpoint, one bearer token, a JSON body naming the query and the surfaces. AgentGEO's managed-scraper engine, maintained for you does the fetching, and every engine comes back in the same schema.

curl -X POST https://api.agentgeo.org/v1/fetches \
  -H "Authorization: Bearer ag_live_..." \
  -H "Content-Type: application/json" \
  -d '{"query": "best noise cancelling headphones", "surfaces": ["chatgpt", "perplexity"]}'

# → {
#     "answers": [
#       { "surfaceKey": "chatgpt", "answerText": "...",
#         "sources": [{ "title": "...", "url": "https://...", "position": 1 }] },
#       { "surfaceKey": "perplexity", "answerText": "...", "sources": [ … ] }
#     ]
#   }

Full guides: curl and Python — or run one fetch with no signup in the raw-answer playground.

Scraping vs the answer-access layer at a glance

The honest grid. The scraper column describes the steady state, not the demo.

Everything the scraper was for, without the scraper

The point of scraping was never the browser — it was the data. The answer-access layer returns all of it:

• Six engines, one call: ChatGPT, Perplexity, Google AI Overviews, Google AI Mode, Copilot, and Gemini via the surfaces array.
• The full answer text each engine returned — the thing the scraper was inching toward through the DOM.
• Structured citations — sources[] with title, URL, and position; no DOM archaeology.
• Provider metadata — surfaceKey and providerFields, so every record is attributable.
• MCP-native — claude mcp add agentgeo -- npx -y agentgeo-mcp --key ag_live_... puts the data inside Claude Code, Cursor, Codex, or any MCP client.
• Usage billing — subscription plus overage under a spend cap, starting on a free tier with no credit card.

Retire the scraper

If you need one exotic surface for a one-off study, keep Playwright in the drawer — it is the right tool for that. If your product depends on a steady feed of AI answers with stable citations, stop maintaining a bot and make the POST.