ChatGPT scraping rate limits — and the way around them

If you're hitting a ChatGPT scraping rate limit, the honest answer is that the limit isn't the real problem — it's the first of a queue. A homegrown scraper that drives a headless browser against chatgpt.com will trip rate limiting, then Cloudflare's bot challenge, then a login wall, then a DOM rename that silently empties your selectors — and even when it works you're left extracting citations out of rendered HTML by hand. For a one-off pull, a browser script is fine. At any cadence — daily, across a query set, on a schedule — the maintenance curve turns vertical. This piece is honest about where DIY holds up and where a managed answers API, which returns answerText and a structured sources[] without a browser, is the cheaper path.

The short version: a ChatGPT scraping rate limit is a signal that you're on the retail surface, not an API — and the retail surface is defended, unstable, and unstructured on purpose. You can throttle, rotate, and re-selector your way past each obstacle for a while, but the obstacles are load-bearing for OpenAI and they change without notice. The way around the whole class of problem is to stop driving a browser: POST your query to https://api.agentgeo.org/v1/fetches with "surfaces": ["chatgpt"] and read back the answer text and citations as JSON. Below: why each DIY layer degrades, when it's still fine to scrape yourself, and the managed call that skips the fleet.

Why DIY ChatGPT scraping degrades

The rate limit is the symptom people notice first, but it's rarely the one that ends the project. A scraper pointed at a consumer web app inherits every defense that app runs, plus the instability of a UI that was never a contract. These stack:

• Rate limiting is per-IP and aggressive. The moment your request cadence looks non-human, you get throttled or 429'd. Rotating proxies buys headroom, but proxy pools that aren't flagged cost real money, and residential IPs get burned as fast as you cycle them. You are now maintaining an anti-detection budget, not a scraper.
• Cloudflare and bot detection escalate. Past the rate limit sits a managed challenge — JS execution, TLS fingerprinting, behavioural checks. Headless Chrome has a detectable fingerprint; you patch it, they update the check, you patch again. This is a cat-and-mouse loop you don't win, you only stay in.
• Login walls gate the good answers. Much of ChatGPT's browsing behaviour sits behind an authenticated session. Automating login means storing credentials, surviving MFA, and refreshing session tokens that expire — and a logged-in scraper that trips detection risks the account, not just the request.
• The DOM is not an API. Citations render as nested markup with class names that are hashed and rebuilt on deploy. Your selector matched last Tuesday and returns an empty list today, with no error — the scrape 'succeeds' and delivers nothing. You find out when the numbers look wrong, not when the code runs.
• Browser fleets are infrastructure you now own. One Playwright process per concurrent query, each eating 300–500 MB, each needing restart on crash, each needing a headless Chrome kept patched. A four-query dashboard becomes a fleet with a memory budget and an on-call rotation.

None of these is unbeatable in isolation. The trouble is that they're additive and they move independently — you can have the proxy rotation solved the week Cloudflare ships a new challenge, and the login flow solid the week the citation markup gets renamed. The steady state isn't 'working scraper.' It's 'scraper that works until the next change,' maintained forever.

When scraping it yourself is genuinely fine

This isn't an argument that you can never open a browser. For a one-off — you want to see what ChatGPT says about one query, once, today — a Playwright script is the right tool and this section is over. Copy the answer, read the citations off the screen, move on. The economics only invert when you need the pull to be repeatable.

Read the table as a threshold, not a verdict. Everything above the 'on a schedule' row is legitimately DIY territory. Below it, the thing you're maintaining stops being a scraper and starts being an evasion pipeline — and the cost of keeping it green outweighs the cost of the call that skips it.

The managed path: one POST, no browser

The alternative isn't a better scraper — it's not scraping. AgentGEO runs the browsing surface for you and hands back the finished answer as data: the full answerText plus a sources[] array where each entry is a title, a url and a position. No headless Chrome, no proxy pool, no selector that rots on the next deploy. The whole client is one HTTP call.

import requests

resp = requests.post(
    "https://api.agentgeo.org/v1/fetches",
    json={
        "query": "best project management tools for remote teams",
        "surfaces": ["chatgpt"],
        "web_search": True,  # chatgpt is the only surface that reads this
    },
    headers={"Authorization": "Bearer ag_live_your_key_here"},
    timeout=200,  # the API holds slow scrapes up to 180s - stay above that
)
resp.raise_for_status()

answer = resp.json()["answers"][0]
if answer.get("status") == "failed":
    raise SystemExit("record failed - skip or redeem, see failure modes")

print(answer["answerText"][:400], "...\n")
for src in answer.get("sources") or []:
    print(f"{src['position']:>2}. {src['url']}")

The same request works from the shell — useful for a one-line check or a cron entry that doesn't warrant a script. Swap ag_live_ for an ag_test_ key to dry-run the call at zero credits before you spend anything:

curl -sS https://api.agentgeo.org/v1/fetches \
  -H "Authorization: Bearer ag_live_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "best project management tools for remote teams",
    "surfaces": ["chatgpt"],
    "web_search": true
  }' \
  --max-time 200 \
  | jq '.answers[0] | {status, answerText: .answerText[0:200], sources}'

Two things the browser route can't hand you cleanly show up here for free. First, sources[] arrives ordered and parsed — position starts at 1 and is the closest thing to a rank, no HTML archaeology required. Second, an empty sources[] is a real answer, not a scrape failure: ChatGPT only cites when it browsed, so an empty list legitimately means 'didn't browse.' A scraper can't distinguish 'the citation block moved' from 'there was no citation block'; the API tells you which.

Failure modes the managed path still has

Not scraping removes the whole rate-limit-and-detection class of problem, but the network is still the network. Two failure modes remain, and both are cheap to handle:

• A client timeout below 180 seconds cancels healthy requests. The API holds the connection open for up to 180 seconds while a slow scrape finishes on its side, so timeout=30 gives up on answers that were about to arrive — and requests with no timeout at all hangs forever on a dead socket. timeout=200 sits above the server's budget and below infinity.
• A failed record is a 200, not an exception. A record can carry status: "failed" while the HTTP call returns 200 — raise_for_status() won't catch it, so check answer["status"] explicitly. When the failure was a scrape that exceeded the 180-second budget, the record costs zero credits and carries providerFields.snapshot_id; a follow-up POST with a top-level "snapshot_id" and the same single surface redeems the finished answer instead of paying to re-run it.
• An empty sources[] is not a failure. This is the signal, not the bug. ChatGPT cites only when it browsed; an answer produced from the model's own weights comes back with a full answerText and no sources. Record the zero and move on — retrying mostly reproduces it while billing another completed record.
• web_search nudges, it doesn't force. Sending web_search: true raises the odds the fetch browses, but a query the model is confident about may still answer from memory. That's a property of the query, not a knob you can crank — and it's worth knowing, because a query that never browses can't be won with a better page.

Summary

A ChatGPT scraping rate limit is the visible edge of a defended, unstable, unstructured surface — behind it sit Cloudflare, login walls, a shifting DOM, and a browser fleet you'd own forever. For a one-off pull, scrape it yourself and don't overthink it. For anything on a cadence, the maintenance curve turns vertical, and the way around is to stop driving a browser: one POST to the fetch endpoint returns answerText and a structured, ordered sources[], with an empty list carrying its own meaning. The same request shape covers the other five engines — see the Python API page for the multi-surface version, or the build-your-own scraper comparison for the full cost side-by-side.