Check AI answers for hallucinations about your brand

Checking AI answers for hallucinations about your brand means reading answerText — the prose an engine writes — against a list of facts you know to be true, and flagging where they diverge. Engines confidently state wrong pricing, invent features you don't ship, and compare you to competitors that folded years ago. None of that shows up in citations; it's in the sentences. So you POST your brand-intent queries to https://api.agentgeo.org/v1/fetches, pull answerText from each surface, and check it against a ground-truth fact list. Be clear about what this can and can't do. Software flags candidates — passages that mention a price, a plan name, a feature, or a competitor in a way that contradicts your facts. A human decides whether each candidate is actually a hallucination. There is no accuracy score to compute and no number to invent here; the honest deliverable is a shortlist of suspicious sentences a person reads. Below: the fact list, the fetch, a candidate-flagger, and a frank account of its limits.

AI engines hallucinate about brands the same way they hallucinate about everything else: fluently and with total confidence. They'll quote a price you never charged, describe an integration you don't offer, or tell a buyer you were "acquired last year" when you weren't. This is a real risk to monitor — not a metric to dress up with fabricated precision. The tool reads answerText across engines, matches it against facts you control, and hands a human a list of passages worth a second look. The human is the judge; the code is the net.

Hallucinations live in the text, not the citations

A citation tells you the engine linked a page. It says nothing about whether the sentence next to it is true. An engine can cite your real pricing page and still write "$49/month" when your plan is $29 — the citation is correct and the claim is wrong. So this check ignores sources[] entirely and reads answerText, because that's where a false claim about your brand actually appears. Three kinds show up repeatedly:

• Wrong pricing. A stale or invented number — a price from an old plan, a tier that never existed, a "free forever" claim you retired. Buyers act on these directly, which makes them the highest-priority class to catch.
• Invented features. A capability you don't ship, stated as fact — "Acme includes a built-in CRM" when it doesn't. Sends the wrong buyers to your door and the right ones away disappointed.
• Dead or wrong competitor comparisons. "Acme is a lighter alternative to a product that shut down two years ago," or a comparison to a company you've never competed with. Anchors your positioning to something false.

Build the ground-truth fact list

The flagger is only as good as the facts you give it. Write down what's true — the things engines get wrong — in a form the code can check: your real prices, your real plan names, features you do and don't have, and competitors a comparison should never invoke. This list is the whole intelligence of the check; the matching around it is mechanical.

# Ground truth about your brand. Keep it current — a stale fact list
# produces false flags, which trains people to ignore the tool.
FACTS = {
    # Prices that ARE correct. A price token in the answer that isn't
    # one of these is a candidate for review.
    "valid_prices": {"$29", "$29/mo", "$79", "$79/mo", "$0"},

    # Plan names you actually sell.
    "valid_plans": {"Starter", "Team", "Enterprise"},

    # Features you do NOT ship. If the answer says you do, flag it.
    "features_we_lack": ["built-in CRM", "phone support", "on-premise install"],

    # Companies a comparison should never invoke: defunct, or never a rival.
    "never_compare_to": ["Initech", "Umbrella Legacy", "OldCorp"],
}

# Your brand's names and aliases, so we only inspect sentences about YOU.
BRAND_ALIASES = ["Acme", "Acme Desk", "AcmeHQ"]

Fetch and flag candidates

The script fetches your brand-intent queries across the surfaces you track, isolates the sentences that mention your brand, and tests each against the fact list. It emits candidates — never verdicts. A candidate is "this sentence mentions a price that isn't one of yours" or "this sentence claims a feature you don't ship." Whether it's genuinely a hallucination is a call the reviewer makes with the sentence in front of them.

#!/usr/bin/env python3
"""Flag candidate false claims about your brand in AI answers, for human review."""
import csv
import re
from pathlib import Path

import requests

from facts import FACTS, BRAND_ALIASES

API = "https://api.agentgeo.org/v1/fetches"
KEY = "ag_live_your_key_here"
OUT = Path("hallucination_candidates.csv")
SURFACES = ["chatgpt", "perplexity", "google_ai_overview", "gemini", "copilot"]
FIELDS = ["query", "surface", "kind", "sentence", "detail", "fetchedAt"]

# Brand-intent queries: the questions where an engine talks ABOUT you.
QUERIES = [
    "how much does Acme cost",
    "what features does Acme have",
    "is Acme good for small teams",
    "Acme vs competitors",
]

BRAND_RE = re.compile(
    r"\b(?:" + "|".join(re.escape(a) for a in BRAND_ALIASES) + r")\b", re.I
)
PRICE_RE = re.compile(r"\$\d[\d,]*(?:\.\d+)?(?:/mo|/month|/yr)?", re.I)


def sentences_about_brand(text):
    """Rough split, then keep only sentences that name the brand."""
    for sent in re.split(r"(?<=[.!?])\s+", text):
        if BRAND_RE.search(sent):
            yield sent.strip()


def flag(sentence):
    """Yield (kind, detail) candidates. Presence != confirmed hallucination."""
    # Prices that aren't ones we charge.
    for price in PRICE_RE.findall(sentence):
        if price not in FACTS["valid_prices"]:
            yield "suspect_price", price
    # Features we don't ship, asserted about us.
    for feature in FACTS["features_we_lack"]:
        if re.search(r"\b" + re.escape(feature) + r"\b", sentence, re.I):
            yield "invented_feature", feature
    # Comparisons we should never appear in.
    for rival in FACTS["never_compare_to"]:
        if re.search(r"\b" + re.escape(rival) + r"\b", sentence, re.I):
            yield "dead_comparison", rival


def fetch(query):
    resp = requests.post(
        API,
        json={"query": query, "surfaces": SURFACES},
        headers={"Authorization": f"Bearer {KEY}"},
        timeout=200,  # the API holds slow scrapes up to 180s
    )
    resp.raise_for_status()
    return resp.json()


rows = []
for query in QUERIES:
    run = fetch(query)
    for answer in run["answers"]:
        if answer.get("status") == "failed":
            print(f"failed record, skipping: {answer['surfaceKey']} / {query}")
            continue
        surface = answer["surfaceKey"]
        text = answer.get("answerText") or ""
        for sent in sentences_about_brand(text):
            for kind, detail in flag(sent):
                rows.append({
                    "query": query,
                    "surface": surface,
                    "kind": kind,
                    "sentence": sent,
                    "detail": detail,
                    "fetchedAt": answer.get("fetchedAt", ""),
                })

new_file = not OUT.exists()
with OUT.open("a", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=FIELDS)
    if new_file:
        writer.writeheader()
    writer.writerows(rows)

print(f"appended {len(rows)} candidate(s) to {OUT} — all need human review\n")
for r in rows:
    print(f"  [{r['surface']}] {r['kind']} ({r['detail']})")
    print(f"      {r['sentence']}\n")

Every row is a candidate and every candidate needs a human. The sentence column exists precisely so the reviewer reads the claim in context before deciding — "$49" flagged as a suspect price might be the engine correctly quoting a competitor's plan in the same breath as yours, which is not a hallucination about you at all. The tool narrows a wall of prose to a handful of sentences worth reading. That's the job; it doesn't do the judging.

What this honestly can't do

A pattern-matcher over generated prose catches the hallucinations it was told to look for and misses the ones it wasn't. Being straight about the boundary is what keeps the tool trustworthy:

• It flags, it doesn't judge. Every candidate is a passage that might be false. Confirming it is a human read against reality. Treat the CSV as a review queue, not a verdict list — and never publish a count of "hallucinations found" as if the code decided.
• It only catches enumerated errors. The flagger knows the prices, features, and comparisons you wrote into facts.py. A wrong claim in a category you didn't anticipate — a fabricated founding date, an invented compliance certification — sails through. Grow the fact list as reviewers find new failure classes.
• Paraphrase evades exact matching. "Free for teams up to five" contradicts your pricing without containing a dollar sign, so PRICE_RE never sees it. String and regex matching catch the literal; genuinely fuzzy claims need a human reading the answer, or an LLM judge you supervise — and an LLM judge can itself hallucinate, so it never gets the last word.
• No accuracy number is real here. You can count candidates and count confirmed errors after review, but any "hallucination rate" is bounded by what your fact list covers and how well your matcher fires. Report what a human confirmed, on a defined query set, over a stated period — not a manufactured precision figure.

Summary

Hallucinations about your brand live in answerText, not in citations — so you build a ground-truth fact list, fetch your brand-intent queries across engines, isolate the sentences that name you, and flag anything that contradicts the facts. The output is a review queue of candidate false claims, each with the sentence attached, for a human to judge. No accuracy score is invented; the tool narrows the prose, the person decides, and a claim earns escalation only after it recurs across several runs. To watch how brands get named rather than misdescribed, see finding competitor mentions in AI answers; the Python API page covers the endpoint beyond this task.