Ask It a Hundred Times: How to Audit Your AI Visibility Without Fooling Yourself

Otlume GEO series, part 3 of 10 · Level: Beginner to Intermediate

Rand Fishkin went into his most recent big study expecting to prove that AI visibility tracking was pointless.

He said so himself. SparkToro's stated hypothesis was that AI tools produce such randomized recommendation lists, and people phrase prompts so differently, that tracking a brand's visibility is a waste of time.

To test it, SparkToro and Patrick O'Donnell of Gumshoe.ai (an AI tracking company, which SparkToro disclosed) got 600 volunteers to run 12 prompts through ChatGPT, Claude and Google's AI a combined 2,961 times, then normalized every answer into an ordered list of brands.

The headline result is the one everyone quotes. There's less than a 1 in 100 chance that ChatGPT or Google's AI, asked about a hundred times, gives you the same list of brands in any two responses. For the same list in the same order, it's closer to 1 in 1,000. Even the length of the list wandered, sometimes two or three items, sometimes ten or more.

But here's the part people skip.

Fishkin also wrote that the work "largely disproven half the hypothesis." The rankings were noise. The visibility wasn't. Across many runs, how often a brand appeared at all turned out to be a reasonably meaningful number.

That's the whole method of an honest audit, in one sentence.

Don't look at positions. Count appearances across many runs.

In part 2 I described four gates, access, retrieval, selection, attribution. An audit is just checking each gate in an order that doesn't waste your time. I'll walk through how I'd do it. I'm not going to pretend this is the only way, and some of it is tedious.

The tedium is kind of the point. It's what stops you from fooling yourself.

1. Start with the gate you can check in five minutes, access.

Open yoursite.com/robots.txt in a browser. Look for lines naming GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, Bingbot. Then decide, on purpose, which ones you want. Remember from part 2 that training bots and search bots are separate. Blocking GPTBot keeps you out of OpenAI training. Blocking OAI-SearchBot keeps you out of ChatGPT search answers. Those are different decisions.

When I checked a few large publishers today, The New York Times and the BBC both disallow the search bots too, not only the training ones. For a major publisher with licensing deals and litigation, that's a strategy. For a twelve-person software company, it's probably an accident. If you find a blanket "block all AI" block you didn't consciously choose, that's your first fix, and it's the cheapest win in this whole series.

Then check what robots.txt can't show you. Some CDNs and hosting providers have their own "block AI bots" toggles. Google's AI features guide specifically says to make sure crawling is allowed "by any CDN or hosting infrastructure," not just robots.txt. If your important content only appears after heavy JavaScript runs, or sits behind a login or a cookie wall, a fetcher may get an empty page.

A simple test is to fetch the page with curl and see whether the text you care about is in the raw HTML.

2. Check whether anything is actually visiting.

This is where your server logs or analytics come in. You want to see whether the AI fetchers and crawlers from step 1 are requesting your pages, which pages, and how often.

Otlume's AI traffic feature does exactly this for sites built on it, it tells AI crawler requests apart from search crawler requests using User-Agent matching and shows top pages and trends. I want to quote how the product describes its own limits, because it's the right attitude for any log analysis. It says collection is "best-effort," that "User-Agents can be faked," and that "a request doesn't mean your content was indexed or cited."

Whatever tool you use, hold it to that. A visit from ChatGPT-User means someone's question caused a fetch of your page. It doesn't mean you were cited, and it definitely doesn't mean you were recommended.

3. Build a prompt set before you run a single prompt.

This is the step people skip, and it's the one that matters most.

Write down 20 to 50 questions, split into three groups. Branded questions mention you by name ("What does [brand] do?", "Is [brand] legit?", "[brand] pricing"). Category questions are what a stranger asks before they know you exist ("best [category] for [audience]", "how do I [solve the problem you solve]"). Comparison questions name you alongside alternatives ("[brand] vs [competitor]").

Then, and this is important, write each question several different ways. SparkToro also measured how differently people phrase prompts when they want the same thing. The average semantic similarity across human-written prompts with the same intent was 0.081, which is to say barely similar at all. People explain, add constraints, add their situation. If you only test the tidy keyword version of a question, you're testing a question almost nobody asks.

If you sell in more than one language, write separate sets. Don't just translate. A 2025 study from the University of Toronto (Chen et al.) found that some engines, ChatGPT in particular in their tests, cite almost entirely different websites when the same question is asked in a different language. I'll come back to that in part 8.

4. Run each prompt many times, on each engine you care about.

Fresh chat every time. Logged out if you can, or at least with memory off, because personalization skews results. Note the date, the engine, the mode (AI Overview versus AI Mode, ChatGPT with search versus without).

How many runs? I don't have a magic number and I'd distrust anyone who claims one. SparkToro's prompts were run roughly 60 to 100 times per platform. For a first audit, five runs per prompt per engine is a reasonable minimum that a human can actually do.

That's still a lot. 30 prompts, 3 engines, 5 runs is 450 answers. This is why tracking tools exist, and also why I'm wary of any tool that runs each prompt once a day and calls the result your "rank."

For each answer, record three things. Were you mentioned? Were you cited with a link? Was what it said about you correct?

5. Score appearance rate, not position.

For each prompt group, calculate how often you appeared, as a percentage of runs. Do the same for two or three competitors. That gives you a crude share of voice, which is the subject of part 7.

Don't average positions. SparkToro's data says positions are close to random from run to run. A brand that shows up first once and fifth four times isn't "rank 4.2." It's a brand that appears 100% of the time at an unstable position.

That's a good result, by the way.

6. Read the answers for errors, and trace them.

When an answer gets something wrong about you, a price, a feature, a founding year, a confusion with a different company, click the citations. Usually the wrong fact came from somewhere. An old press release, a stale directory listing, a review site that never updated, a forum post. Write it down.

That list of wrong sources is the most actionable output of the whole audit, because fixing them helps every engine at once.

7. Check the platforms' own reports, where they exist.

They mostly don't, but it's starting. In February 2026, Bing Webmaster Tools launched an AI Performance report in public preview that shows how often your pages are cited in Microsoft Copilot, Bing's AI summaries and some partner integrations. It reports total citations, average cited pages per day, page-level citation counts, and "grounding queries," the phrases the AI used when it retrieved your content. Bing notes that the grounding query data is a sample. Google Search Console includes AI Overviews and AI Mode in the overall "Web" performance data rather than splitting them out, according to Google's own documentation. So you can't isolate AI there.

If you do nothing else from this step, verify your site in Bing Webmaster Tools and look at the grounding queries. They show you which sub-questions you're being retrieved for, which connects straight back to the fan-out idea in part 2.

Now let me show you what this looks like when the result isn't flattering, because I think that's more useful than a success story.

I ran two prompts today through an AI-backed search assistant, the kind that writes an answer and cites sources. Branded first. "What is Otlume?" The answer was accurate, a no-code platform for agent-ready websites with structured data, sitemaps, robots.txt and llms.txt. All five sources it cited were otlume.com pages.

Then a category prompt. "Best platform to build an AI agent-ready personal website with llms.txt and JSON-LD for GEO." The answer recommended three other products and listed five sources.

Otlume wasn't among them.

Two runs is nowhere near enough to conclude anything statistically, I know. I'm showing it because the pattern is so common it's almost a law for new brands. Branded questions are answered from your own site, because your site is the main thing on the web that talks about you. Category questions are answered from whoever the wider web has written about.

Otlume launched its policies in October 2026 and has no third-party coverage I could find. Of course it doesn't show up for category questions yet. If I were auditing this properly I'd run that category prompt 20 different ways, 5 times each, on several engines, and I'd expect a number close to zero.

That number is the honest baseline. You improve from there.

I was tempted to leave this out. It's not a great look, an Otlume article showing Otlume missing. But the whole point of an audit is that it tells you something you'd rather not hear, and if I only showed you flattering checks you'd have no reason to trust the unflattering ones in your own data.

Let me also name the most common way audits go wrong, because I'd bet most of you have seen it.

Someone asks ChatGPT one question, gets a nice answer that mentions the brand, screenshots it, and it ends up in a slide deck as proof that "we're visible in AI." Or the opposite, one bad answer gets forwarded to leadership with a panicked subject line.

Both are the same mistake.

A single generative answer is one draw from a distribution. SparkToro's numbers say you'd essentially never get that exact answer again. Fishkin even warned that AI visibility "experts" could weaponize this, just as sketchy SEO salespeople did in decades past. Show a client the one run where they look bad, sell them a fix, show them a run where they look good.

The defense is boring and it works. Many prompts, many phrasings, many runs, recorded with dates.

There's a story from medical history I like here. In the 1830s the French physician Pierre-Charles-Alexandre Louis did something that was oddly radical at the time. Instead of trusting his impression of individual patients, he counted. He tallied outcomes for pneumonia patients bled early versus late, and the numbers suggested bloodletting did far less than doctors believed.

His "numerical method" annoyed plenty of colleagues who preferred clinical judgment. It was also right.

AI answers put every marketer in Louis's position whether they like it or not. You can't read meaning off a single answer any more than a doctor could read it off a single patient. You count.

One more practical note on choosing competitors, because it changes what the numbers mean. Pick the brands the AI engines actually name in your category, not the ones your sales team worries about. They're often different.

After a first round of category prompts, list every brand that appeared at least twice. Those are your real competitors in this channel, and some of them may be review sites or marketplaces rather than companies like yours. If a comparison site like a big review aggregator keeps appearing as the answer, that tells you something too. It tells you where the engine goes to decide, and that's a place you probably need to be. Part 6 is about exactly that.

So here's what you should have at the end of this audit. A robots.txt and CDN setup you chose on purpose. Some sense of which AI fetchers visit and what they read. A prompt set in three groups, in real human phrasing, in each language you sell in. Appearance rates for you and a few competitors on each engine. And a list of wrong facts with the sources they came from.

That's your map. In part 4 we start changing things, beginning with the pages themselves, and I'll show you why the most famous piece of GEO advice, "add statistics," is a lot less settled than it sounds.