AI visibility tools: what they measure and miss

// In this article+

AI visibility tools measure how often a brand appears, gets cited, or contributes to a computed share of voice inside AI-generated answers from ChatGPT, Perplexity, Google's AI Overviews, and similar systems. Ahrefs, Semrush, and Profound each built one, and each pulls its number from a different prompt set, on a different schedule, using a different collection method: synthetic API prompts, licensed consumer-panel conversations, or scraped consumer sessions. Semrush's own AI Visibility Index research ↗︎ found that 45% of marketing leaders cannot accurately measure their brand's visibility inside AI-generated answers. That gap is the market this tool category exists to close, and it is also why two vendors can hand the same brand two different scores.

What do "AI visibility tools" actually measure?

The category tracks a handful of core metrics, and the vendors mostly agree on the names even when they disagree on the numbers.

Mentions. Ahrefs states in its help article on AI visibility metrics ↗︎ that "a single mention is counted when a brand appears at least once in an AI generated response"; it does not matter how many times the brand name appears within that response. One appearance and ten appearances in the same answer count the same.

Citations and found-in rate. A citation is a mention that comes with a link or an explicit source attribution back to the brand's own domain. Found-in rate is the share of tracked prompts where the brand shows up at all, cited or not.

Share of voice. Computed against named competitors across the same prompt set, share of voice is the metric most dashboards lead with, and it is also the one most sensitive to which prompts got run in the first place.

Impressions. A rougher proxy: an estimate of how many real user sessions might have surfaced the brand, built from the vendor's own model of query volume rather than from any single run.

Profound takes a different approach to all four metrics. Instead of writing its own prompts, it draws on "anonymized conversations from double opt-in consumer panels of real AI users", according to its announcement of the Profound Index ↗︎. The work of getting a brand surfaced or cited in an AI-generated answer goes by three names in the industry: generative engine optimization (GEO), answer engine optimization (AEO), and AI search optimization. They describe one discipline. This piece uses GEO for the rest of it.

Two tools rarely agree on a number for the same brand, because "mention," "citation," and "share of voice" are defined slightly differently by each vendor, and because the prompt set behind each number is different too. A brand scoring a high found-in rate on one dashboard and a low share of voice on another is not necessarily wrong on either count. Semrush says as much in its own documentation on where its AI visibility data comes from ↗︎: "AI search and LLM responses are fast-changing and highly personalized, which means no platform can provide exact numbers on visibility".

How do these tools actually collect their data?

Three genuinely different collection methods sit behind the same dashboard label. metehan.ai's methodology breakdown of AI visibility tools ↗︎ draws the first split, between synthetic prompts sent through a vendor's own API calls and sessions scraped from a chat interface's own consumer front end. A third method sits outside that split entirely: licensed conversations pulled from a real consumer panel, the model Profound uses instead of writing its own prompts (see below). Each method answers a slightly different question, and none of them answers the one a marketing lead usually assumes it does: what did my actual customer see.

The three vendors named above differ on where the prompts come from, how often they run, and what the number gives up in exchange.

Source: Ahrefs' Brand Radar methodology ↗︎, Semrush's AI visibility data documentation ↗︎ and AI Visibility Index release ↗︎, and Profound's announcement of the Profound Index ↗︎.

The practical trade-off: a synthetic-prompt tool controls exactly what gets asked, at the cost of the prompt possibly not matching real usage. A panel-based tool reflects real usage, at the cost of the sample being whatever the panel happened to ask. Ask a vendor directly which method produced the number on the screen before comparing it to a competitor's tool.

How often do they sample, and does that make the number reliable?

Collection method decides what gets asked. Sampling frequency decides how much a single number can be trusted.

Classic rank trackers were built for a deterministic system: a page held a position, and checking it once a day was enough to know where it stood. A language model does not hold a position. Ahrefs refreshes its prompt runs monthly and reports against a 90-day window, as its Brand Radar methodology ↗︎ explains. Semrush updates daily, according to its AI visibility data documentation ↗︎. Evertune, a rival vendor, claims in its ranking of AI visibility tools ↗︎ to run each prompt up to 100 times per model, against what it describes as the market norm: "Most GEO tools sample each prompt once a day".

That gap matters because a model is non-deterministic. The same prompt run twice can return two different answers, with different brands named in each. Kevin Indig's warning in Growth Memo's prompt tracking brief ↗︎ is the standard worth holding every vendor to: "AI Visibility is measurable. The mistake is running it like a rank tracker".

metehan.ai's methodology transparency checklist ↗︎ asks a version of the same question from the buyer's side: can you see the raw answers a run actually produced, or only the vendor's summary of them. "If you cannot see the underlying AI answers, you are being asked to take screenshots on faith".

What a single run tells you, and what it does not. One prompt, run once, tells you what one model said to one hypothetical user on one day. It does not tell you what that model says on a repeat run, what it says to a logged-in user with history, or what it says a week later. Treat a single-session score as a data point, not a verdict, and weight any vendor comparison by how many times a prompt actually got run, not just by how many prompts sit in the library. The same rule drives a weekly AI visibility check you can run by hand.

What do AI visibility tools consistently miss?

No tool, regardless of method or frequency, closes three specific gaps.

Personalization. Lily Ray documented a case where ChatGPT rewrote a generic query into a location-specific one based on her own account history, which changed which brands got recommended in the answer, according to PPC Land's report on personalization and LLM tracking ↗︎. "Personalization in LLMs definitely presents a major challenge to LLM tracking tools," as the same reporting puts it. A synthetic prompt run from a clean, logged-out session cannot reproduce what a returning customer with search and purchase history actually sees. The tool's prompt and the customer's prompt are, technically, different questions.

Free-tool blind spots. Search Console, Google Analytics 4, and Microsoft Clarity each cover a piece of the picture, and none covers all of it. tryaivo's review of the free AI visibility options ↗︎ is blunt about the limit: "No single tool shows the whole picture". Search Console shows which queries triggered an AI Overview click-through, but nothing from inside ChatGPT or Perplexity. GA4 can show landing pages from an AI-referral source once traffic arrives, but nothing about the mentions that never produced a click. Clarity shows session behavior after arrival, not what happened inside the model before the visit.

The default prompt set. Every paid tool ships with a starter prompt library, and every one of those libraries is generic. Aleyda Solis is direct about the risk in her guide to building a representative prompt library ↗︎: "Their default prompt sets won't automatically reflect your products, audiences, markets, competitors, constraints or business priorities". A dashboard populated with the wrong prompts still produces a confident-looking number. It is a confident number about the wrong question.

Which gaps still need a human to close?

A vendor dashboard answers one question: presence. It cannot answer two others that decide whether presence is worth anything. In her framework for measuring AI search presence and impact ↗︎, Aleyda Solis frames the three as separate layers, presence, readiness, and business impact, arguing that "a brand can now be surfaced, recommended, and materially influence a purchase decision in AI search without necessarily generating a click".

Presence is what every tool in this piece already measures: mentions, citations, share of voice.

Readiness is a separate audit a dashboard does not run. Solis lists ten characteristics a page needs to be a good candidate for citation in the first place: accessible, extractable, useful, fresh, differentiated, recognizable, consistent, corroborated, credible, and transactable. None of that shows up in a share-of-voice chart. A low presence score paired with a high readiness score points to a prompt-coverage problem. A low score on both points to a content problem that more prompt tracking will not fix.

Business impact is the layer furthest from any dashboard: whether a mention or a citation ever changed a purchase decision, which none of the tools in this piece claim to measure directly.

This is the audit work underneath any GEO program, and it is a large part of what an AI SEO agency does once the visibility number is on the screen: score the site against the readiness list, then decide whether the fix is more prompt coverage or more source-worthy content. Rebuilding the prompt library comes first, since a readiness score is only useful against prompts that reflect the business. Solis's own instruction, from her guide to building a prompt library ↗︎: "validate and customize the prompt library before using it for prioritization". The larger shift this tool category exists to track is laid out in GEO vs SEO: what changes when the engine answers.

What to do this week

1. Pull each shortlisted vendor's methodology page and check whether it answers metehan.ai's disclosure questions for AI visibility tools ↗︎: run frequency, model version, and whether raw AI answers are available to inspect.
2. Ask each vendor directly whether it uses synthetic API prompts, licensed consumer-panel data as the Profound Index ↗︎ does, or scraped consumer sessions, since the collection method changes the number on the screen.
3. Rebuild the tool's default prompt set from real products, audiences, and named competitors before using any of it to prioritize work, as Aleyda Solis's guide to building a prompt library ↗︎ advises.
4. Treat single-session results as directional, not final, given how personalization can change what a real user sees ↗︎, and ask whether the vendor supports persona or location variants.
5. Check what the dashboard cannot reach: Google Merchant Center's AI Performance Insights fields for ecommerce accounts, and GA4 landing pages from AI-referral traffic, as covered in Search Engine Roundtable's daily recap ↗︎.
6. Score the site against Aleyda Solis's AI search readiness list ↗︎, accessible, extractable, fresh, differentiated, so a low presence score points to a specific fix rather than just a lower number.

FAQ

Do AI visibility tools all report the same number for the same brand?

No. Different prompt sourcing, different sampling frequency, and different collection methods, synthetic API prompts, licensed consumer panels, or scraped sessions, produce different numbers for the same brand. Semrush states outright that "no platform can provide exact numbers on visibility", and Ahrefs, Semrush, and Profound build their prompt sets differently enough that a direct comparison rarely means what it looks like it means.

Can a free tool replace a paid AI visibility tracker?

Not on its own. Search Console, GA4, and Clarity each cover part of the picture, and none covers every engine or shows what happened before a click, if a click ever happened. tryaivo's review of the free options concludes "no single tool shows the whole picture". The free tools are worth layering under a paid one, not instead of it.

How many prompts do I need to track to get a reliable signal?

Raw prompt count matters less than run frequency and how representative the prompt set is. Evertune claims some vendors run each prompt up to 100 times per model, against a market norm it describes as once a day, and Kevin Indig's guidance is to build the set around real buyer journeys rather than run it like a rank tracker. A small, representative, frequently run prompt set beats a large, generic one run once.