# How MentionBeat Measures AI Visibility — Methodology

*Last updated: 2026-07-10. This page describes exactly how our numbers are produced,
including what we do **not** do. Every claim here corresponds to shipped, testable code.*

Most AI-visibility tools publish no methodology. We think the collection method, the
sampling design, and the statistics are the product — so here they are.

## 1. Collection: real engines, real answers

Every measurement response comes from the **real engine's own API**, called with your
tracked prompt, at a realistic sampling temperature. We do not infer AI answers from
Google rankings, and we never silently substitute one model for another.

| Engine | How we query it | Web-grounded? |
|---|---|---|
| ChatGPT | OpenAI API | Optional (OpenAI web search) |
| Claude | Anthropic API | Optional (Anthropic web-search tool) |
| Gemini | Google Gemini API | Optional (Google Search grounding) |
| Perplexity | Perplexity Sonar API | Always (natively grounded) |
| Grok | xAI API | Optional (Live Search over web + X) |
| DeepSeek | DeepSeek API | **No** — parametric only, and we refuse to label it otherwise |
| Google AI Overviews | SERP capture (SerpApi) | Always |
| Google AI Mode | SERP capture (SerpApi) | Always |
| Microsoft Copilot, Meta AI | **Not yet measured.** They expose no API or SERP surface; a browser-capture adapter is on the roadmap. We'd rather show a gap than simulated data. |

- **Grounded vs. parametric answers are measured and reported separately.** A brand can
  dominate a model's parametric memory yet vanish when the engine searches the live web
  (or vice versa) — collapsing the two hides the difference that matters.
- Citations are extracted from each provider's native metadata (Perplexity citation
  lists, OpenAI URL annotations, Gemini grounding chunks, xAI citations, SERP
  references) — not regex-guessed from answer text.
- Every call records cost, latency, token counts and the provider-reported model
  version, so engine-version drift is visible instead of mysterious.

## 2. Sampling: repetition and honesty about variance

LLMs are non-deterministic; a single query is an anecdote, not a measurement.

- Each (prompt × engine) cell runs **multiple repetitions** (default 5) at realistic
  temperature — never one-shot snapshots.
- Prompt sets are generated across an **intent taxonomy** (informational, commercial,
  comparison, recommendation, problem/solution), personas and locales, then human-approved.
- **Geographic sampling is honest about its mechanism.** A prompt's locale (e.g. `fi-FI`)
  geo-pins SERP engines natively (Google's own country/language parameters — mode
  `native`). Chat APIs cannot be geo-pinned without proxy infrastructure, so they receive
  an explicit, recorded location hint instead (mode `hinted`). Every response stores which
  mode produced it — we never present a hinted answer as a natively geo-sampled one.
  Tools that geo-sample via proxies don't tell you this distinction exists.
- **Anti-leakage design:** headline visibility uses only *unbranded* prompts — questions
  a real buyer would ask before knowing your name. Prompts that contain the brand are
  tracked, but reported separately and excluded from headline share-of-voice. Asking an
  engine "tell me about Acme" and counting the reply as visibility is grading your own exam.
- Interrupted runs resume idempotently; failed calls are retried and error rates are
  reported (a run with 30% provider errors says so on the report).

## 3. Scoring: a calibrated judge, not keyword matching

Responses are scored by an LLM judge that extracts brand mentions, position within the
answer, sentiment, recommendation strength, cited domains, and factual claims. Because
judges are also LLMs, we treat the judge itself as an instrument to calibrate:

- **Consensus mode:** the judge can run multiple independent passes per response with
  majority voting, recording the agreement fraction.
- **Human calibration:** judge output is validated against human-labeled gold sets;
  we compute Cohen's κ, sensitivity and specificity, and apply **Rogan–Gladen
  bias correction** to mention rates — so a judge that over- or under-detects mentions
  doesn't silently skew your trend.
- A deterministic literal-match safety net catches mentions the judge misses, with
  provenance recorded (you can always see *why* something was counted).

## 4. Statistics: confidence intervals or it didn't happen

- Headline metrics carry **95% confidence intervals from a cluster bootstrap (2,000
  resamples) that resamples prompts, not individual responses** — repetitions of the
  same prompt are correlated, and treating them as independent understates uncertainty
  (the standard mistake in this category).
- Small slices use Wilson intervals; every rate you see has an interval next to it.
- **Trend deltas are flagged "likely real" only when intervals don't overlap.** We would
  rather tell you "this wiggle is noise" than sell you a dashboard of noise.
- Share of voice comes in two variants: simple mention share, and position-weighted
  (a #1 recommendation counts more than a footnote), de-duplicated per response.
- Demand weighting: prompts can carry demand weights; weighted and unweighted metrics
  use the same bootstrap. Where weights are modeled estimates, they are labeled as such.
  Estimated weights are relative (0–100), built from a log-scaled search-demand proxy
  (keyword/Trends volume, optionally calibrated against your Search Console impressions)
  times mild intent-class priors (chat usage skews informational/comparison); prompts
  with no signal keep a uniform weight and are flagged — never an absolute-volume claim.
- A power calculator recommends repetition counts before you spend budget.

## 5. Accuracy intelligence: grounded in your approved facts

We don't just count mentions — we fact-check what engines *say* about you against your
**single source of truth (SSOT)**: a provenance-tracked store of approved product facts
where every fact carries its source and human-review status.

- Claims that **contradict** an approved fact are reported as hallucinations, bound to
  the specific fact they contradict — directly actionable.
- Claims we **can't verify** are reported separately (unverifiable ≠ false).
- Cited domains are classified own / competitor / third-party, mapping your off-site
  corroboration footprint.

## 6. Experiments: measured lift, not vibes

Content changes are evaluated as **baseline → treatment experiments**: per-metric lift
with confidence intervals, significant only when intervals separate. A **confound guard**
flags experiments where the model set, repetition count, profile version, or provider
model version changed mid-experiment — those results are marked confounded instead of
being quietly reported as wins.

## 7. Live / modeled honesty

The authoring studio can preview engine behavior through a persona simulation when no
API key is configured. Every such run is labeled **modeled** (or **mixed**), end to end.
Modeled data never appears in measurement reporting as if it were live. If a vendor key
is missing, you see a gap or a label — never an imitation passed off as the real thing.

## 8. Known limitations (the part most vendors skip)

1. **API vs. UI divergence.** We query engine APIs; consumer web UIs can route to
   different configurations. A browser-capture adapter with a published API-vs-UI
   reconciliation report is planned. Until then: our results measure the engines'
   API-served answers, stated plainly.
2. **No consumer panel.** We do not have clickstream/panel data of real user prompts;
   prompt sets are generated + human-curated, and demand weights are estimates unless
   you supply real volumes.
3. **Geographic sampling** is currently locale-of-prompt, not IP-of-origin;
   region-parameterized querying is in development (SERP engines already accept
   country/language parameters).
4. **SERP-captured engines** (AI Overviews, AI Mode) depend on the SERP provider's
   rendering; absence of an AI answer for a query is itself recorded as a data point.
5. **Non-determinism is managed, not eliminated.** Repetition + intervals bound it;
   they don't make an LLM deterministic. Treat point estimates as centers of intervals.

---

*Questions about the methodology, or want the statistical details? The measurement
engine's metrics module implements everything above; ask us and we'll show you the code.*