Ask the same question twice and you can get two answers. How to measure AI visibility so a move on the chart is a move in the world.
AI assistants do not return the same answer every time. That single fact breaks most of the habits carried over from rank tracking, and it is why a screenshot of one answer is not evidence of anything.
Every answer is a draw
Treat each answer as one draw from a distribution. One draw tells you what was possible. Many draws tell you what is likely. A visibility score should be computed over the draws, never over one.
Draw more than once
Vidrys re-draws every prompt on every sweep, and on Grow and above draws each prompt twice per engine within a sweep. The second draw is what separates a real move from run-to-run variation.
Put an interval on the number
A score without an interval invites you to read noise as progress. Vidrys computes a confidence interval by bootstrapping over prompts, and treats a change between two scores as real only when it is larger than their combined uncertainty.
Version the scoring
Scoring methods improve. When the formula, the judge or the model behind it changes, the score version changes with it, and scores from different versions are never trended together. Otherwise last quarter's report quietly becomes wrong.
Wait for enough evidence before comparing
A head-to-head against a competitor needs you both to appear in enough of the same answers to mean anything. Vidrys waits for eight co-appearances before it reports one.
Questions to ask any visibility vendor
- 1How many times is each prompt drawn per engine, per sweep?
- 2Is there a confidence interval on the score, and is it shown?
- 3What happens to a change that is smaller than the interval?
- 4What happens to my history when the scoring method changes?