One AI answer is not a result: sampling and honest trendlines

AI assistants do not return the same answer every time. That single fact breaks most of the habits carried over from rank tracking, and it is why a screenshot of one answer is not evidence of anything.

Why is every answer a draw?

Because the assistant samples it. Treat each answer as one draw from a distribution: one draw tells you what was possible, many draws tell you what is likely. A visibility score should be computed over the draws, never over one — the same question asked tomorrow, with the same sources available, can come back with a different shortlist.

How often should each prompt be asked?

Often enough that a trend rests on many answers. Vidrys asks every tracked prompt again on every sweep, and computes each score over all the answers in the window you pick — 7, 30 or 90 days — never over a single sweep. How many answers that is depends on the plan:

Answers a month is prompts × assistants × sweeps. A plan that sweeps less often gets a wider interval around the same score, and Vidrys shows that width rather than hiding it.

Why does a score need an interval?

Because a score without one invites you to read noise as progress. Vidrys computes a 95% confidence interval by bootstrapping over prompts — the resampling method Bradley Efron introduced in 1979, applied to whole prompts rather than single answers, because the answers to one prompt move together — and treats a change between two scores as real only when it is larger than their combined uncertainty.

What happens when the scoring changes?

Scoring methods improve. When the formula, the judge or the model behind it changes, the score version changes with it, and scores from different versions are never trended together. Otherwise last quarter's report quietly becomes wrong. Vidrys stamps every score with the version that produced it, and its API returns that version beside each number.

When is a head-to-head worth reporting?

Only once you and the competitor have appeared in enough of the same answers for the comparison to mean anything. Vidrys waits for eight co-appearances before it reports one. Below that, a single shared answer could swing the comparison from one side to the other.

What should you ask any visibility vendor?

1. 1How many times is each prompt drawn per engine, per sweep?
2. 2Is there a confidence interval on the score, and is it shown?
3. 3What happens to a change that is smaller than the interval?
4. 4What happens to my history when the scoring method changes?