Decide what visibility means for this audit
Start with a buyer decision you can describe in a sentence. A small accounting firm might want to know whether a business owner who asks about outsourced bookkeeping in Phoenix sees the firm among the choices. A software company might want to know whether a buyer who already knows its name receives accurate integration information before booking a demo. Those questions belong in different groups. The first tests discovery. The second tests whether people get dependable information about a known company. A model can do well on one and poorly on the other.
Write down three things before you run a prompt. Name the company or product you are looking for, describe the person and job behind the question, and choose the outcome that would matter to that person. For a shortlist question, the outcome may be whether the answer recommends your product as one of the options. For a support question, it may be whether the answer points to the right setup instructions and states the current requirement. A name in a passing footnote might count as a mention while doing little for either objective.
Keep four observations distinct. A mention is a clear reference to the intended company. An official link takes a reader to the company's website or another address you have approved under a written rule. A recommendation presents the company as a candidate for the specific job. A supported claim has evidence that actually establishes what the answer says. An answer can recommend a product, link to a review, and misstate the price in the same paragraph. One score would hide which part deserves attention.
Specify the unit you will report. The most straightforward unit is one completed answer to one exact question under one recorded configuration. If a question is asked on five dates, that is five observations for that question. It is still one question. If your report counts whether a company appeared at least once across five dates, label that result as question coverage over five checks. Readers should be able to tell what a numerator and denominator refer to without asking for the spreadsheet.
Also decide where the audit ends. You can observe responses shown to your test accounts. You usually cannot observe the exact number of people who made the same request, the distribution of versions across all users, or whether a particular answer caused a purchase. Lead and revenue data can answer a separate business question. Keep your AI answer audit focused on the experience you actually observed and the practical decisions the evidence supports.
Build a panel of real buyer questions
Ask the people who work on your product to list recurring decision points they already know about. Existing public support documentation, product comparison pages, and anonymized themes from your own sales notes can help. Never put names, private URLs, account identifiers, unpublished prices, or sensitive customer questions into a public AI service. Convert a question into a representative hypothetical when the details belong to a real person. Preserve the decision the person was trying to make.
Include a mix of discovery, comparison, fit, and fact questions. Discovery prompts should usually leave your brand out. Try a category with constraints that a buyer would actually name, such as a team size, location, tool, budget, or required capability. Comparison prompts should reflect choices buyers seriously consider. Fit prompts should test a specific condition, such as whether a tool works for a three-person agency. Use named questions to check facts you can verify, like the official setup guide or supported region.
Choose questions before seeing how your brand performs. Selecting only queries where you already appear will overstate discovery. Selecting only queries where competitors appear will make the same audit look worse. One practical way to avoid both errors is to give each intent group a fixed number of questions and keep a short note on why each was chosen. If there is little prior evidence about how buyers phrase a question, label it as exploratory and keep its results separate from the permanent panel until you decide to retain it.
Make every question specific enough to interpret. “What is the best platform?” can produce an impressive-looking shortlist that tells you little about your actual market. “What should a two-person PR team compare when choosing a tool to find relevant podcasts and send pitches?” gives the answer a defined job. Do not add constraints the buyer would never use simply because they make the model name your company. For a location-dependent service, record the market in the question or in the run configuration. For a global product, decide whether to test the same language in several markets or isolate one market first.
Save exact wording and version the panel. Give every prompt a stable identifier and keep the literal prompt next to it. A rewrite from “best tools for agencies” to “best affordable tools for agencies” introduces an additional criterion. Keep that change visible with a new version or a different identifier. Group wording variants if you want to test phrasing sensitivity. Avoid counting a dozen slight variants as a dozen unrelated buyer needs.
A manageable starting panel. Pick five discovery questions, five questions about fit, five comparisons, and five branded fact checks. Two views and two dates would produce 80 planned answers before retries. This is a workload example. Choose the sample size based on your decisions and the variation you can tolerate.
Record exactly how each answer was produced
One AI brand can offer more than one interface, model, and search configuration. An answer generated in a consumer chat with web search enabled can behave differently from an answer generated through an API, and the user interface may change the model without exposing a precise version. Record the service, interface, chosen model label, search mode that was shown, language, market, and check timestamp. Record what is unknown as unknown. A clean label is useful only if the label means the same thing next time.
If you use a chat interface, start each check in a fresh conversation. A model that has just discussed your company can carry that context into a follow-up even if the next question contains no brand name. Where you can, use a consistent personalization setting and account context. Document whether browsing was selected, whether the answer visibly used a source panel, and whether the interface switched modes during the response. Save the visible result you plan to evaluate. Backend guesses about which model was used are weak evidence.
Pick a test schedule that makes business sense and write it down. Comparing Monday's US answers to Friday's UK answers changes timing and market together. Periods of high news activity can also make results more changeable. A recurring comparison should hold those conditions steady wherever possible. A time-of-day pattern may genuinely matter for some topics, but treat it as a separate question and compare several matched times. An observed difference between two dates tells you the experience changed across those checks. It does not tell you which underlying event produced the change.
Use a short run log. Record whether an answer completed, whether the result can be retained under your data policy, and any incident that affects interpretation. A login challenge, browser issue, quota message, unsupported region, blocked source, or timeout goes into the log. Set a limited retry rule before running the batch. For example, retry a transport error once after a brief pause and retain the error as a separate record. If a completed answer disappoints you, that is a result rather than a reason to rerun the prompt until you see a nicer answer.
When your goal is to compare two search modes, run the same question in both modes during the same round. Retain the two answers as a pair. If you make the change by choosing a different product or model, describe it as a comparison of the two setups. People reading your report can then judge the trade-off you actually tested. Make a similar distinction when you are comparing two dates within a single mode. These are different questions and lead to different denominators.
Write rules for scoring and evidence
Decide in advance how reviewers will recognize the intended company. Use the product name, official domain, company name, and distinctive descriptors to disambiguate namesakes. A name match by itself is often insufficient. If the answer calls a marketing product a developer framework and sends the reader to a completely different provider, flag the identity as wrong or uncertain even when the spelling is identical. For the cases that are unclear, save the supporting sentence and mark the outcome for human review.
Choose what counts as an official link. A reasonable default is a direct link to the canonical company domain or a specified subdomain under its control. Record the final destination when the displayed link passes through a safe, expected tracking redirect. A link to a magazine story about your company may be valuable evidence, but that is a third-party link. Decide whether official public profiles on other hosts belong in a separate field. If one answer contains the same official URL three times, you can count one answer with an official link while preserving individual URLs for a source analysis.
For recommendations, score the actual role of the reference. A company listed as one of three options has a different role from a company mentioned only as a warning, an example of something unaffordable, or the vendor that another product replaces. You can store the displayed rank among eligible recommendations when the answer has a meaningful order. If the answer groups products by use case instead of ranking them, record the section and leave rank empty. Assigning positions from formatting alone creates precision the answer never supplied.
A citation audit is about the connection between a claim and the source attached to it. Split compound claims into parts. An answer may say a service has a free tier and integrates with a product. The same link can support the integration and leave the free tier unverified. Open the source and store a short relevant passage or the heading that contains the evidence. Dates and plan names matter. If the source is unavailable, label it unavailable. If the source discusses the topic generally, report partial support or unknown instead of treating any link as proof.
Before scaling up, give two reviewers a small shared set of tricky answers. Compare labels, then discuss cases with disagreement and revise the rubric. Use examples that include a name collision, a reference without a clickable link, a narrow third-party citation, and a mixed positive and negative recommendation. Reviewers should still preserve the initial independent labels when they adjudicate a disagreement. That lets you report what reliability looked like and decide which observations require a second reader.
Use denominators that match the question
Calculate a mention rate by dividing answers with a valid mention by completed answers assessed for that field. If your definition of presence combines a mention or official link, call the result brand presence and use that exact rule wherever you show the number. Calculate official-link rate the same way with the link flag. The two quantities can overlap, so adding them usually double counts some answers. When you want an exclusive breakdown, report mention only, link only, both, and neither under one denominator.
Suppose a round planned 50 answers and returned 45 responses you could score. The company was present in 18 of the completed answers. The observed presence rate was 18 out of 45, or 40%. The completion rate was 45 out of 50, or 90%. You still have five planned checks with unknown outcomes. If someone asks for a range over all 50 planned checks, the lowest possible presence is 18 out of 50, or 36%, and the highest is 23 out of 50, or 46%. Show the unknowns separately if a failed view behaves differently from the others.
Keep prompt and engine mix visible. Suppose an audit runs more named-brand questions after a site update than before. The pooled mention rate can rise because named prompts naturally draw the brand into the answer, even when every intent group performs the same. Give each group its own count and rate. If you need one number, use a stated weighting rule and the same weights across rounds. An equal-weight average across four equally relevant groups answers a different question from an answer-weighted rate dominated by whichever group had the most observations.
Distinguish position among appearances from coverage. A company that appears four times at positions one, three, four, and five has an average position of 3.25 among appearances. If the next round contains only one appearance and it is first, the average position improves to one while overall presence declines. Report both when rank matters. A top-three coverage rate can also be useful if you define whether the denominator is all eligible answers or only those containing the brand. Stick with one definition across the comparison.
Document how you aggregate repeat answers. If one question contributes six checks and another contributes one, an answer-weighted total emphasizes the first. You can first summarize within each question and then give questions equal weight when that better reflects your decision. Do not mix those estimates in one time series. Include the count of unique questions as well as completed observations so readers can tell how much variety the panel actually covered.
Compare changes using the same questions
A before-and-after comparison is easier to read when you can align the same question under the same configuration. Join the two rounds on the prompt version and all of the conditions that matter for your decision. Leave an observation unmatched if the date has no corresponding check or the configuration changed. Report how many planned pairs were usable, how many were missing, and whether failures were concentrated in one view. An average from all available answers on each date is still a valid summary of those available answers when labeled that way. It answers a different question from the matched panel.
For each matched pair, note whether the brand appeared both times, was absent both times, gained presence, or lost presence. The earlier presence count is present both times plus losses. The later count is present both times plus gains. To see how much movement took place, add gains and losses. To see the net change, subtract losses from gains. Dividing both results by the number of valid pairs lets you show changes on the same scale without losing the underlying counts.
| Outcome | Count |
|---|---|
| Present both times | 16 |
| Absent both times | 18 |
| Gained presence | 10 |
| Lost presence | 6 |
In the fictional table, presence rose from 22 of 50 to 26 of 50, or from 44% to 52%. That is eight percentage points and four more appearances on net. Sixteen of the 50 pairs changed, or 32%. The relatively quiet net figure masks 10 gains and six losses. The gains are worth inspecting for sources that appeared. The losses may show a segment where an important detail disappeared. Neither total identifies the cause on its own.
Repeat this logic for official links if links matter. A company can retain brand presence while an official link disappears and a citation to another publisher takes its place. Look at the four outcome flags side by side before deciding whether the next task is a discovery problem, source-quality problem, or factual correction. A set of answers that continues to link to an official page while misquoting it requires a different response from a set that can no longer fetch the page.
Choose a sensible pairing rule when you have more than two dates. Adjacent-date pairs tell you how often a question switches between successive checks. Comparing everything to the first date tells you whether later observations resemble that baseline. Counting all combinations can make one repeatedly tested prompt dominate the comparison. Match the rule to the decision and disclose it. If a question contributes to more than one pair, those pairs can be related. Avoid treating a large pair count by itself as proof of independent replication.
Distinguish a change you observed from its cause
A new product page, a pricing change, an engine update, and a new news story can all happen in the same week. When answers change after you publish, check exactly what changed in the answers and which visible sources appeared. A time sequence can suggest a hypothesis. It does not establish that the publication was responsible. If the change appears only in one mode or one question category, describe that scope rather than reporting an across-the-board improvement.
If you want a stronger test, choose a comparison group before making the change. For example, compare a group of matched questions about the updated product with another group tested on the same dates, under the same settings, where the relevant source stays stable. Check the groups on several dates before the update to see whether their trends were moving in similar directions. If one group was already rising and the other falling, the post-update difference has several plausible explanations. Look for changes in eligibility, prompt mix, prices, territories, and seasonal demand as well.
Suppose the updated group moves from 30% to 50% presence and the comparison group rises from 40% to 50%. The first group gained 20 percentage points. The second gained 10. The difference in changes is 10 points. That arithmetic is a useful description. A causal interpretation needs a reason that the groups would otherwise have followed a similar change, and the setup needs a way to account for competing explanations. In a small operational audit, it is often clearer to report the observed changes and the sources affected than to attach a precise claim about what your work caused.
Preserve dates and editable copies of changed pages. You want to know when you published, what a crawler could retrieve, when a new answer was observed, and which content was available during each check. If your CMS updated both a guide and the homepage at the same time, record both. Avoid making a second uncontrolled round of edits immediately after the first unless a factual error or business risk warrants immediate correction. When an urgent fix is necessary, explain it in the timeline instead of trying to preserve a clean experiment at a reader's expense.
For reporting, use verbs that fit the evidence. “Appeared in 26 of 50 matched answers after the update” says what you saw. “The answer began linking to the new page in four checks” names a verifiable event. “The page caused a 20% lift in AI visibility” requires a much stronger design and clarification of what “20%” means. Use percentage points for differences between rates. Use relative change only when helpful, and show both starting and ending rates so the calculation is transparent.
Read a real repeat-check study
The VisibilityOS repeat-check study illustrates why matching matters. We compared 817 answer pairs across 36 monitored questions, eight engine views, and three tracking configurations for PodPitch-owned websites. Each pair checked the same question on different dates within the recorded setup. Presence meant either a named brand or a link to the official domain. The sample describes monitored websites and questions. It was assembled from existing checks rather than designed as a random sample of all buyer searches.
Presence changed in 82 of the 817 pairs, or about 10%. The status was gained 45 times and lost 37 times, leaving 735 comparisons with the same status. The net balance was eight more presences on the later side of the pairs, less than one percentage point across the 817 pairs. A separate official-link flag changed in 79 comparisons. A pair can change on both flags. The published totals do not give the exact overlap, so an overall count of pairs that changed on either measure would require joint labels.
Engine views provide context. ChatGPT web had three presence changes in 107 comparisons. Grok had 21 in 81. Those are observed rates for the tested views and sample. The extent of date coverage differed, and some questions contributed more than one observation. You can use the table to see which groups were more variable in this particular dataset. A comparison designed to rank engines by stability would require coverage and sampling choices that support that claim.
Notice what happens if you only retain the published gains, losses, and combined unchanged count. You can calculate how often the status switched, and you can calculate the net difference across pairs. You need the number of unchanged pairs that were present both times to reconstruct the starting and ending rates. If 150 of the unchanged pairs were present in both checks, the two rates would be about 22.9% and 23.9%. If 400 were present in both checks, they would be about 53.5% and 54.5%. These are hypothetical splits of the same totals. Keep all four cells in your own audit so your reader can reconstruct both rates.
Use this example to decide how often to check a result that matters. If your company disappeared from a single recommendation, repeat the exact prompt and review the links before assuming the broader picture changed. If a model directs people to a substantially incorrect official page, save the original and investigate the source immediately. The appropriate threshold depends on how consequential the observed error is, how much corroboration you have, and how quickly you can correct the source.
Build a report someone else can audit
A concise report starts with the decision it supports. For example, “Should we update the primary setup guide for agencies that ask about the Slack integration?” gives the rest of the report a purpose. Follow with the exact observation, the number of completed and matched checks, and the slice of questions and settings you assessed. Include a brief explanation of how you scored presence or accuracy. A reader should be able to distinguish a global statement about an engine from a small observation made under a defined set of conditions.
Keep a compact summary with four rows that can be repeated each month. The first is coverage of unbranded discovery questions. The second is accuracy on branded questions with verified answers. The third is direct official links on questions where a buyer needs to act. The fourth is run quality, including missing responses and unmatched configurations. Pick one or two especially informative prompts as examples, use exact source excerpts when appropriate, and make sure the examples reflect the data rather than the most dramatic event.
What to save with a row
- The stable prompt identifier, exact prompt text, intent group, timestamp, engine view, model label, market, and language.
- The completed answer or a permissible evidence excerpt, the exact source URLs, and the rule used to match the brand identity.
- The status of mention, official link, recommendation, and each factual claim that matters to the selected decision.
- The reason for missing data, a reviewer note on uncertainty, and the paired observation identifier for a comparison.
For a priority issue, include a clear next action and an owner within your organization. “Update the setup document to state which plan includes CSV export, with an effective date” makes the diagnosis testable. “Improve AI visibility” is too broad to assign. Record the original URL and the expected change in an answer or source. When you revisit the issue, include the original observation, the later evidence, and the number of times the correction held.
Choose one consistent reporting cadence. You might report an operational observation quickly and reserve monthly comparisons for a broader review. Compare like periods when the underlying question mix is stable. If you add a new market or engine, mark that addition so the audience knows why the total grew. A panel with a stable core and a small rotating exploratory group often lets teams track trends and learn about new buyer questions without mixing the two.
Turn a dashboard change into a specific investigation
Imagine that a team sees its discovery score move from 46% to 38%. The first check is whether the metric and denominator stayed the same. If one round used only completed answers and the next treated timeouts as absence, rebuild the report before investigating search. If both rounds were scored consistently, inspect matched pairs and find which question groups contributed the losses. Do the losses cluster around one capability, region, or model? Do the same answers still include the name in a different context?
Next, examine the source paths. A report that says the company is absent may conceal a bad identity label if the product changed names or appears under its company name. In other cases the model mentions the right company but points to an unofficial site. Compare the final URL with your approved domains and check whether the text actually refers to the same product. Save the answer and a dated copy of the relevant page. If you are changing a canonical page, check that the real content and status on the public URL match what your own CMS preview shows.
Work from evidence you can reproduce. If an error is visible on the company site, correct it there. If the answer attributes a claim to a reviewer who never made it, you can document that mismatch and update your own material with a clear verified statement. If a deployment changed an address, test the old and new URLs using the public reader path. Repeat your original prompts on a later date. Record separately whether the target page is reachable, whether it is available in the relevant search tool, and whether it appears in a new response.
Sometimes the report reflects an actual shift in buyer fit. A competitor may now offer a feature the buyer asked for, or your product may only support a workflow on certain plans. Prioritizing accurate information in those cases helps people select the right option and makes the audit more useful. If leadership wants to improve presence for a category where the product lacks a required capability, the decision may belong in the product roadmap rather than a content edit. Keep the audit close to the buyer's objective.
Escalate when a wrong claim could cause harm or costs, even if it appears in just one answer. For lighter issues, several matched observations can help prioritize the work. Preserve failures and mixed results. An audit becomes more valuable when its readers know the team reports uncertainty, states what the sample covered, and distinguishes a clear fix from a promising idea.
Run your first month in four rounds of work
- Design the panel. Agree on the buyer question groups and the intended company identity. Choose one or two views that matter for the audience and a repeat interval your team can sustain. Save the exact questions, the definition of presence, and the rubric. Collect a few difficult examples to settle disagreement before recording a baseline.
- Run the baseline and repeat it. Use recorded conditions and capture source evidence. Review missing answers separately. Record any change in the model label or the behavior of the interface. Mark invalid and unmatched checks before you compute the measures. Ask a second reviewer to check a small set of ambiguous cases.
- Pick one high-value issue. Look for frequent, consequential issues with clear evidence. A basic page that misstates a product's availability deserves attention before a project to influence an uncertain ranking. Document the exact change you make, the URL, the effective date, and the audience the change is meant to help.
- Run the same panel again. Report matched outcomes and any changes to sources. Look for a repeated pattern for the affected questions and document what stayed uncertain. If an important outcome has yet to change, keep observing while deciding whether you need a new test or an independently verified source.
Give the process a simple stopping rule. For example, revisit a corrected fact until successive matched rounds show the updated source and the right statement for the questions that matter. A presence change on one low-priority question may require only a note. A recurring absence for an important buyer segment may deserve a controlled test with more prompts. These choices make the audit an operational tool. Each round should end with a decision, an owner, and a record you can compare next time.
Keep three artifacts together. Save the scorecard with counts and rules, the evidence log with URLs and timestamps, and the change log with what your team revised. At the next review, open all three before discussing the overall rate. A difference often becomes understandable once you can see the underlying questions, the exact sources, and the conditions of the checks. If those records are missing, invest in improving the next measurement before promising a large change on the strength of one number.
The public VisibilityOS audit worksheet provides an example of how to retain match keys and outcome labels. Adapt the structure to your own decision and data policy. Keep real buyer questions and private answer evidence in an approved workspace. Export just the counts that a broader audience needs, and include enough method to make them understandable.
Common measurement questions
How many prompts should I start with?
Use enough questions to cover the buyer decisions you actually care about and enough repeat checks to see how much an answer varies for the same question. A small panel you can review carefully is a sensible start. Expand the segments where decisions keep coming up or where the initial evidence is unstable. When presenting percentages, show the counts so people can see how much one change would affect the result. In a five-answer group, one different outcome moves the rate by 20 percentage points.
Should I include questions that mention my product?
Yes, when you want to check a named product's factual accuracy, source quality, or suitability for a buyer's stated needs. Keep those prompts in a labeled group. Discovery questions are most informative when the name is absent and a buyer's actual requirements lead the response. Mixing named and unnamed questions into a single unlabeled percentage makes both signals harder to interpret.
What if the tool will not show the model version?
Record the visible interface, mode, model label, and date. Leave the precise version unknown if it is unavailable. Keep those observations in a separate group if the label changes. You can still learn what users of that interface saw within your checks. Be precise about whether a trend reflects a stable configuration or a product interface whose underlying version can change.
Can a citation stand in for a mention?
Choose a definition based on the question you need to answer. A linked product can be reachable even when its name is weakly displayed. A named product can appear prominently with no route to the official website. Combining the two into a presence flag is useful for a broad exposure measure. Keeping the original flags makes it possible to distinguish discoverability from useful attribution.
What should I tell leadership next week?
Share one bounded observation and what you are doing about it. “We saw four losses on matched setup questions. Three later answers cited a stale guide with a wrong prerequisite. We updated the current help page and will rerun those same questions next week.” That description gives leadership an observed issue, a source, a change, and a follow-up. Make the same report when results are mixed so everyone can see the real state.