Start here
For Claude search visibility, review Claude-SearchBot and Claude-User. ClaudeBot is for potential training material. Keep those decisions separate, then build pages that answer the questions you want Claude to help buyers resolve.
A useful Claude audit starts with the actual answer: what does it claim, which sources support those claims, and where does your brand fit? The assistant’s writing style is not a reliable guide to its source-selection rules. A long article is not automatically more likely to be cited.
Use the right crawler controls
Anthropic’s crawler reference distinguishes potential training collection, search indexing and user-directed retrieval. Allowing ClaudeBot alone is not a substitute for reviewing the other two agents.
| Agent | Job | What to review |
|---|---|---|
| ClaudeBot | Potential training collection | Your training-data preference. |
| Claude-SearchBot | Search indexing | Access to the public pages you want discoverable. |
| Claude-User | Retrieval for user requests | Access when a user asks Claude to consult a page. |
Anthropic says these agents honor robots.txt. Inspect the rules on each relevant host, including documentation subdomains, and check whether your firewall returns a challenge. A robots.txt rule is a crawling instruction, not an authentication mechanism for private content.
Separate an access failure from a content gap
- 1.Open the public page without signing in. Confirm the important text is available, not hidden behind a session or an interaction.
- 2.Inspect robots.txt and the response your CDN returns to legitimate crawler traffic. Verify bot identity using provider guidance.
- 3.Ask a representative question in Claude with web search available. Save the answer and sources, including cases where your site is absent.
- 4.Read the cited sources to identify the missing information. Do not conclude that crawler access failed just because another source was selected.
Give a recommendation something to stand on
Choose a real constraint your buyer cares about. For a contract product, that might be whether external signers need accounts, whether a signature history can be exported, or which plan includes a feature. Answer it directly, explain the conditions and link to documentation. A model cannot establish a capability merely because your homepage calls the product enterprise-ready.
Worked example, not a ranking test
Before: “Secure contracts for modern teams.” After: a section titled “Can external clients sign without an account?” with the actual sign-in requirement, an example signing flow and a link to the product documentation. The second version resolves a concrete question; it is not a promise of improved citations.
Include limitations when they help someone choose correctly. Avoid made-up comparisons and unsupported superlatives. If a claim rests on a customer outcome, show the method and permission to use the evidence. Do not hide a qualification in a footnote that changes the meaning of the headline.
Read the surrounding claim, not just the citation
A citation can be attached to a sentence about a competitor or a limitation. Record the passage it supports. Distinguish a link to your documentation from an explicit recommendation of your product, and distinguish both from an incidental mention.
When an answer contains an error, trace it to the cited source before deciding what to update. Correcting an old help page and asking a review publisher to update a plan name are different tasks. Keep a record of the change and the date it became public.
Use comparable follow-up checks
Run the same prompts again with comparable settings. Keep Claude product results separate from API measurements, and do not silently replace failed answers with zero visibility. Repeated samples help characterize variability, but a few runs do not prove that an edit caused an improvement.
Put the findings to work
August connects visibility tracking with a reviewable next step: inspect the answers and citations, prepare a draft, publish through your own workflow, then check what changed. Copy the content or create a GitHub draft PR; you control what goes live. A follow-up comparison is evidence to examine, not proof that one edit caused a gain.
Get started with August. Compare the current plans: Starter includes 25 tracked prompts across one brand workspace; Pro includes 75 across three, with Google AI Overviews and AI Mode alongside the four core engines. Prompt allowances are shared across the account.
Related reading
Compare the ChatGPT access controls and Perplexity access controls before applying the same robots.txt policy to every service.
Questions, answered.
Is ClaudeBot the search crawler?
No. ClaudeBot collects potential training material. Claude-SearchBot and Claude-User have separate search and user-retrieval roles.
Should I make every article longer for Claude?
No. Add the detail a reader needs to verify the answer. Length alone is not an established citation requirement.
Why can a competitor appear when my site is accessible?
Access only makes retrieval possible. The question, selected sources and answer construction still determine what appears. Inspect the returned evidence before diagnosing the cause.
Find out what AI tells your customers.
Book a call and we’ll show you exactly what AI says about you.