4 things are stopping AI agents from reading example.com properly.

Why agents care. Agents check robots.txt before fetching. Without one, well-behaved crawlers guess, and some back off entirely.

Most crawlers treat a missing robots.txt as "everything allowed", but it costs you the chance to state your rules explicitly.

Fix. Add a robots.txt at your domain root listing which agents may crawl and where your sitemap lives.

Why agents care. This is the single most common reason a site is invisible to AI agents — a blanket disallow that nobody revisited.

Why agents care. llms.txt is an emerging convention: a Markdown index of your site written for language models, so an agent can orient itself without crawling every page.

This is a young convention, not a standard — its absence is not a defect. But it is cheap to add and it is the clearest way to tell an agent what your site is and where the important pages are.

Fix. Publish /llms.txt: a Markdown file with an H1 for your site, a short summary, and linked lists of your key pages.

Why agents care. A sitemap is the cheapest way for an agent to learn the shape of your site without following every link.

Fix. Generate a sitemap.xml and add a `Sitemap:` line to robots.txt pointing at it.

Why agents care. A feed or JSON endpoint lets an agent read your content without parsing a page layout, which is faster and far less error-prone.

Agents fall back to scraping the rendered page, which is brittle and loses structure.

Fix. If you publish a stream of content, expose a feed and link it with `<link rel="alternate" type="application/rss+xml">`.

Why agents care. Most agents read the HTML your server sends and never execute JavaScript. If the words only appear after hydration, the agent sees an empty page.

To an agent that does not execute JavaScript — which is most of them — this page is effectively blank. This is the highest-impact problem on this report.

Fix. Server-render or statically generate your content. In Next.js that means keeping page content in Server Components; in a SPA it means adding SSR or prerendering.

• Extracted text: “Example DomainThis domain is for use in documentation examples without needing permission. Avoid use in operations.Learn more…”

Why agents care. These two strings are what an agent quotes when it tells a user what your page is.

Fix. Write a specific title and a 120–160 character description that states what the page offers.

• <title> Example Domain

Why agents care. Schema.org JSON-LD is the one format every major agent already parses. It turns "some text on a page" into facts.

Agents have to infer everything about this page from prose.

Fix. Add a `<script type="application/ld+json">` block describing the page — Organization and WebSite at minimum, plus Product, Article or FAQPage as appropriate.

Why agents care. A <main> element tells an agent which part of the page is content and which is navigation and boilerplate.

The page uses generic <div> elements throughout.

Fix. Wrap the primary content in a <main> element so agents can skip navigation and boilerplate.

Why agents care. Open Graph tags are a reliable, low-effort summary of the page that many agents read before the body.

Fix. Add og:title, og:description, og:type and og:url to the page head.

Why agents care. Without a canonical, an agent citing your page may pick a tracking-parameter variant that later 404s.

Fix. Add `<link rel="canonical" href="…">` pointing at the preferred absolute URL for this page.

Why agents care. An agent filling a form matches your labels to the data it holds. An unlabelled input is a field it will skip or fill wrongly.

Why agents care. An agent choosing between links has only the link text to go on. “Click here” tells it nothing.

Fix. Replace “click here” and bare icon links with text that names the destination, or add an aria-label.

• “Learn more” → https://iana.org/domains/example

Why agents care. Alt text is the only way a text-based agent knows what an image contains.

Why agents care. Content Signals let you say what agents may do with your content — train on it, ground an answer in it, show it in search — instead of one blunt allow or block.

Why agents care. Markdown is far cheaper for an agent to read than HTML — no layout to strip, no markup filling its context window.

This is an emerging standard, so not having it is normal today. It is listed because adopting it early is cheap and puts you ahead of almost every other site.

Fix. Honour `Accept: text/markdown` by returning a Markdown rendering of the page.

Why agents care. Web Bot Auth lets an agent cryptographically prove who it is, so you can let real assistants through while still blocking scrapers impersonating them.

This is an emerging standard, so not having it is normal today. It is listed because adopting it early is cheap and puts you ahead of almost every other site. It matters most if you already block AI crawlers — it is how you let the legitimate ones back in.

Fix. Publish a JWKS directory at `/.well-known/http-message-signatures-directory`, or enable signed-agent verification at your CDN.

Why agents care. An MCP server card is how an agent discovers that your product exposes tools it can call, rather than a page it has to read.

This is an emerging standard, so not having it is normal today. It is listed because adopting it early is cheap and puts you ahead of almost every other site.

Fix. If your product has an API worth calling, publish an MCP server at `/.well-known/mcp.json` so agents can find and use it.

Why agents care. RFC 9727 gives agents one predictable place to find your APIs instead of guessing at paths or scraping docs.

This is an emerging standard, so not having it is normal today. It is listed because adopting it early is cheap and puts you ahead of almost every other site.

Fix. Publish an RFC 9727 catalogue linking your OpenAPI documents, so agents can discover your API without reading your docs site.

Why agents care. An A2A agent card tells other agents what yours can do, so your product can be delegated to rather than only browsed.

This is an emerging standard, so not having it is normal today. It is listed because adopting it early is cheap and puts you ahead of almost every other site.

Fix. If you expose agent-facing capabilities, describe them in an A2A agent card at `/.well-known/agent.json`.