TL;DR
This checklist gives five reproducible self-check steps for AI indexing: verify robots.txt allows AI crawlers, sitemap.xml works, llms.txt exists, page JSON-LD structured data is complete, and hreflang/SSR are in place. Passing all five completes the basic setup for AI indexing.
When to use
Content-site operators who want their pages cited by generative AI (ChatGPT, Claude, Google AI Overview, etc.) should first confirm the site is accessible and parseable by AI crawlers. This checklist applies to any own website; all commands run directly in a macOS/Linux terminal.
Five-step checklist
Step 1: Confirm robots.txt allows AI crawlers
Run curl -s https://your-domain/robots.txt and check for Allow entries for common AI/search crawlers:
- GPTBot, ChatGPT-User, OAI-SearchBot (OpenAI)
- ClaudeBot, Claude-SearchBot (Anthropic)
- Google-Extended, Googlebot (Google)
- Bingbot, Microsoft-Copilot (Microsoft)
If missing, add User-agent: <name> and Allow: / entries for these crawlers.
Step 2: Confirm sitemap.xml is reachable and lists articles
Run curl -s https://your-domain/sitemap.xml and confirm it returns XML containing article URLs, with zh/en/ja alternate links for each. The sitemap is the entry point for search engines and AI crawlers.
Step 3: Confirm llms.txt exists
Run curl -s https://your-domain/llms.txt and confirm HTTP 200 with a site description and an article index (Markdown link list). llms.txt is a standard text index for LLMs and can be read directly.
Step 4: Confirm JSON-LD structured data on article pages
Fetch any article page source and check for a script tag of type application/ld+json; verify the @type matches the content:
- Knowledge articles:
Article - News:
NewsArticle - Data pages:
Dataset - Q&A pages:
FAQPage - Comparison pages: may add
ItemList
Structured data helps AI understand page type and cite key fields (headline, datePublished, etc.).
Step 5: Confirm hreflang, canonical and server-side rendering
Inspect the page source for <link rel="alternate" hreflang="..."> covering each language, <link rel="canonical" ...>, and whether the body text appears directly in HTML (not JS-rendered). Most AI crawlers do not execute JavaScript, so server-side output is a prerequisite for parseable content.
Notes
- Replace "your-domain" with the real domain before each step; this checklist assumes a publicly accessible site, not login-protected pages.
- Passing the checks only means content is technically indexable; it does not guarantee AI will cite it. Content quality and authority remain the core factors.
- All commands are read-only and do not modify server configuration; editing robots.txt etc. must be done server-side.
References
- llms.txt specification: https://llmstxt.org/
- Example: https://www.knowflowhub.com/robots.txt
- Example: https://www.knowflowhub.com/llms.txt