search-visibility

{
  "name": "search-visibility",
  "displayName": "Search Visibility: SEO and AI Search",
  "description": "For website owners and marketers: turn Google Search Console data into a ranked list of SEO fixes, and check whether AI assistants can find, read and cite your site.",
  "version": "1.4.0",
  "author": {
    "name": "Ivan Misic",
    "url": "https://ivanmisic.net"
  },
  "homepage": "https://ivanmisic.net/toolshed/plugins/search-visibility",
  "repository": "https://github.com/imisic/claude-marketplace",
  "license": "MIT",
  "keywords": [
    "seo",
    "search-console",
    "ai-search",
    "seo-audit",
    "indexing",
    "structured-data",
    "organic-traffic",
    "content-strategy",
    "analytics"
  ],
  "documentationUrl": "https://github.com/imisic/claude-marketplace/tree/main/plugins/search-visibility#readme",
  "privacyPolicyUrl": "https://github.com/imisic/claude-marketplace/tree/main/plugins/search-visibility#privacy",
  "supportUrl": "https://github.com/imisic/claude-marketplace/issues",
  "termsOfServiceUrl": "https://github.com/imisic/claude-marketplace/blob/main/LICENSE"
}

# search-visibility changelog

## 1.4.0

`a-geo-optimizer` gains `scripts/geo_probe.py`, a deterministic first pass over robots.txt (per AI crawler), `llms.txt`, JSON-LD blocks and head tags. It reports evidence with a severity and exits non-zero when anything at or above `--severity` is found, so the audit starts from a candidate set instead of hand-fetching, and the model spends its reading on what a script cannot judge. Stdlib only, no network in its tests.

`a-geo-optimizer` no longer states citation percentages, multipliers or word-count quotas as rules; the figures had no named source. Its llms.txt guidance now matches what Lighthouse actually checks (an H1, at least one markdown link, at least 50 characters), and `geo_probe.py` reports those as `llms-format` and its own spec heuristic separately as `llms-spec`. `a-seo-gsc` describes page+query pairs as site-wide shared-query candidates to investigate, not proof of cannibalization, and asks for a same-intent check before recommending a merge.

## 1.3.2

Clearer name, description and keywords for the plugin directory. The README now opens by saying plainly what the plugin does and for whom, and its privacy section separates what the plugin does from how Claude itself handles your conversation. Skill descriptions reworded for accuracy; no behaviour changes.

## 1.3.1

Fixes that landed after 1.3.0 was first published went out under the same version number, so an install from that first day never received them: the guided API setup, the native Windows check, and calling the bundled scripts through `${CLAUDE_PLUGIN_ROOT}` so they work from your own project directory. This release carries no new behaviour; the new number is what makes those installs update.

## 1.3.0

`a-seo-gsc` can now pull Search Console data itself instead of asking you to download three zips. `scripts/fetch_gsc.py` authenticates with a Google service account and writes the same CSV shapes the manual export produces, so the parser reads either one and the export route still works unchanged. Setup is in `references/export-guide.md`.

The API is not just less clicking. The UI export caps its query and page tables near a thousand rows where the API returns up to 25,000 top rows per request, it writes a matching previous window so trend analysis runs without you remembering to tick Compare, and it returns page+query pairs. Google still does not guarantee every row through the API.

That last one adds an analysis the skill could not do before. No manual export tells you which of your pages a given query hit, so "which of my own pages compete for this term" was unanswerable. The parser now detects it, and flags the case worth acting on first: where the page Google shows most often is not the one that ranks best.

Coverage and Links still need a manual export. Neither has an API, and the guide now says so rather than leaving you to find out. The URL Inspection API is not a substitute for the coverage report, because it only answers for URLs you hand it and so can never surface index bloat from URLs you never submitted.

The script is standard library only and signs its request with openssl, so there is nothing to pip install. Your key is read from a path you choose and is never written to disk or sent anywhere but Google. Raw exports and arithmetic stay local; Claude receives the parser's findings in the session, and the plugin adds no telemetry.

If you want the API but have not set it up, the skill now walks you through it one step at a time and verifies at the end, rather than handing you a link. It also recognises the two 403s that point somewhere other than where they read: the one that means the API is switched off, and the one that means the service account is not on the property yet.

Installed copies now invoke the bundled fetcher and parser through `${CLAUDE_PLUGIN_ROOT}`, so they work from the customer's project directory instead of depending on the current working directory.

Setup now checks the environment before key creation. The bundled API fetcher needs Python 3 and `openssl` on macOS, Linux, or WSL; native Windows users get the WSL or manual-export choice before setup starts.

Downloading the reports by hand stays a first-class route, not a fallback. The skill opens by asking which one you are on and gives the full export list inline either way. The only thing the download route gives up is the page and query pairing, so that one analysis is skipped rather than reported as clean, and for a small site the API is often not worth setting up to get it.

## 1.2.2

The marketplace and storefront copy now describe AI-search work as an eligibility audit and fix plan, not a promise of citations.

## 1.2.1

The README now says which commands you get after installing, and the GEO acronym is spelled out where it first appears. Four hard-banned words swapped for plain ones in `a-geo-optimizer`'s reference docs.

## 1.2.0

The skills in this plugin now carry an `agents/openai.yaml` sidecar, so they show up in Codex's skill picker with a proper name and one-line description instead of a raw folder name. `a-seo-gsc` is explicit-only in Codex, `a-geo-optimizer` is not, matching how each behaves in Claude Code.

## 1.1.0

The two skills now differ in who can start them, because they are used at opposite moments.

`a-seo-gsc` is yours to invoke only. It is a session you sit down to run once a Search Console export is in hand, and nothing about a conversation tells Claude that has happened.

`a-geo-optimizer` still fires on its own. Its whole value is arriving while a page is being built, before anyone thinks to ask whether an assistant will be able to quote it.

The README table now names the mode per skill.

## Earlier

1.0.0 predates this file. See the git history at https://github.com/imisic/claude-marketplace.

# search-visibility

search-visibility helps website owners and marketers decide what to fix in Google search and get their site ready for AI assistants. `a-seo-gsc` turns your Google Search Console (GSC) data into a ranked action plan: which pages lost traffic, which searches you nearly rank for, and what is blocking indexing. `a-geo-optimizer` checks whether AI assistants such as ChatGPT, Claude and Perplexity can reach, read and cite your pages, and lists the fixes. Neither can guarantee rankings or citations.

Once it is listed, you can also add it from Anthropic's plugin directory: **Customize > Plugins** in claude.ai, or `/plugin` in Claude Code.

## Install

```
/plugin marketplace add imisic/claude-marketplace
/plugin install search-visibility@imisic
```

Then the commands are `/search-visibility:a-seo-gsc` and `/search-visibility:a-geo-optimizer`.

The longer write-up lives on the storefront: [ivanmisic.net/toolshed/plugins/search-visibility](https://ivanmisic.net/toolshed/plugins/search-visibility).

## The two skills

| Skill | Starts from | Answers | Invoked by |
|---|---|---|---|
| `a-seo-gsc` | Your Google Search Console data, over the API or exported | What is actually costing me traffic, and in what order do I fix it | you only |
| `a-geo-optimizer` | The site or page itself | Can an AI assistant reach this, parse it, and quote it | you or Claude |

`a-seo-gsc` is a session you sit down to run once your data is in hand, so Claude will not start it for you. `a-geo-optimizer` is the opposite: Claude can start it while you are building a page, which is when its fixes are cheapest.

### a-seo-gsc

Give it a property name and it produces a prioritized action plan instead of a wall of rows. It covers the diagnoses that matter and that people usually miss: pages with impressions but no clicks, striking-distance queries (searches where you rank just off page one, positions 11 to 20), keyword cannibalization where your own pages compete, indexing failures (`crawled - currently not indexed`, `discovered - currently not indexed`), and index bloat.

`scripts/fetch_gsc.py` pulls the Performance data over the Search Console API, so most of the downloading is gone. It is also more than the export gives you: the UI caps its query and page tables near a thousand rows where the API returns up to 25,000 top rows per request, it fetches a matching previous window so trend analysis always runs, and it returns **page+query pairs**. Google does not guarantee every row through the API, even with pagination. The standard export lists queries and pages separately, so "which of my own pages compete for this query" means one page-filtered export per page; the API answers it for the whole site at once. The parser now answers it, and flags the case worth fixing first, where the page Google shows most often is not the one that ranks best.

Coverage and Links still need a manual export, because neither has an API. The guide says which two reports to grab and why the URL Inspection API is not a substitute for the coverage one.

Prefer to keep downloading CSVs? That still works exactly as before. The fetcher writes Google's own export shapes, so the parser cannot tell the two apart.

It adapts to the site type, because a 300-page blog, a 50k-SKU store, a docs site, and a local business fail in different ways. `scripts/parse_gsc.py` pre-processes the data deterministically, so the numbers come from arithmetic rather than from a model's reading of a CSV.

`references/export-guide.md` covers the one-time API setup, the exact click-paths for the reports you still export by hand, and which alternative sources work (Bing Webmaster Tools, an Ahrefs or Semrush CSV).

For the same setup with an explanation of what each step does, read the [full Search Console API walkthrough](https://ivanmisic.net/blog/ai-tools/search-console-api-page-query-pairs). The installed guide remains complete and walks through the setup without depending on the article.

The bundled API fetcher needs Python 3 and `openssl` on macOS, Linux, or WSL. On native Windows, the skill offers WSL or the manual-export route before asking you to create a key.

The fetcher saves the raw exports locally and sends requests only to `oauth2.googleapis.com` and `www.googleapis.com`. The bundled parser computes the findings on your machine, then Claude reads those findings in the session under your Claude Code plan's data controls. The plugin adds no telemetry, and your service-account key never leaves your machine.

### a-geo-optimizer

Generative engine optimization, the AI-search counterpart to SEO. Being useful is not enough if the assistant cannot fetch the page, cannot tell what it says, or cannot find a quotable claim in it.

Covers crawler access, structured data, and content shaped so an answer can be lifted out of it. An audit starts with a bundled script, `geo_probe.py`, that checks robots.txt per AI crawler, `llms.txt`, JSON-LD and head tags, so the model spends its reading on what a script cannot judge. It recommends changes that improve the page's eligibility for citation, not a guaranteed result. Use it when building a page as well as when auditing one: the cheapest time to fix this is before publishing.

## Examples

- `/search-visibility:a-seo-gsc example.com: which pages lost clicks over the last three months, and what should I fix first?`
- `My Search Console exports are in ./gsc-exports. Build the prioritized action plan from them.`
- `Use a-geo-optimizer to check whether AI assistants can fetch, parse and cite example.com, and list the fixes.`

## Privacy

`a-seo-gsc` reads Search Console data for a property you own. On the API route, `fetch_gsc.py` signs a request with the service-account key file you provide and sends it only to Google (`oauth2.googleapis.com`, `www.googleapis.com`); the key is read from the path you give it and never written elsewhere. Exports are saved in your project and analysed on your machine. `a-geo-optimizer` fetches public pages of the site you ask it to check. The plugin adds no telemetry and sends nothing to its author. Your conversation with Claude is processed by Anthropic under its usual terms; that is separate from the plugin. Retention: the plugin's author never receives any of it, and the only thing kept is what the plugin writes in your own project, which you can delete at any time.

## Support

Questions, bugs or a security concern: open an issue at https://github.com/imisic/claude-marketplace/issues, or email hello@ivanmisic.net.

## Using them together

Run `a-seo-gsc` first when you have data, because it tells you where the loss actually is and stops you optimizing a page nobody was going to visit. Run `a-geo-optimizer` on a new site or page, where there is no data yet, and on anything you want quoted by an assistant.

`a-seo-gsc` only produces findings and a plan. `a-geo-optimizer` can also make the fixes it recommends (crawler rules, structured data, page structure) when you ask it to. Either way, you decide what lands.

---
name: a-geo-optimizer
description: "Optimize websites for AI search visibility (GEO/AEO) so ChatGPT, Claude, Gemini, Perplexity and Copilot can find and cite their pages. Use for AI SEO, GEO, AEO, generative or answer engine optimization, llms.txt, schema markup for AI, robots.txt for AI bots, WebMCP, agentic browsers and structured data audits for AI visibility, \"make my site AI-friendly\", \"optimize for ChatGPT\" or \"get cited by AI\". Also use when building a new site or page, even without an explicit GEO request. Covers crawler access, content structure and AI referral monitoring, not general copy editing. For your site's Search Console traffic analysis, use a-seo-gsc."
---

# GEO Optimizer: AI Search Visibility Skill

Make any web project discoverable and citable by AI assistants (ChatGPT, Claude, Gemini, Perplexity, Copilot). This skill covers technical implementation, content structure, and ongoing monitoring.

## When to use this skill

- **New site/page creation**: Bake GEO in from the start
- **Existing site audit**: Diagnose and fix AI visibility gaps
- **Content writing/restructuring**: Format content for AI citation
- **Template development**: Build PHP/HTML templates with GEO built-in
- **robots.txt / schema / sitemap work**: Ensure AI crawler compatibility
- **Competitive analysis**: Compare AI citation performance

## How to use this skill

### Step 1: Determine the task type

Ask yourself which mode applies:

| Mode | When | What to do |
|------|------|------------|
| **Full audit** | User wants the full review | Run all checklists from `references/technical-checklist.md` |
| **New build** | Creating a site/page from scratch | Follow the build-time integration guide below |
| **Content optimization** | Writing or restructuring content | Follow `references/content-strategy.md` |
| **Quick fix** | Specific technical task (robots.txt, schema, etc.) | Jump to relevant section in `references/technical-checklist.md` |

### Step 2: For full audits, follow this sequence

1. **Technical foundation**: Start with `python3 "${CLAUDE_SKILL_DIR}/scripts/geo_probe.py" <url> [more page URLs] --json`. It fetches robots.txt (per-crawler allow/disallow/unmentioned, Sitemap lines), `/llms.txt` and `/llms-full.txt` (presence, size, first heading, format), and every page's JSON-LD blocks (`@type` values, parse errors, missing template fields) plus `<title>`, meta description, canonical, `<html lang>` and a no-JS body-text heuristic. Its findings are evidence with a severity, never recommendations: judge that candidate set, then read `references/technical-checklist.md` and verify what the probe cannot see:
   - robots.txt allows AI crawlers (exact user-agents in `references/crawler-reference.md`)
   - Server-side rendering (critical: most AI crawlers don't execute JS)
   - Schema.org JSON-LD markup on key pages
   - Page speed (FCP < 0.4s target)
   - XML sitemap + RSS/Atom feed
   - HTTPS, mobile-friendly
   - llms.txt file: if present, it should pass Lighthouse's llms-txt audit, which checks three things: an H1 header, at least one markdown `[text](url)` link, and at least 50 characters (checklist section 5)
   - Agentic Browsing readiness (checklist section 13): run PageSpeed Insights and read the Agentic Browsing category; agent-critical a11y (programmatic names, valid ARIA, touch targets) and CLS now double as agent-readiness signals. WebMCP only matters for interactive sites (search, cart, booking)
   - If behind Cloudflare: fetch the LIVE robots.txt and check for injected Content-Signal directives (`ai-train=no` is the managed default; `ai-input=no` would hurt AI citation, see crawler-reference section 12)

2. **Content structure**: Read `references/content-strategy.md` and evaluate:
   - Answer-capsule format (40-60 word direct answer after each heading)
   - Front-loaded key content (the answer near the top, not after the background)
   - Sections sized to the question each one answers
   - Evidence where it helps the reader, with sources for any statistic
   - Question-style headings matching conversational queries
   - FAQ sections with real user questions
   - Author bios with credentials

3. **Entity and authority signals**: Check:
   - Organization schema with consistent NAP (Name, Address, Phone)
   - Author schema with credentials and external profile links
   - Third-party presence (review sites, directories, Reddit, YouTube)
   - Internal linking structure (pillar-cluster architecture)

4. **Monitoring setup**: Recommend:
   - GA4 AI traffic segment (regex for AI referral sources)
   - Monthly prompt testing across AI platforms
   - GEO monitoring tool appropriate to budget

### Step 3: For new builds, integrate GEO at template level

When building PHP/HTML templates, include these by default:

**In the `<head>` of every page:**
```html
<!-- JSON-LD schema block (populated per page type) -->
<script type="application/ld+json" id="schema-jsonld">
<!-- Populated server-side per template -->
</script>

<!-- Markdown alternate for long-form content (optional) -->
<link rel="alternate" type="text/markdown" href="<?= $markdown_url ?>" />

<!-- Standard meta for AI extraction -->
<meta name="description" content="<?= htmlspecialchars($meta_description) ?>" />
<meta name="author" content="<?= htmlspecialchars($author_name) ?>" />
<meta property="article:published_time" content="<?= $published_date ?>" />
<meta property="article:modified_time" content="<?= $modified_date ?>" />
```

**Content templates should enforce:**
- Single H1 per page
- Logical H2/H3 hierarchy
- First paragraph = direct answer to the page's core question
- Visible "Last Updated" date reflecting genuine content changes
- Author bio block with photo, credentials, external links

**Server-side requirements:**
- All content rendered server-side (SSR), not client-side JS
- JSON-LD generated server-side per page template
- Clean semantic HTML (proper heading hierarchy, lists, tables)
- Fast TTFB and FCP (< 0.4s target)

### Step 4: Output deliverables

Depending on the task, produce one or more of:

- **robots.txt**: Ready to deploy, with all AI crawlers allowed (see `references/crawler-reference.md` for exact user-agents)
- **llms.txt / llms-full.txt** (optional, low-medium priority): Markdown site overview, plus an optional full-content companion. As of 2026-07 adoption as a retrieval source is still uneven: AI dev tools (Claude Code, Cursor, Copilot, Windsurf, Cline, Aider) fetch both and some providers (Anthropic, Perplexity) confirm support, but Google doesn't use it for Search and no major provider has committed to it in production answer systems. What changed in 2026: Lighthouse/PageSpeed Insights now audits the file (Agentic Browsing category) for an H1, at least one markdown link and at least 50 characters, so if one exists it should pass those checks; write links as `- [name](url): note` bullets, never plain `Title: URL` lines. Generate it when trivial alongside other work, notably when the site already exposes per-page markdown, since a route can then concatenate it cheaply. Don't prioritize it over robots.txt, schema, sitemap, or content.
- **Schema JSON-LD blocks**: Per page type (Article, Product, FAQPage, Organization, etc.)
- **Content restructured**: In answer-capsule format with front-loaded answers
- **Audit report**: Findings and priority-ordered recommendations
- **PHP template code**: With GEO elements built into the template layer
- **Sitemap + RSS/Atom feed**: If missing or incomplete
- **GA4 configuration**: AI referral traffic tracking setup

## Key principles to always follow

1. **SSR is non-negotiable**: If content requires JavaScript to render, AI crawlers (except Googlebot) cannot see it. Always flag this.

2. **Front-load answers**: The most important content goes near the top of the page. AI systems tend to extract from the top.

3. **Be specific, not promotional**: AI systems filter out sales copy. Use definitive, factual language with specific numbers, dates, and names. Entity density matters.

4. **Earned media > owned content**: Assistants lean on third-party mentions, not just what a site says about itself. Recommend off-site authority building where the business can earn it.

5. **Freshness matters**: Recently updated pages are favoured, so recommend a refresh cycle for pages whose facts change, with real updates rather than date bumps.

6. **Cross-platform optimization**: Different assistants cite different sources. Optimize for all major AI systems, not just one.

7. **Don't block crawlers by accident**: Always check the LIVE robots.txt for inherited CMS/CDN blocks on AI user-agents, injected Content-Signal directives (Cloudflare's managed robots.txt defaults to `ai-train=no` on 3.8M+ domains), and CDN-level bot blocking that fires before robots.txt is read (test with `curl -A "GPTBot"`).

8. **Agent-readiness is the new layer**: Lighthouse's Agentic Browsing category (2026) grades llms.txt format, WebMCP tools, agent-critical accessibility, and CLS. Agentic browsers (OpenAI Atlas, Perplexity Comet, Gemini in Chrome) drive real sessions through the rendered page and a11y tree, so accessibility fixes now double as agent task-completion fixes.

## Reference files

- `references/technical-checklist.md`: Complete technical implementation checklist (robots.txt, schema, sitemap, llms.txt, page speed, rendering)
- `references/content-strategy.md`: Content formatting, answer-capsule structure, citation optimization, topic clustering
- `references/crawler-reference.md`: All AI crawler user-agents, which platforms they serve, robots.txt templates, and platform-specific submission guides
- `scripts/geo_probe.py`: deterministic first phase (robots.txt, llms.txt, JSON-LD, head tags), stdlib only; `--json` for phase two, exit 1 when any finding at or above `--severity` exists. The embedded crawler list mirrors `references/crawler-reference.md` sections 1 and 7, so update both together.

interface:
  display_name: "GEO Optimizer"
  short_description: "Make a page reachable and quotable by assistants"

# Content Strategy for AI Citations

How to structure, write, and maintain content that AI systems extract, cite, and recommend.

**How to read this file.** It gives direction, not measured effects. Earlier versions quoted citation percentages and multipliers from industry analyses that named no source; they were removed because a figure with no source and no stated limits cannot justify a rule. If you quote a number to a client or on a page, quote one you have checked against a named primary source, with its date, the platforms it measured and its limits.

## Table of Contents

1. [The Answer-Capsule Format](#1-the-answer-capsule-format)
2. [Front-Loading Critical Content](#2-front-loading-critical-content)
3. [Section Structure and Length](#3-section-structure-and-length)
4. [Heading Strategy](#4-heading-strategy)
5. [Statistical Density and Source Attribution](#5-statistical-density-and-source-attribution)
6. [Entity Density](#6-entity-density)
7. [Language and Tone](#7-language-and-tone)
8. [FAQ Sections](#8-faq-sections)
9. [Comparison Tables](#9-comparison-tables)
10. [Content Formats That Get Cited](#10-content-formats-that-get-cited)
11. [Author Authority Signals](#11-author-authority-signals)
12. [Content Freshness](#12-content-freshness)
13. [Topic Cluster Architecture](#13-topic-cluster-architecture)
14. [Off-Site Authority Building](#14-off-site-authority-building)
15. [Intent Modifier Phrases](#15-intent-modifier-phrases)
16. [Brand Sentiment Auditing](#16-brand-sentiment-auditing)
17. [What NOT to Do](#17-what-not-to-do)

---

## 1. The Answer-Capsule Format

The core content pattern for AI citation. Every major section should follow this structure:

```
[Question-style H2 heading]

[40-60 word direct answer, the "capsule"]

[Supporting detail, evidence, examples, 80-120 additional words]

[Optional: data table, comparison, or specific example]
```

**Example (good):**
```markdown
## How long does it take to set up a Synology NAS?

A basic Synology NAS setup takes 15-30 minutes for hardware assembly
and initial DSM installation. Full configuration including RAID setup,
shared folders, user accounts, and package installation typically
requires 1-2 hours. Advanced setups with Docker containers, reverse
proxies, and automated backups can take a full day.

The process breaks down into three phases. First, physical assembly:
insert drives, connect cables, power on. Second, DSM installation
via find.synology.com or Synology Assistant. Third, configuration
of storage pools, shared folders, and services.
```

**Example (bad, buries the answer):**
```markdown
## NAS Setup Guide

Network Attached Storage devices have become increasingly popular
in recent years. There are many brands to choose from, each with
their own strengths. In this guide, we'll walk through everything
you need to know about setting up your device. But first, let's
understand what a NAS actually is...

[answer appears 500 words later]
```

---

## 2. Front-Loading Critical Content

**Put the answer near the start of the page.** Readers and AI systems both meet the opening first; background and method can follow.

Rules:
- The most important, quotable, citable content goes at the top
- First paragraph after H1 should answer the page's core question
- No lengthy introductions, no "in this article we will discuss..."
- Get to the point immediately, then expand

For product/service pages: Lead with what it is, who it's for, and what problem it solves. Pricing and key differentiators in the first 3 paragraphs.

For blog posts: Lead with the answer or key finding. Background and methodology come after.

---

## 3. Section Structure and Length

**Size each section to the question it answers.** Give a distinct question its own section, long enough to answer it completely and short enough that the answer is not buried.

Each section should be:
- Self-contained (extractable without needing context from other sections)
- Focused on one specific sub-question
- Bounded by a descriptive heading
- Complete enough to stand alone as an answer

Articles over 2,900 words average 5.1 citations vs 3.2 for content under 800 words. Longer is better, but only if each section adds genuine value.

---

## 4. Heading Strategy

Use question-style headings that mirror how people ask AI assistants.

**Good headings (match conversational queries):**
- "How does [X] compare to [Y]?"
- "What are the pricing options for [service]?"
- "How long does [process] take?"
- "What's the best [tool] for [use case]?"
- "Is [product] worth it in 2026?"

**Bad headings (traditional SEO style, hard for AI to map to queries):**
- "Features"
- "Overview"
- "Technical Specifications"
- "Our Approach"
- "More Information"

Also: ChatGPT internally appends intent modifiers like "detailed guide", "step-by-step tutorial", "in-depth comparison" to its search queries. Including these phrases naturally in your headings or opening paragraphs can improve retrieval.

---

## 5. Statistical Density and Source Attribution

**Include evidence where it helps the reader.** The original GEO research (the 2023 paper "GEO: Generative Engine Optimization") found that adding cited sources, quotations and statistics improved visibility on its own benchmark; treat that as direction, not a guarantee for any particular assistant.

Rules:
- Use a statistic when it is the clearest evidence for a claim, not to meet a quota
- Always attribute statistics to their source
- Use specific numbers, not vague claims ("3.2× faster" not "much faster")
- Include dates with stats to signal freshness
- Prefer original data you've collected over recycled third-party stats

**Good (hypothetical figures, use your own sourced data):** "According to our 2026 client survey, 73% of small businesses in Croatia report spending less than 2 hours per week on social media marketing."

**Bad:** "Many businesses find social media marketing time-consuming."

---

## 6. Entity Density

**Name things specifically.** Specific names, dates, places and products make a passage easier to understand and to quote than vague references.

Entities = proper nouns, specific names, dates, places, product names, company names, standards, technologies.

- Use full, specific names on first mention (not just pronouns)
- Include dates, version numbers, specific locations
- Reference specific standards, frameworks, or methodologies by name
- Mention specific tools, platforms, or competitors by name when relevant

**High entity density (hypothetical figures):** "Zagreb-based agencies using Figma and Webflow in 2026 report 40% faster delivery times compared to agencies still using WordPress page builders."

**Low entity density:** "Agencies using modern design tools report faster delivery times than those using older platforms."

---

## 7. Language and Tone

AI systems prefer **definitive, authoritative language** over hedging.

**Preferred:**
- "The best approach is..."
- "This costs €500-800 per month"
- "Setup takes 2-3 business days"
- "The main drawback is..."

**Avoided by AI systems:**
- "It depends on various factors..."
- "There are many considerations to keep in mind..."
- "Results may vary..."
- "Contact us to learn more" (dead-end for AI)

**Critical:** AI systems filter out overtly promotional or sales-heavy content. Write as a knowledgeable advisor, not a salesperson. Neutral, factual language with specific claims wins citations.

---

## 8. FAQ Sections

FAQ sections with question-style headings nearly double citation chances. AI systems extract Q&A pairs directly, especially when paired with FAQPage schema.

**Requirements:**
- Use actual questions your customers/readers ask
- Match the conversational phrasing people use with AI ("How do I..." not "Methodology for...")
- Each answer direct and complete, as long as the question needs
- Include the questions readers actually ask, not a set number
- Pair with FAQPage JSON-LD schema (see technical-checklist.md)

**Good FAQ structure:**
```markdown
## Frequently Asked Questions

### How much does a website redesign cost in Croatia?

A professional website redesign in Croatia typically costs €2,000-8,000
for small business sites, €8,000-25,000 for mid-size corporate sites,
and €25,000+ for complex e-commerce platforms. Pricing depends on page
count, custom functionality, and whether content migration is included.

### How long does a typical web project take?

Most small business websites take 4-8 weeks from kickoff to launch.
Mid-size projects run 8-16 weeks. Enterprise-level builds with custom
integrations can extend to 3-6 months.
```

---

## 9. Comparison Tables

**Use a table when readers compare options.** A table states the relationships between items more clearly than prose.

Use HTML tables or Markdown tables for:
- Product vs. competitor comparisons
- Feature comparisons across tiers/plans
- Pros vs. cons
- Technology comparisons
- Before/after scenarios

Keep tables clean and data-rich. AI systems extract structured relationships from tables better than from prose.

---

## 10. Content Formats That Get Cited

Formats that answer a clear question tend to be quoted:

| Format | Notes |
|--------|-------|
| "Best X" lists | Useful when the list is honest about criteria |
| How-to guides | Especially with step-by-step structure |
| Comparison articles | X vs Y format |
| Definitions and explainers | "What is X" format |
| Case studies | With specific metrics and outcomes |

Pick the format that fits what your readers are trying to decide, not the one assumed to be cited most.

---

## 11. Author Authority Signals

Cited pages average 4.1 expert quotations vs 2.4 on non-cited pages.

For every content page, include:
- Author name and photo
- Professional title/role
- Years of experience or credentials
- Links to LinkedIn, professional profiles, or personal site
- Brief description of relevant expertise

For the Organization/brand:
- Clear "About Us" page with founding date, team, and mission
- Google Business Profile (fully optimized)
- Consistent Name, Address, Phone (NAP) across all platforms
- Wikidata entry (for entity recognition by AI)

---

## 12. Content Freshness

**Keep pages current when their facts change.**

Rules:
- Review key pages on a schedule that matches how fast their facts change
- Update with substantive changes (not just timestamp bumps)
- A changed date with no changed content misleads readers and earns nothing
- Display a visible "Last Updated: [date]" on the page
- The `dateModified` in your Article schema must match the visible date

---

## 13. Topic Cluster Architecture

**Group related pages into clusters.** A hub page that links its detailed pages, and links back from them, makes the whole topic easier to follow and to crawl.

Structure:
```
Pillar Page: "Complete Guide to [Topic]" (2,000-4,000 words)
├── Cluster: "How to [subtopic A]" (1,200-2,000 words)
├── Cluster: "Best [tools] for [subtopic B]" (1,500-2,500 words)
├── Cluster: "[Topic] vs [Alternative]" (1,000-1,500 words)
├── Cluster: "[Topic] for [specific audience]" (1,200-2,000 words)
└── Cluster: "Common [topic] mistakes and how to fix them" (1,000-1,500 words)
```

Rules:
- Pillar links to all cluster pages
- Each cluster page links back to pillar
- Cluster pages cross-link to related clusters
- Use descriptive anchor text

---

## 14. Off-Site Authority Building

**Earn mentions elsewhere.** Assistants draw on what other sites say about a brand, not only on the brand's own pages.

Priority platforms for AI citations:
1. **YouTube**: demo videos, tutorials, case studies.
2. **Reddit**: build an authentic presence in relevant subreddits.
3. **Review platforms**: G2, Clutch, Capterra, Trustpilot.
4. **Industry comparison articles**: Get listed in "best [service] in [location/niche]" articles.
5. **Guest posts**: Contribute to industry publications with expert commentary.
6. **Wikipedia/Wikidata**: Create or update entity entries for your brand/key people.
7. **Google Business Profile**: Primary data source for local AI recommendations.

Active profiles on the platforms your audience actually uses matter more than a presence everywhere.

---

## 15. Intent Modifier Phrases

ChatGPT appends modifier phrases to internal search queries when users ask questions. Including these naturally in your content improves retrieval:

- "detailed guide"
- "step-by-step tutorial"
- "in-depth comparison"
- "comprehensive overview"
- "complete guide"
- "how to [specific task]"
- "best [thing] for [use case]"
- "[thing] vs [alternative]"
- "pricing", "cost", "how much"
- "pros and cons"
- "review" / "honest review"

Don't stuff these artificially. Use them in headings and opening paragraphs where they fit naturally.

---

## 16. Brand Sentiment Auditing

Regularly test what AI tools say about your brand:

1. Ask each major AI tool: "Give me pros and cons of [Your Brand]"
2. Ask: "What are the best [your service type] companies in [your location]?"
3. Ask: "Compare [Your Brand] vs [Competitor]"
4. Document what's accurate, what's wrong, what's missing

If AI is hallucinating facts about you:
- Update your About page with clear, declarative sentences
- Update llms.txt with correct information
- Add specific factual claims to schema markup
- Build corrective third-party content (reviews, profiles, mentions)

AI systems pick up corrections faster from multiple independent sources than from your own site alone.

---

## 17. What NOT to Do

**Content that gets ignored or filtered by AI:**

- **Sales copy and promotional language**: AI filters out overtly commercial content
- **Vague, hedging language**: "It depends" answers get skipped in favor of specific ones
- **Thin content under 500 words**: Rarely provides enough value for citation
- **AI-generated content at scale without editorial enhancement**: Produces patterns AI learns to skip
- **Keyword-stuffed content**: The original GEO research showed keyword stuffing produced zero gains
- **Content with no original information**: If AI can generate the same content itself, it won't cite you
- **Content behind login walls**: AI crawlers cannot access paywalled content
- **Information scattered across multiple thin pages**: Consolidate into one page that covers the whole topic
- **Timestamp-only refreshes**: Changing the date without substantive updates doesn't fool AI systems
- **Ignoring a platform**: different assistants cite different sources; optimize for all

**The Chegg lesson:** If your content is commodity information that AI can generate independently, you'll be displaced. Content must include what AI cannot replicate: original data, proprietary research, real case studies with specific metrics, expert quotes from named professionals, and first-person experience.

# AI Crawler Reference

Complete reference of AI crawler user-agents, which platforms they serve, and how to configure access.

## Table of Contents

1. [Crawler Quick Reference Table](#1-crawler-quick-reference-table)
2. [OpenAI Crawlers](#2-openai-crawlers)
3. [Anthropic (Claude) Crawlers](#3-anthropic-claude-crawlers)
4. [Google AI Crawlers](#4-google-ai-crawlers)
5. [Perplexity Crawlers](#5-perplexity-crawlers)
6. [Microsoft/Bing Crawlers](#6-microsoftbing-crawlers)
7. [Other AI Crawlers](#7-other-ai-crawlers)
8. [Complete robots.txt Template](#8-complete-robotstxt-template)
9. [Platform Submission Guide](#9-platform-submission-guide)
10. [GA4 AI Traffic Tracking](#10-ga4-ai-traffic-tracking)
11. [Agentic Browsers](#11-agentic-browsers)
12. [Content Signals in robots.txt](#12-content-signals-in-robotstxt)

---

## 1. Crawler Quick Reference Table

| User-Agent | Company | Purpose | Block = |
|-----------|---------|---------|---------|
| `GPTBot` | OpenAI | Training data collection | Less model familiarity |
| `OAI-SearchBot` | OpenAI | Search index for ChatGPT Search | Invisible in ChatGPT search |
| `ChatGPT-User` | OpenAI | Real-time user-triggered fetch | May not respect robots.txt |
| `ClaudeBot` | Anthropic | Training data collection | Less model familiarity |
| `Claude-SearchBot` | Anthropic | Search index for Claude web search | Invisible in Claude search |
| `Claude-User` | Anthropic | Real-time user-triggered fetch | May not respect robots.txt |
| `Googlebot` | Google | Search index (also feeds AI Overviews) | No Google visibility at all |
| `Google-Extended` | Google | Gemini training data only | No Gemini training (AI Overviews unaffected) |
| `PerplexityBot` | Perplexity | Search index + real-time crawl | Invisible in Perplexity |
| `Bingbot` | Microsoft | Bing index (feeds ChatGPT + Copilot) | No Bing/ChatGPT/Copilot visibility |

**Key insight:** For maximum AI search visibility, you must allow at minimum: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot, and Bingbot. The training crawlers (GPTBot, ClaudeBot, Google-Extended) are optional but recommended.

---

## 2. OpenAI Crawlers

**GPTBot**
- Purpose: Collects web content for training OpenAI models
- User-agent string: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)`
- Respects: robots.txt
- Impact of blocking: 73% fewer ChatGPT citations (correlation, not direct causation)

**OAI-SearchBot**
- Purpose: Indexes content for ChatGPT's web search feature
- User-agent string: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)`
- Respects: robots.txt
- Impact of blocking: Removed from ChatGPT search results

**ChatGPT-User**
- Purpose: Real-time page fetching when a user's conversation triggers web lookup
- User-agent string: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)`
- Respects: robots.txt partially (operates on behalf of human user)
- Impact of blocking: Reduced but not eliminated ChatGPT access

---

## 3. Anthropic (Claude) Crawlers

**ClaudeBot**
- Purpose: Collects web content for training Claude models
- User-agent string: `claudebot` (case variations exist)
- Respects: robots.txt (including non-standard Crawl-delay directive)
- Impact of blocking: Less model familiarity with your content

**Claude-SearchBot**
- Purpose: Indexes content for Claude's web search (via Brave Search supplementation)
- User-agent string: `Claude-SearchBot`
- Respects: robots.txt
- Impact of blocking: Reduced visibility in Claude web search

**Claude-User**
- Purpose: Real-time fetching during user conversations with web search enabled
- User-agent string: `Claude-User`
- Respects: robots.txt partially (operates on behalf of human user)
- Impact of blocking: Reduced but not eliminated Claude access

**Note:** Claude's search primarily uses Brave Search's independent index. Ensure your site is visible in Brave Search for maximum Claude visibility.

---

## 4. Google AI Crawlers

**Googlebot**
- Purpose: Google's main search crawler. Feeds both traditional search AND AI Overviews.
- User-agent string: Various (Googlebot, Googlebot-Mobile, etc.)
- Respects: robots.txt
- Impact of blocking: Complete loss of Google search AND AI Overview visibility
- **Critical:** You CANNOT block AI Overviews without losing all Google search visibility. There is no separate AI Overview crawler.

**Google-Extended**
- Purpose: Controls training data for Gemini and Vertex AI products
- User-agent string: `Google-Extended`
- Respects: robots.txt
- Impact of blocking: Blocks Gemini training. Does NOT affect AI Overviews or Google Search.
- **Note:** Blocking Google-Extended is a viable choice if you want search visibility but don't want to contribute to Gemini training.

---

## 5. Perplexity Crawlers

**PerplexityBot**
- Purpose: Indexes content for Perplexity's AI search engine
- User-agent string: `PerplexityBot`
- Respects: robots.txt
- Impact of blocking: Invisible in Perplexity search results

Perplexity maintains its own independent index. It's the most citation-transparent platform (numbered inline citations in every response). Reddit content dominates at 46.5% of Perplexity citations.

**Perplexity Publishers' Program:** Partnership program with revenue sharing (80/20 split favoring publishers). Currently 30+ partners including TIME, Fortune, Der Spiegel.

---

## 6. Microsoft/Bing Crawlers

**Bingbot**
- Purpose: Bing's main search crawler. Feeds Bing Search, ChatGPT (via Bing), and Microsoft Copilot.
- User-agent string: `Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)`
- Respects: robots.txt
- Impact of blocking: Complete loss of Bing, ChatGPT (Bing-backed), and Copilot visibility

**Critical:** Bingbot is the single most impactful crawler for ChatGPT visibility. Industry analyses have reported ~87% overlap between ChatGPT citations and Bing's top results (figure undated, verify against a current source before quoting). If you do nothing else, make sure Bingbot can access your site and you've submitted to Bing Webmaster Tools.

**Bing Content Submission API:** Enables pushing structured facts directly for Copilot sidebar answers.

---

## 7. Other AI Crawlers

| User-Agent | Company | Purpose |
|-----------|---------|---------|
| `cohere-ai` | Cohere | Training/search |
| `Bytespider` | ByteDance | Training for AI products |
| `Applebot` | Apple | Apple Intelligence/Siri |
| `Applebot-Extended` | Apple | Apple Intelligence training |
| `FacebookBot` | Meta | Meta AI features |
| `meta-externalagent` | Meta | Meta AI training |
| `Amazonbot` | Amazon | Alexa/AI features |
| `Timesbot` | Perplexity (for publisher partners) | Premium content |

---

## 8. Complete robots.txt Template

### Maximum AI Visibility (recommended)

```
# Standard search
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# AI Search (required for AI search visibility)
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# AI Training (recommended for maximum citation likelihood)
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /
Crawl-delay: 10

User-agent: Google-Extended
Allow: /

# Apple Intelligence
User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Allow: /

# Blocked paths (customize per site)
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /tmp/
Disallow: /private/
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/

Sitemap: https://yoursite.com/sitemap.xml
```

### Selective (search only, no training)

```
# Standard search
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# AI Search only
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Block AI training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://yoursite.com/sitemap.xml
```

---

## 9. Platform Submission Guide

### Google Search Console
- URL: https://search.google.com/search-console
- Submit: XML sitemap
- Feeds: Google Search + AI Overviews + Gemini
- Monitor: Performance report (AI Overview data merged under "Web" search type)

### Bing Webmaster Tools
- URL: https://www.bing.com/webmasters
- Submit: XML sitemap + implement IndexNow
- Feeds: Bing + ChatGPT + Microsoft Copilot
- **This is the #1 priority submission for ChatGPT visibility**

### Brave Search
- Brave uses its own independent index (Web Discovery Project)
- No manual submission process
- Focus on standard SEO + allowing Brave's crawler
- Claude's search draws from Brave's index

### Perplexity
- No manual submission
- Allow PerplexityBot in robots.txt
- Perplexity's own index discovers content independently
- Publishers' Program for partnership: https://www.perplexity.ai/hub/partnerships

### Google Business Profile
- URL: https://business.google.com
- Critical for local/service businesses
- Primary data source for AI local recommendations
- Complete all fields: services, hours, photos, Q&A, posts

---

## 10. GA4 AI Traffic Tracking

### Create an AI referral traffic segment

In GA4, go to Explore > create a custom segment with traffic source filter:

**Regex for AI referral sources:**
```
perplexity\.ai|chatgpt\.com|chat\.openai\.com|claude\.ai|gemini\.google\.com|copilot\.microsoft\.com|you\.com|phind\.com
```

Apply this to:
- Session source/medium (for full session data)
- Landing page (to see which pages AI tools are sending traffic to)

### Track AI-influenced conversions

Add "How did you find us?" to lead forms with options:
- ChatGPT
- Claude
- Perplexity
- Google AI Overview
- Other AI tool

This captures dark AI traffic (users who get a recommendation from AI but then go straight to your site).

### Isolate likely AI Overview impressions in Google Search Console

No direct filter exists. Workarounds:
- Filter by query length (10+ words)
- Filter by conversational keywords: "compare", "top", "best", "vs", "how to", "what is"
- These queries are more likely to trigger AI Overviews

---

## 11. Agentic Browsers

A category that matters since late 2025: full browsers where an AI agent drives real user sessions. These are NOT crawlers and mostly cannot be managed via robots.txt.

| Product | Company | Notes |
|-|-|-|
| Atlas | OpenAI | Launched October 2025. Agent Mode executes multi-step tasks autonomously |
| Comet | Perplexity | Chromium-based; traffic looks like a normal Chrome session |
| Gemini in Chrome | Google | The main consumer of WebMCP tools during the origin trial |

**Implications for GEO:**
- Their traffic is largely indistinguishable from human sessions in analytics; you cannot segment it with a user-agent regex.
- They interact with your site through the rendered page and the accessibility tree, which is why Lighthouse's Agentic Browsing category audits a11y and CLS (see technical-checklist section 13).
- Sites that work well for keyboard/screen-reader users work well for agentic browsers. Broken forms, unlabeled buttons, and layout shift break agent task completion (and the sale that came with it).

---

## 12. Content Signals in robots.txt

Cloudflare's Content Signals Policy (September 2025) extends robots.txt with usage-category signals, expressed per user-agent group:

```
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
```

The three signals: `search` (build a search index), `ai-input` (feed content into AI answers at query time, i.e. RAG/grounding), `ai-train` (training or fine-tuning). The policy text frames restrictions as an express reservation of rights under EU DSM Directive 2019/790 Article 4.

**Why a GEO skill cares:**
- **Cloudflare's managed robots.txt injects `search=yes, ai-train=no` for 3.8M+ domains.** If the site sits behind Cloudflare, fetch the LIVE robots.txt and check whether a managed policy has been layered on top of (or replaces) the site's own file. A client maximizing AI visibility may be sending "don't train on me" without knowing it.
- For maximum AI visibility, either omit content signals entirely or set them permissively. `ai-input=no` works against citation in AI answers.
- Same audit lesson as user-agents: the robots.txt you deploy is not always the robots.txt the world sees. CDN-level bot management (Cloudflare Bot Fight Mode, WAF rules, security plugins) can also block AI crawlers before robots.txt is ever read; test with `curl -A "GPTBot" https://site.com/` and check for 403s.

# Technical Checklist for AI Search Visibility

Complete technical implementation guide. Work through sequentially for new builds, or use as an audit checklist for existing sites.

## Table of Contents

1. [Server-Side Rendering](#1-server-side-rendering)
2. [Robots.txt Configuration](#2-robotstxt-configuration)
3. [Schema.org Structured Data](#3-schemaorg-structured-data)
4. [XML Sitemap + RSS/Atom Feed](#4-xml-sitemap--rssatom-feed)
5. [llms.txt File](#5-llmstxt-file)
6. [Page Speed Optimization](#6-page-speed-optimization)
7. [Search Console Submissions](#7-search-console-submissions)
8. [IndexNow Implementation](#8-indexnow-implementation)
9. [Markdown Alternate Links](#9-markdown-alternate-links)
10. [Meta Tags for AI Extraction](#10-meta-tags-for-ai-extraction)
11. [HTTPS and Security](#11-https-and-security)
12. [Internal Linking Architecture](#12-internal-linking-architecture)
13. [Lighthouse Agentic Browsing Audit](#13-lighthouse-agentic-browsing-audit)

---

## 1. Server-Side Rendering

**Priority: CRITICAL. This is a binary gate**

Most AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) do NOT execute JavaScript. Analysis of 500M+ GPTBot fetches showed zero JS execution. Only Googlebot reliably renders JS.

**Audit check:**
```bash
# Test what a non-JS crawler sees
curl -s -A "GPTBot" https://yoursite.com/page | grep -c "your-expected-content"
# If 0 matches, your content is invisible to AI crawlers
```

**For PHP sites:** This is rarely an issue since PHP renders server-side. But check for:
- Content loaded via AJAX after page load
- React/Vue/Angular widgets embedded in PHP pages
- Lazy-loaded content that requires scroll events
- Content behind JavaScript tabs/accordions (use `<details>`/`<summary>` instead)

**For JS frameworks (React, Vue, Angular, Next.js, Nuxt):**
- Enable SSR or Static Site Generation (SSG)
- Use `getServerSideProps` (Next.js) or equivalent
- Pre-render critical pages at minimum

---

## 2. Robots.txt Configuration

**Priority: CRITICAL. Blocking crawlers = invisible to AI**

See `crawler-reference.md` for complete user-agent list. Here's the recommended robots.txt template:

```
# === Standard Search Crawlers ===
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# === AI Search Crawlers (allow for search visibility) ===
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# === AI Training Crawlers (optional, allow if you want to be in training data) ===
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

# === Block paths that shouldn't be crawled ===
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /tmp/
Disallow: /private/

Sitemap: https://yoursite.com/sitemap.xml
```

**Important distinctions:**
- **Search crawlers** (OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, PerplexityBot): These power real-time search features. Blocking them removes you from AI search results.
- **Training crawlers** (GPTBot, ClaudeBot, Google-Extended): These collect data for model training. Blocking them doesn't affect real-time search but reduces long-term model familiarity with your content. Sites blocking GPTBot were cited 73% less often.
- ChatGPT-User and Claude-User operate on behalf of human users and may not fully respect robots.txt blocks.

**Audit check:**
```bash
# Fetch and inspect current robots.txt
curl -s https://yoursite.com/robots.txt
# Look for blanket "Disallow: /" rules or blocks on AI user-agents
```

---

## 3. Schema.org Structured Data

**Priority: HIGH. GPT-4 accuracy jumps from 16% to 54% with structured data**

All schema must be:
- In JSON-LD format (Google-recommended, easiest for AI to parse)
- Rendered server-side in the HTML `<head>` (not injected via JS)
- Accurately matching visible page content (mismatches reduce trust)

### Schema templates by page type

**Organization (site-wide, on every page):**
```json
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Your Company Name",
  "url": "https://yoursite.com",
  "logo": "https://yoursite.com/logo.png",
  "description": "One-sentence description of what you do",
  "foundingDate": "2020",
  "address": {
    "@type": "PostalAddress",
    "streetAddress": "123 Main St",
    "addressLocality": "Zagreb",
    "addressCountry": "HR"
  },
  "contactPoint": {
    "@type": "ContactPoint",
    "email": "info@yoursite.com",
    "contactType": "customer service"
  },
  "sameAs": [
    "https://www.linkedin.com/company/yourcompany",
    "https://www.facebook.com/yourcompany",
    "https://twitter.com/yourcompany"
  ]
}
```

**Article / BlogPosting:**
```json
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Article Title Here",
  "description": "2-sentence summary of what this article covers",
  "author": {
    "@type": "Person",
    "name": "Author Name",
    "url": "https://yoursite.com/about/author-name",
    "jobTitle": "Author's Role/Title",
    "sameAs": ["https://linkedin.com/in/author"]
  },
  "publisher": {
    "@type": "Organization",
    "name": "Your Company Name",
    "logo": { "@type": "ImageObject", "url": "https://yoursite.com/logo.png" }
  },
  "datePublished": "2026-03-01",
  "dateModified": "2026-03-20",
  "mainEntityOfPage": "https://yoursite.com/blog/article-slug",
  "image": "https://yoursite.com/images/article-hero.jpg"
}
```

**FAQPage (powerful for AI extraction):**
```json
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "What is [topic]?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "A direct, complete answer to the question."
      }
    },
    {
      "@type": "Question",
      "name": "How does [topic] compare to [alternative]?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Specific comparison with concrete details."
      }
    }
  ]
}
```

**Product (for e-commerce/service pages):**
```json
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Product Name",
  "description": "Clear product description",
  "image": "https://yoursite.com/images/product.jpg",
  "brand": { "@type": "Brand", "name": "Brand Name" },
  "sku": "SKU-12345",
  "offers": {
    "@type": "Offer",
    "price": "99.00",
    "priceCurrency": "EUR",
    "availability": "https://schema.org/InStock",
    "url": "https://yoursite.com/products/product-name"
  },
  "aggregateRating": {
    "@type": "AggregateRating",
    "ratingValue": "4.5",
    "reviewCount": "42"
  }
}
```

**HowTo (for tutorials/guides):**
```json
{
  "@context": "https://schema.org",
  "@type": "HowTo",
  "name": "How to [do thing]",
  "description": "Brief overview",
  "step": [
    {
      "@type": "HowToStep",
      "name": "Step 1 title",
      "text": "Step 1 instructions"
    }
  ]
}
```

**LocalBusiness (for agency/local service sites):**
```json
{
  "@context": "https://schema.org",
  "@type": "ProfessionalService",
  "name": "Agency Name",
  "description": "What you do in one sentence",
  "url": "https://yoursite.com",
  "address": { ... },
  "geo": { "@type": "GeoCoordinates", "latitude": "45.815", "longitude": "15.982" },
  "openingHours": "Mo-Fr 09:00-17:00",
  "priceRange": "$$",
  "areaServed": "Croatia"
}
```

### PHP implementation pattern

```php
// In your base template or header include
function render_schema_jsonld(array $schema_data): string {
    return '<script type="application/ld+json">' .
           json_encode($schema_data, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE | JSON_PRETTY_PRINT) .
           '</script>';
}

// Per page template, build the schema array and render in <head>
$schema = [
    '@context' => 'https://schema.org',
    '@type' => 'Article',
    'headline' => $article->title,
    'datePublished' => $article->published_at,
    'dateModified' => $article->updated_at,
    // ... etc
];
echo render_schema_jsonld($schema);
```

---

## 4. XML Sitemap + RSS/Atom Feed

**Priority: HIGH**

**XML Sitemap requirements:**
- Include all indexable pages
- Accurate `<lastmod>` timestamps (critical for AI crawlers detecting fresh content)
- Split by content type if site is large (blog-sitemap.xml, product-sitemap.xml)
- Submit to both Google Search Console AND Bing Webmaster Tools
- Reference in robots.txt: `Sitemap: https://yoursite.com/sitemap.xml`

**RSS/Atom feed:**
- Google recommends pairing sitemaps with RSS/Atom feeds for change detection
- Microsoft's NLWeb protocol uses RSS feeds as a primary data source
- Include at minimum: title, link, description, pubDate/updated, author
- Auto-generate from your CMS or build a `/feed.xml` endpoint

```xml
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Site Name</title>
    <link>https://yoursite.com</link>
    <description>Site description</description>
    <atom:link href="https://yoursite.com/feed.xml" rel="self" type="application/rss+xml"/>
    <item>
      <title>Article Title</title>
      <link>https://yoursite.com/blog/slug</link>
      <description>First 200 chars of article</description>
      <pubDate>Wed, 20 Mar 2026 10:00:00 +0000</pubDate>
      <guid>https://yoursite.com/blog/slug</guid>
    </item>
  </channel>
</rss>
```

---

## 5. llms.txt File

**Priority: LOW-MEDIUM (uneven adoption as a retrieval source, but since 2026 Lighthouse/PageSpeed Insights audits the file, so one that fails its checks shows up as a red audit to anyone who tests your site)**

Place at `https://yoursite.com/llms.txt`. Markdown-formatted overview of your site for AI consumption. Takes minutes to create, costs nothing.

**Format is now machine-checked.** Lighthouse's llms-txt audit (Agentic Browsing category, see section 13) runs three regular expressions over the raw file, per its source (`core/audits/agentic/llms-txt.js` in the GoogleChrome/lighthouse repo, checked 2026-10-11). It fails the file when any of these is missing:
- An H1 header anywhere in the file (`# Title`).
- At least one markdown link, `[text](url)`. Plain `Title: URL` lines do not count and fail with "File does not appear to contain any links".
- At least 50 characters.

A server error when fetching the file also fails the audit; a 404 is reported as not applicable, because the file is optional. Lighthouse checks nothing else: section order, blockquotes and H2 sections are the llmstxt.org format, worth following, but not part of the audit. `geo_probe.py` reports the three Lighthouse checks as `llms-format` and its own spec heuristic (the H1 should be the first line) separately as low-severity `llms-spec`.

```markdown
# Site Name

> One-sentence description of what this site is about.

## About
Brief 2-3 sentence description of the organization, what it does, who it serves.

## Key Pages
- [Homepage](https://yoursite.com/): Brief description
- [Services](https://yoursite.com/services/): What services are offered
- [Blog](https://yoursite.com/blog/): Topics covered

## Popular Content
- [Article Title](https://yoursite.com/blog/article): One-line summary
- [Guide Title](https://yoursite.com/guides/guide): One-line summary

## Contact
- Email: info@yoursite.com
- Location: Zagreb, Croatia
```

Optionally create `llms-full.txt`: the same overview followed by the full markdown of your key pages/articles, so a tool gets deep context in one fetch. This is cheap when the site already exposes per-page markdown (e.g. a `/article/markdown` endpoint or a content/export service): a single route can concatenate the `llms.txt` intro with each published page's markdown. Serve it `X-Robots-Tag: noindex, follow` (it duplicates page bodies, so avoid search-index duplicate content) and exclude any gated/premium content so you don't leak it. When you bound the dump (top-N, page cap), say what was dropped rather than silently truncating.

**Priority:** LOW / optional. Future-proofing, not a required output.

**Current status (as of 2026-07, per third-party trackers, not primary sources; verify before quoting):** Adoption is growing but uneven (roughly 10% of large domain samples). Some providers have confirmed support (Anthropic by late 2024, Perplexity by 2025), and by 2026 most major Western AI platforms offer partial support. The most concrete, verifiable adoption is in AI dev tools: Claude Code, Cursor, Windsurf, GitHub Copilot, Cline, Aider all fetch `/llms.txt` and `/llms-full.txt`, especially for documentation/reference sites. Google has said it does not use llms.txt for Search, and as of Q1 2026 no major AI provider (OpenAI, Google, Anthropic, Meta, Mistral) has publicly committed to reading it in production answer systems. What HAS changed: Google now validates the file in Lighthouse and PageSpeed Insights (Agentic Browsing category, section 13), which makes it a visible pass/fail checkbox for clients, tools, and audits. Bottom line: still don't prioritize it over robots.txt, schema, sitemap, or content structure, but if you ship one, make sure it passes the three Lighthouse checks above.

---

## 6. Page Speed Optimization

**Priority: HIGH. Pages with FCP < 0.4s get 3× more AI citations**

Target metrics:
- First Contentful Paint (FCP): < 0.4 seconds
- Time to First Byte (TTFB): < 200ms
- Largest Contentful Paint (LCP): < 2.5 seconds

AI crawlers abandon slow pages more aggressively than traditional search crawlers.

Quick wins for PHP sites:
- Enable OPcache
- Use a reverse proxy cache (Varnish, Nginx FastCGI cache)
- Optimize images (WebP, lazy loading for below-fold)
- Minimize CSS/JS blocking resources
- Use a CDN for static assets
- Database query optimization (avoid N+1 queries)

---

## 7. Search Console Submissions

**Priority: HIGH**

Submit your sitemap to both:
- **Google Search Console**: https://search.google.com/search-console
- **Bing Webmaster Tools**: https://www.bing.com/webmasters

Bing is critical because ChatGPT and Copilot both use Bing's index. 87% of ChatGPT citations match Bing's top results.

---

## 8. IndexNow Implementation

**Priority: MEDIUM-HIGH**

IndexNow instantly notifies Bing (and by extension ChatGPT/Copilot) when content is published or updated.

```php
// After publishing or updating content
function notify_indexnow(string $url, string $api_key): void {
    $endpoint = "https://api.indexnow.org/indexnow";
    $params = http_build_query([
        'url' => $url,
        'key' => $api_key,
        'host' => parse_url($url, PHP_URL_HOST),
    ]);

    $ch = curl_init("{$endpoint}?{$params}");
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_TIMEOUT, 5);
    curl_exec($ch);
    curl_close($ch);
}
```

Place your API key file at `https://yoursite.com/{api-key}.txt`.

---

## 9. Markdown Alternate Links

**Priority: LOW (experimental, but zero-cost if you have Markdown source)**

If you write content in Markdown (e.g., from Obsidian, a CMS, or a static site generator), serve the raw Markdown version alongside HTML and signal it:

```html
<link rel="alternate" type="text/markdown" href="/blog/post-slug.md" />
```

AI models process Markdown natively. This gives them a clean, uncluttered version of your content.

---

## 10. Meta Tags for AI Extraction

**Priority: MEDIUM**

Ensure every important page has:
```html
<meta name="description" content="Concise, factual 150-160 char description" />
<meta name="author" content="Author Name" />
<meta property="og:title" content="Page Title" />
<meta property="og:description" content="Same as meta description" />
<meta property="og:type" content="article" />
<meta property="og:url" content="https://yoursite.com/page" />
<meta property="og:image" content="https://yoursite.com/images/og-image.jpg" />
<meta property="article:published_time" content="2026-03-01T10:00:00Z" />
<meta property="article:modified_time" content="2026-03-20T10:00:00Z" />
```

---

## 11. HTTPS and Security

**Priority: CRITICAL (baseline requirement)**

All pages must be served over HTTPS. This is a ranking factor for Google/Bing and a trust signal for AI systems. No exceptions.

---

## 12. Internal Linking Architecture

**Priority: HIGH. Pillar-cluster architecture delivers 2.7× citation multiplier**

Structure sites as topic clusters:
- **Pillar page**: Covers a whole topic end to end (2000+ words)
- **Cluster pages**: Detailed sub-topic articles linking back to pillar
- **Bidirectional links**: Pillar links to clusters, clusters link to pillar
- Use descriptive anchor text (not "click here")

82.5% of ChatGPT citations link to pages within established topic hierarchies, not isolated articles.

---

## 13. Lighthouse Agentic Browsing Audit

**Priority: MEDIUM (new in 2026, the first mainstream tool that grades agent-readiness; category is explicitly "under development and subject to change")**

Chrome Lighthouse (Chrome 150+, also surfaced on pagespeed.web.dev) added an "Agentic Browsing" category that scores how well a site works for AI agents. Unlike other categories there is no 0-100 weighted score: each audit reports pass/fail (or a pass ratio) independently.

**What it checks:**

1. **llms.txt presence and format**: see section 5 for the three checks (an H1, at least one markdown `[text](url)` link, at least 50 characters).
2. **WebMCP integration**: WebMCP (Web Model Context Protocol) is a W3C proposal that lets a page expose callable tools to AI agents via `navigator.modelContext` instead of the agent guessing at the UI. Announced February 2026, in Chrome origin trial (Chrome 149-156); the main consuming agent today is Gemini in Chrome. Lighthouse verifies both declarative tools (defined in HTML) and imperative tools (registered from JS). Only relevant for interactive sites (search, cart, booking, filters); content/publishing sites can safely skip it. Reference: https://developer.chrome.com/docs/ai/webmcp
3. **Agent-centric accessibility subset**: agents drive pages through the accessibility tree, so Lighthouse filters the a11y audits that matter for machine interaction: every interactive element has a programmatic name, ARIA roles are valid and on compatible elements, identical links have the same purpose, touch targets have sufficient size/spacing. Accessibility fixes now double as agent-readiness fixes; pitch them that way.
4. **Layout stability (CLS)**: agents target elements by position, so layout shift breaks them the same way it annoys humans.

**Audit check:** run the site through https://pagespeed.web.dev/ and read the Agentic Browsing section at the bottom of the report, or use Lighthouse in Chrome DevTools (Chrome 150+).

**Watch for false positives:** entrance animations that fade content in (opacity 0 to 1 on scroll-reveal) can make the contrast and visibility audits fail elements whose static styles pass. Verify the computed static colors before "fixing" contrast findings caused by animation timing.

#!/usr/bin/env python3
"""
geo_probe.py - deterministic first phase of a GEO (AI search visibility) audit.

Fetches what the skill's technical checklist tells the model to inspect by hand on every
run: robots.txt (per AI crawler: allowed, disallowed, unmentioned; Sitemap and
Content-Signal lines), /llms.txt and /llms-full.txt (presence, size, first heading,
format), and each page's JSON-LD blocks plus the head tags AI extraction depends on.

Plain Python, standard library only. It reports evidence, never recommendations: the
model judges the candidate set in phase two and reads the pages only for what a probe
cannot see (wording, citability of the content itself).

Usage:
    python3 geo_probe.py <url-or-domain> [more page URLs...]
    python3 geo_probe.py <url> --json            # machine-readable, for phase two
    python3 geo_probe.py <url> --severity high   # report and gate on high only
    python3 geo_probe.py <url> --timeout 5       # per-request timeout in seconds

Exit code is 1 when any finding at or above --severity exists (default: low, so any
finding), 0 when the reported set is empty, 2 on bad arguments.
"""
import argparse
import json
import re
import sys
import urllib.error
import urllib.parse
import urllib.request
from html.parser import HTMLParser

# Source of truth for this list: references/crawler-reference.md, section 1 (quick
# reference table) and section 7 (other AI crawlers). The reference may not ship next to
# this script, so the tokens are embedded; update both together. Tier decides severity
# when a crawler is disallowed: "search" powers live answers (blocking removes the site
# from AI search), "training" only reduces long-term model familiarity, "other" is
# smaller platforms.
CRAWLERS = {
    "Googlebot": "search", "Bingbot": "search", "OAI-SearchBot": "search", "ChatGPT-User": "search",
    "Claude-SearchBot": "search", "Claude-User": "search", "PerplexityBot": "search",
    "GPTBot": "training", "ClaudeBot": "training", "Google-Extended": "training",
    "cohere-ai": "other", "Bytespider": "other", "Applebot": "other", "Applebot-Extended": "other",
    "FacebookBot": "other", "meta-externalagent": "other", "Amazonbot": "other", "Timesbot": "other",
}
TIER_SEVERITY = {"search": "high", "training": "medium", "other": "low"}

# Fields per @type, from references/technical-checklist.md section 3 templates and the
# entity checks in SKILL.md step 2. First tuple is required (missing = medium), second is
# recommended (missing = low). Aliases map subtypes onto the template they follow.
SCHEMA_FIELDS = {
    "Organization": (("name", "url"), ("logo", "sameAs", "address")),
    "WebSite": (("name", "url"), ()),
    "Article": (("headline", "author", "datePublished"), ("dateModified", "publisher")),
    "FAQPage": (("mainEntity",), ()),
    "Product": (("name", "offers"), ("description", "brand")),
    "HowTo": (("name", "step"), ()),
}
SCHEMA_ALIASES = {
    "LocalBusiness": "Organization", "ProfessionalService": "Organization", "Corporation": "Organization",
    "BlogPosting": "Article", "NewsArticle": "Article", "TechArticle": "Article",
}

SEVERITY_RANK = {"high": 0, "medium": 1, "low": 2}
# Below this many characters of visible body text, a homepage is almost certainly a JS
# shell (`<div id="root"></div>` plus a bundle). Heuristic: a legitimately terse landing
# page trips it too, which is why the finding says so in its evidence.
MIN_BODY_TEXT = 400
# shortcut: single attempt per URL, no retry or backoff. A flaky host shows up as
# `unreachable` and the model re-runs; retries would hide intermittent blocking, which is
# itself a finding for AI crawlers.
USER_AGENT = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36"
MAX_BODY_BYTES = 5 * 1024 * 1024  # shortcut: cap reads; nothing we inspect lives past 5 MB
SKIP_TEXT_TAGS = {"script", "style", "noscript", "template", "svg", "head"}
VOID_TAGS = {"area", "base", "br", "col", "embed", "hr", "img", "input", "link", "meta", "param", "source",
             "track", "wbr"}


# ---------------------------------------------------------------- fetching

def fetch(url, timeout):
    """One GET. Returns a dict with http_status, text, content_type, final_url, error.
    Network and HTTP errors land in the dict rather than raising, so every URL yields
    a finding instead of a crash."""
    out = {"url": url, "http_status": None, "text": "", "content_type": "", "final_url": url, "error": None}
    req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT, "Accept": "*/*"})
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            raw = resp.read(MAX_BODY_BYTES)
            out["http_status"] = resp.status
            out["final_url"] = resp.geturl()
            out["content_type"] = resp.headers.get_content_type()
            charset = resp.headers.get_content_charset() or "utf-8"
    except urllib.error.HTTPError as e:
        # A 404 is an answer, not a failure: robots.txt or llms.txt may simply not exist.
        out["http_status"] = e.code
        try:
            raw = e.read(MAX_BODY_BYTES)
        except Exception:  # noqa: BLE001 - the status is what matters here
            raw = b""
        out["content_type"] = e.headers.get_content_type() if e.headers else ""
        charset = (e.headers.get_content_charset() if e.headers else None) or "utf-8"
    except Exception as e:  # noqa: BLE001 - URLError, socket.timeout, ssl errors, bad host
        out["error"] = f"{type(e).__name__}: {e}"
        return out
    try:
        out["text"] = raw.decode(charset, errors="replace")
    except LookupError:  # unknown charset label in the header
        out["text"] = raw.decode("utf-8", errors="replace")
    return out


def normalize_target(arg):
    """Accept `example.com`, `example.com/path` or a full URL. Returns (origin, page_url)."""
    if not re.match(r"^[a-z][a-z0-9+.-]*://", arg, re.I):
        arg = "https://" + arg
    parts = urllib.parse.urlsplit(arg)
    if not parts.netloc:
        raise ValueError(f"not a URL or domain: {arg}")
    origin = f"{parts.scheme}://{parts.netloc}"
    page = urllib.parse.urlunsplit((parts.scheme, parts.netloc, parts.path or "/", parts.query, ""))
    return origin, page


# ---------------------------------------------------------------- robots.txt

def parse_robots(text):
    """Groups of (agent tokens, rules) plus Sitemap and Content-Signal lines. Follows the
    REP grouping: consecutive User-agent lines share one rule block; a rule line closes
    the agent list. Comments and blank lines are ignored."""
    groups, sitemaps, signals = [], [], []
    agents, rules, collecting = [], [], False
    for raw in text.splitlines():
        line = raw.split("#", 1)[0].strip()
        if not line or ":" not in line:
            continue
        key, _, val = line.partition(":")
        key, val = key.strip().lower(), val.strip()
        if key == "user-agent":
            if not collecting:
                if agents:
                    groups.append((agents, rules))
                agents, rules = [], []
                collecting = True
            agents.append(val.lower())
            continue
        collecting = False
        if key in ("allow", "disallow") and agents:
            rules.append((key, val))
        elif key == "sitemap":
            sitemaps.append(val)
        elif key == "content-signal":
            signals.append({"agents": list(agents), "value": val, "line": line})
    if agents:
        groups.append((agents, rules))
    return groups, sitemaps, signals


def _rule_matches_root(path):
    # Empty Disallow means "nothing disallowed"; `*` and `$` are the only wildcards.
    if not path:
        return False
    pattern = "^" + re.escape(path).replace(r"\*", ".*")
    if pattern.endswith(r"\$"):
        pattern = pattern[:-2] + "$"
    return re.match(pattern, "/") is not None


def root_access(rules):
    """Longest-match evaluation of `/`; Allow wins ties, as Google and Bing do it."""
    best = None
    for kind, path in rules:
        if not _rule_matches_root(path):
            continue
        key = (len(path), kind == "allow")
        if best is None or key > best[0]:
            best = (key, kind, path)
    if best is None:
        return "allowed", None
    return ("allowed" if best[1] == "allow" else "disallowed"), f"{best[1].capitalize()}: {best[2]}"


def crawler_status(groups, token):
    own = [r for agents, rules in groups if token.lower() in agents for r in rules]
    if any(token.lower() in agents for agents, _ in groups):
        status, rule = root_access(own)
        return {"status": status, "via": token, "rule": rule}
    wildcard = [r for agents, rules in groups if "*" in agents for r in rules]
    if any("*" in agents for agents, _ in groups):
        status, rule = root_access(wildcard)
        return {"status": status, "via": "*", "rule": rule}
    return {"status": "unmentioned", "via": None, "rule": None}


def check_robots(origin, timeout, findings):
    url = origin + "/robots.txt"
    got = fetch(url, timeout)
    data = {"url": url, "http_status": got["http_status"], "status": "present", "sitemaps": [],
            "content_signals": [], "crawlers": {}}
    if got["error"]:
        data["status"] = "unreachable"
        findings.append(finding("robots-unreachable", "medium", url, got["error"]))
        return data
    if got["http_status"] == 404:
        data["status"] = "missing"
        findings.append(finding("robots-missing", "low", url, "HTTP 404: no robots.txt, every crawler is allowed by default"))
        return data
    if not 200 <= got["http_status"] < 300:
        data["status"] = "unreachable"
        findings.append(finding("robots-unreachable", "medium", url, f"HTTP {got['http_status']}"))
        return data
    if got["text"].lstrip()[:1] == "<":
        # Soft 404: a server that answers unknown paths with the homepage.
        data["status"] = "missing"
        findings.append(finding("robots-missing", "low", url, f"HTTP {got['http_status']} but the body is HTML, not a robots.txt"))
        return data

    groups, sitemaps, signals = parse_robots(got["text"])
    data["sitemaps"] = sitemaps
    data["content_signals"] = signals
    if not sitemaps:
        findings.append(finding("robots-no-sitemap", "medium", url, "no Sitemap: line"))
    for sig in signals:
        for part in re.split(r"[,\s]+", sig["value"]):
            name, _, val = part.partition("=")
            if val.lower() != "no":
                continue
            sev = "high" if name in ("ai-input", "search") else "low"
            findings.append(finding("robots-content-signal", sev, url,
                                    f"{sig['line']} (User-agent: {', '.join(sig['agents']) or '?'})"))
    for token, tier in CRAWLERS.items():
        st = crawler_status(groups, token)
        st["tier"] = tier
        data["crawlers"][token] = st
        if st["status"] == "disallowed":
            findings.append(finding("robots-disallow", TIER_SEVERITY[tier], url,
                                    f"{token} ({tier}) blocked by `{st['rule']}` under User-agent: {st['via']}"))
    return data


# ---------------------------------------------------------------- llms.txt

LINK_RE = re.compile(r"\[[^\]]+\]\(https?://[^)\s]+\)")


def llms_format_problems(text):
    """Split llms.txt problems into what Lighthouse fails and what is only this plugin's reading of the spec.

    The first list mirrors Lighthouse's llms-txt audit (core/audits/agentic/llms-txt.js): three regexes over
    the raw text, an H1 anywhere, at least one [text](url) link, at least 50 characters. Nothing else fails
    that audit. The second list is a heuristic from the llmstxt.org format, which puts the H1 first; Lighthouse
    does not check it, so it reports at low severity and never as a Lighthouse failure.
    """
    lighthouse = []
    if not re.search(r"^\s*#\s+.+", text, re.M):
        lighthouse.append("no H1 header (Lighthouse check)")
    if not re.search(r"\[.+\]\(.+\)", text):
        lighthouse.append("no markdown [text](url) links (Lighthouse check)")
    if len(text) < 50:
        lighthouse.append(f"only {len(text)} characters, under Lighthouse's 50 (Lighthouse check)")
    lines = [ln for ln in text.splitlines() if ln.strip()]
    spec = []
    if lines and not lines[0].lstrip().startswith("# "):
        spec.append("first non-blank line is not the H1 the llmstxt.org format puts first (plugin heuristic)")
    return lighthouse, spec


def check_llms(origin, name, timeout, findings):
    url = f"{origin}/{name}"
    got = fetch(url, timeout)
    data = {"url": url, "http_status": got["http_status"], "status": "absent", "size": 0,
            "first_heading": None, "h1_first": False, "link_count": 0}
    if got["error"]:
        data["status"] = "unreachable"
        findings.append(finding("llms-unreachable", "low", url, got["error"]))
        return data
    text = got["text"]
    ok = got["http_status"] and 200 <= got["http_status"] < 300
    is_html = text.lstrip()[:1] == "<" or "html" in got["content_type"]
    if not ok or is_html:
        # A 2xx with an HTML body is a server answering unknown paths with the homepage.
        why = f"HTTP {got['http_status']}" + (" but the body is HTML (soft 404)" if ok and is_html else "")
        findings.append(finding("llms-absent", "low", url, why))
        return data

    data["status"] = "present"
    data["size"] = len(text.encode("utf-8"))
    lines = [ln for ln in text.splitlines() if ln.strip()]
    heading = next((ln.strip() for ln in lines if ln.lstrip().startswith("#")), None)
    data["first_heading"] = heading
    data["h1_first"] = bool(lines) and lines[0].lstrip().startswith("# ")
    data["link_count"] = len(LINK_RE.findall(text))
    lighthouse, spec = llms_format_problems(text)
    if lighthouse:
        findings.append(finding("llms-format", "medium", url, "; ".join(lighthouse)))
    if spec:
        findings.append(finding("llms-spec", "low", url, "; ".join(spec)))
    return data


# ---------------------------------------------------------------- pages

class PageParser(HTMLParser):
    """Collects the head tags the checklist names, the JSON-LD blocks, and the visible
    body text length. Tolerant of unclosed tags the way browsers are."""

    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.title = None
        self.meta_description = None
        self.canonical = None
        self.lang = None
        self.jsonld = []
        self.text_chars = 0
        self._stack = []
        self._title = None
        self._jsonld = None

    def handle_starttag(self, tag, attrs):
        tag = tag.lower()
        a = {k.lower(): (v or "") for k, v in attrs}
        if tag == "html" and self.lang is None:
            self.lang = a.get("lang") or ""
        elif tag == "meta" and (a.get("name") or "").lower() == "description":
            self.meta_description = a.get("content", "")
        elif tag == "link" and "canonical" in (a.get("rel") or "").lower().split():
            self.canonical = a.get("href", "")
        if tag in VOID_TAGS:
            return
        self._stack.append(tag)
        if tag == "script" and "ld+json" in a.get("type", "").lower():
            self._jsonld = []
        elif tag == "title" and self.title is None:
            self._title = []

    def handle_endtag(self, tag):
        tag = tag.lower()
        if tag in VOID_TAGS or tag not in self._stack:
            return
        if tag == "script" and self._jsonld is not None:
            self.jsonld.append("".join(self._jsonld))
            self._jsonld = None
        if tag == "title" and self._title is not None:
            self.title = re.sub(r"\s+", " ", "".join(self._title)).strip()
            self._title = None
        idx = len(self._stack) - 1 - self._stack[::-1].index(tag)
        del self._stack[idx:]

    def handle_data(self, data):
        if self._jsonld is not None:
            self._jsonld.append(data)
        elif self._title is not None:
            self._title.append(data)
        elif not (set(self._stack) & SKIP_TEXT_TAGS):
            self.text_chars += len(re.sub(r"\s+", " ", data).strip())


def flatten_jsonld(node):
    """Yield the page's own entities: top-level arrays and @graph members. Nested
    author/publisher/offers objects are references and stay out, otherwise an Article's
    publisher would be judged against the site-wide Organization template."""
    if isinstance(node, list):
        for item in node:
            yield from flatten_jsonld(item)
    elif isinstance(node, dict):
        if "@type" in node:
            yield node
        if "@graph" in node:
            yield from flatten_jsonld(node["@graph"])


def types_of(node):
    t = node.get("@type")
    return [t] if isinstance(t, str) else [x for x in (t or []) if isinstance(x, str)]


def check_schema_fields(node, url, findings):
    for t in types_of(node):
        canon = SCHEMA_ALIASES.get(t, t)
        if canon not in SCHEMA_FIELDS:
            continue
        required, recommended = SCHEMA_FIELDS[canon]
        for field in required:
            if not node.get(field):
                findings.append(finding("schema-field-missing", "medium", url, f"{t} without `{field}`"))
        for field in recommended:
            if not node.get(field):
                findings.append(finding("schema-field-missing", "low", url, f"{t} without `{field}`"))
        if canon == "FAQPage":
            items = node.get("mainEntity") or []
            items = items if isinstance(items, list) else [items]
            for i, q in enumerate(items):
                answer = q.get("acceptedAnswer") if isinstance(q, dict) else None
                if not isinstance(q, dict) or not q.get("name") or not (isinstance(answer, dict) and answer.get("text")):
                    findings.append(finding("schema-faq-item-incomplete", "medium", url,
                                            f"mainEntity[{i}] lacks name or acceptedAnswer.text"))


def check_page(url, timeout, findings):
    got = fetch(url, timeout)
    data = {"url": url, "final_url": got["final_url"], "http_status": got["http_status"], "title": None,
            "meta_description": None, "canonical": None, "lang": None, "text_chars": 0, "jsonld": []}
    if got["error"]:
        findings.append(finding("unreachable", "high", url, got["error"]))
        return data
    if not 200 <= got["http_status"] < 300:
        findings.append(finding("unreachable", "high", url, f"HTTP {got['http_status']}"))
        return data
    if not got["final_url"].startswith("https://"):
        findings.append(finding("not-https", "high", url, f"served at {got['final_url']}"))

    p = PageParser()
    p.feed(got["text"])
    p.close()
    data.update(title=p.title, meta_description=p.meta_description, canonical=p.canonical, lang=p.lang,
                text_chars=p.text_chars)

    if not p.title:
        findings.append(finding("title-missing", "medium", url, "no <title> text"))
    if not p.meta_description:
        findings.append(finding("meta-description-missing", "medium", url, "no <meta name=description>"))
    if not p.canonical:
        findings.append(finding("canonical-missing", "low", url, "no <link rel=canonical>"))
    if not p.lang:
        findings.append(finding("html-lang-missing", "low", url, "<html> has no lang attribute"))
    if p.text_chars < MIN_BODY_TEXT:
        findings.append(finding("thin-body-without-js", "high", url,
                                f"{p.text_chars} visible body chars without JS execution (heuristic, threshold {MIN_BODY_TEXT})"))

    types_seen = []
    for i, block in enumerate(p.jsonld):
        entry = {"index": i, "ok": False, "types": [], "error": None}
        try:
            parsed = json.loads(block)
        except json.JSONDecodeError as e:
            entry["error"] = str(e)
            snippet = re.sub(r"\s+", " ", block.strip())[:120] or "(empty block)"
            findings.append(finding("schema-parse-error", "high", url, f"block {i}: {e}; starts `{snippet}`"))
            data["jsonld"].append(entry)
            continue
        entry["ok"] = True
        for node in flatten_jsonld(parsed):
            entry["types"].extend(types_of(node))
            check_schema_fields(node, url, findings)
        types_seen.extend(entry["types"])
        data["jsonld"].append(entry)
    if not p.jsonld:
        findings.append(finding("schema-missing", "high", url, "no <script type=application/ld+json> blocks"))
    elif not any(SCHEMA_ALIASES.get(t, t) == "Organization" for t in types_seen):
        findings.append(finding("schema-no-organization", "low", url,
                                f"JSON-LD types present: {', '.join(sorted(set(types_seen))) or 'none'}"))
    return data


# ---------------------------------------------------------------- orchestration

def finding(check, severity, url, evidence):
    return {"check": check, "severity": severity, "url": url, "evidence": evidence}


def probe(target, extra_urls=(), timeout=10):
    origin, first_page = normalize_target(target)
    findings = []
    pages = [first_page] + [normalize_target(u)[1] for u in extra_urls]
    result = {
        "target": origin,
        "timeout": timeout,
        "robots": check_robots(origin, timeout, findings),
        "llms": {name: check_llms(origin, name, timeout, findings) for name in ("llms.txt", "llms-full.txt")},
        "pages": [check_page(u, timeout, findings) for u in pages],
    }
    findings.sort(key=lambda f: (SEVERITY_RANK[f["severity"]], f["url"], f["check"]))
    result["findings"] = findings
    result["counts"] = {s: sum(1 for f in findings if f["severity"] == s) for s in SEVERITY_RANK}
    return result


def filter_findings(findings, min_severity):
    floor = SEVERITY_RANK[min_severity]
    return [f for f in findings if SEVERITY_RANK[f["severity"]] <= floor]


def render_text(result, findings):
    out = [f"\n  geo probe: {result['target']}"]
    r = result["robots"]
    out.append(f"  robots.txt: {r['status']} (HTTP {r['http_status']})   sitemaps: {len(r['sitemaps'])}"
               f"   content-signals: {len(r['content_signals'])}")
    if r["crawlers"]:
        for token, st in r["crawlers"].items():
            via = "not mentioned" if st["status"] == "unmentioned" else f"via {st['via']}"
            rule = f"  {st['rule']}" if st["rule"] else ""
            out.append(f"    {token:<20} {st['status']:<12} {via}{rule}")
    for name, d in result["llms"].items():
        extra = f", {d['size']} B, heading: {d['first_heading'] or 'none'}, links: {d['link_count']}" if d["status"] == "present" else ""
        out.append(f"  /{name}: {d['status']} (HTTP {d['http_status']}){extra}")
    for pg in result["pages"]:
        types = sorted({t for b in pg["jsonld"] for t in b["types"]})
        bad = sum(1 for b in pg["jsonld"] if not b["ok"])
        out.append(f"  page {pg['url']} (HTTP {pg['http_status']})")
        out.append(f"    title: {pg['title']!r}   lang: {pg['lang']!r}   canonical: {pg['canonical']!r}")
        out.append(f"    meta description: {pg['meta_description']!r}")
        out.append(f"    body text: {pg['text_chars']} chars   JSON-LD blocks: {len(pg['jsonld'])}"
                   f" ({bad} unparseable)   types: {', '.join(types) or 'none'}")
    c = result["counts"]
    out.append(f"\n  findings: {len(findings)} reported   high: {c['high']}   medium: {c['medium']}   low: {c['low']}\n")
    for f in findings:
        out.append(f"  [{f['severity'].upper():<6}] {f['check']:<26} {f['url']}")
        out.append(f"           {f['evidence']}")
    if findings:
        out.append("")
    return "\n".join(out)


def main(argv=None):
    ap = argparse.ArgumentParser(description="Deterministic GEO probe: robots.txt, llms.txt, JSON-LD and head tags.")
    ap.add_argument("target", help="domain or URL; its origin is used for robots.txt and llms.txt")
    ap.add_argument("urls", nargs="*", help="additional page URLs to probe")
    ap.add_argument("--json", action="store_true", help="machine-readable output for phase two")
    ap.add_argument("--timeout", type=int, default=10, help="per-request timeout in seconds (default 10)")
    ap.add_argument("--severity", choices=["high", "medium", "low"], default="low",
                    help="minimum severity to report and gate on (default: low = everything)")
    args = ap.parse_args(argv)

    try:
        result = probe(args.target, args.urls, args.timeout)
    except ValueError as e:
        print(str(e), file=sys.stderr)
        return 2
    reported = filter_findings(result["findings"], args.severity)
    result["findings"] = reported
    result["counts"] = {s: sum(1 for f in reported if f["severity"] == s) for s in SEVERITY_RANK}
    if args.json:
        print(json.dumps(result, indent=2, ensure_ascii=False))
    else:
        print(render_text(result, reported))
    return 1 if reported else 0


if __name__ == "__main__":
    sys.exit(main())