search-visibility

{
  "name": "search-visibility",
  "description": "Two halves of getting found: a-seo-gsc turns a Google Search Console export into a prioritized action plan backed by your own data, and a-geo-optimizer makes a site citable by AI assistants (ChatGPT, Claude, Perplexity, Gemini) through crawler access, structured data, and answer-shaped content. One diagnoses from data, the other fixes what the newer search surface needs.",
  "version": "1.0.0",
  "author": {
    "name": "Ivan Misic"
  }
}

# search-visibility

Two skills for the two places people now find things: a search engine, and an AI assistant answering on their behalf. They work from opposite ends. `a-seo-gsc` starts from your measured data and tells you what to fix. `a-geo-optimizer` starts from the site and makes it possible to be cited at all.

## The two skills

| Skill | Starts from | Answers |
|---|---|---|
| `a-seo-gsc` | Your Google Search Console export | What is actually costing me traffic, and in what order do I fix it |
| `a-geo-optimizer` | The site or page itself | Can an AI assistant reach this, parse it, and quote it |

### a-seo-gsc

Point it at a Search Console export and it produces a prioritized action plan instead of a wall of rows. It covers the diagnoses that matter and that people usually miss: pages with impressions but no clicks, striking-distance queries sitting just off page one, keyword cannibalization where your own pages compete, indexing failures (`crawled - currently not indexed`, `discovered - currently not indexed`), and index bloat.

It adapts to the site type, because a 300-page blog, a 50k-SKU store, a docs site, and a local business fail in different ways. `scripts/parse_gsc.py` pre-processes the export deterministically, so the numbers come from arithmetic rather than from a model's reading of a CSV.

`references/export-guide.md` has the exact click-paths to get every report out of Search Console, including how to pull two date ranges for trend diffing, and which alternative sources work (Bing Webmaster Tools, an Ahrefs or Semrush CSV).

Your export is your own data. It stays on your machine: nothing here uploads it, phones home, or sends it anywhere.

### a-geo-optimizer

Generative engine optimization, the AI-search counterpart to SEO. Being useful is not enough if the assistant cannot fetch the page, cannot tell what it says, or cannot find a quotable claim in it.

Covers crawler access (the AI user-agents are not the Googlebot rules you already wrote), `llms.txt`, structured data aimed at citation rather than rich snippets, and content shaped so an answer can actually be lifted out of it. Use it when building a page as well as when auditing one: the cheapest time to fix this is before publishing.

## Install

```
/plugin install search-visibility@imisic
```

## Using them together

Run `a-seo-gsc` first when you have data, because it tells you where the loss actually is and stops you optimizing a page nobody was going to visit. Run `a-geo-optimizer` on a new site or page, where there is no data yet, and on anything you want quoted by an assistant.

Neither one changes code. They produce findings and plans; you decide what lands.

---
name: a-geo-optimizer
description: >
  Optimize any web project for AI search visibility (GEO/AEO). Use this skill whenever building,
  reviewing, auditing, or improving a website discoverability by AI assistants like ChatGPT, Claude,
  Gemini, Perplexity, and Copilot. Triggers include AI SEO, GEO, AEO, AI search, AI optimization,
  AI visibility, AI crawlers, llms.txt, make my site AI-friendly, optimize for ChatGPT, optimize for
  AI, AI citations, generative engine optimization, answer engine optimization, schema markup for AI,
  robots.txt for AI bots, structured data audit, content restructuring for AI, Lighthouse Agentic
  Browsing audit, WebMCP, agentic browsers, making a site agent-ready, or any mention of
  making content citable by AI tools. Also trigger when creating new websites or landing pages (to
  bake in GEO from the start), auditing existing sites, writing blog posts or service pages, building
  PHP/HTML templates, or reviewing content strategy. Even if the user does not explicitly say AI
  optimization, use this skill whenever the output is a web page or web content that would benefit
  from AI discoverability.
---

# GEO Optimizer: AI Search Visibility Skill

Make any web project discoverable and citable by AI assistants (ChatGPT, Claude, Gemini, Perplexity, Copilot). This skill covers technical implementation, content structure, and ongoing monitoring.

## When to use this skill

- **New site/page creation**: Bake GEO in from the start
- **Existing site audit**: Diagnose and fix AI visibility gaps
- **Content writing/restructuring**: Format content for AI citation
- **Template development**: Build PHP/HTML templates with GEO built-in
- **robots.txt / schema / sitemap work**: Ensure AI crawler compatibility
- **Competitive analysis**: Compare AI citation performance

## How to use this skill

### Step 1: Determine the task type

Ask yourself which mode applies:

| Mode | When | What to do |
|------|------|------------|
| **Full audit** | User wants comprehensive review | Run all checklists from `references/technical-checklist.md` |
| **New build** | Creating a site/page from scratch | Follow the build-time integration guide below |
| **Content optimization** | Writing or restructuring content | Follow `references/content-strategy.md` |
| **Quick fix** | Specific technical task (robots.txt, schema, etc.) | Jump to relevant section in `references/technical-checklist.md` |

### Step 2: For full audits, follow this sequence

1. **Technical foundation**: Read `references/technical-checklist.md` and verify:
   - robots.txt allows AI crawlers (exact user-agents in `references/crawler-reference.md`)
   - Server-side rendering (critical: most AI crawlers don't execute JS)
   - Schema.org JSON-LD markup on key pages
   - Page speed (FCP < 0.4s target)
   - XML sitemap + RSS/Atom feed
   - HTTPS, mobile-friendly
   - llms.txt file: if present it MUST pass Lighthouse validation (H1 + markdown `[text](url)` links, see checklist section 5); a malformed one is worse than none because it fails a visible audit
   - Agentic Browsing readiness (checklist section 13): run PageSpeed Insights and read the Agentic Browsing category; agent-critical a11y (programmatic names, valid ARIA, touch targets) and CLS now double as agent-readiness signals. WebMCP only matters for interactive sites (search, cart, booking)
   - If behind Cloudflare: fetch the LIVE robots.txt and check for injected Content-Signal directives (`ai-train=no` is the managed default; `ai-input=no` would hurt AI citation, see crawler-reference section 12)

2. **Content structure**: Read `references/content-strategy.md` and evaluate:
   - Answer-capsule format (40-60 word direct answer after each heading)
   - Front-loaded key content (44% of AI citations come from first 30% of page)
   - Section length (120-180 words per self-contained section)
   - Statistical density (one verifiable stat per 150-200 words)
   - Question-style headings matching conversational queries
   - FAQ sections with real user questions
   - Author bios with credentials

3. **Entity and authority signals**: Check:
   - Organization schema with consistent NAP (Name, Address, Phone)
   - Author schema with credentials and external profile links
   - Third-party presence (review sites, directories, Reddit, YouTube)
   - Internal linking structure (pillar-cluster architecture)

4. **Monitoring setup**: Recommend:
   - GA4 AI traffic segment (regex for AI referral sources)
   - Monthly prompt testing across AI platforms
   - GEO monitoring tool appropriate to budget

### Step 3: For new builds, integrate GEO at template level

When building PHP/HTML templates, include these by default:

**In the `<head>` of every page:**
```html
<!-- JSON-LD schema block (populated per page type) -->
<script type="application/ld+json" id="schema-jsonld">
<!-- Populated server-side per template -->
</script>

<!-- Markdown alternate for long-form content (optional) -->
<link rel="alternate" type="text/markdown" href="<?= $markdown_url ?>" />

<!-- Standard meta for AI extraction -->
<meta name="description" content="<?= htmlspecialchars($meta_description) ?>" />
<meta name="author" content="<?= htmlspecialchars($author_name) ?>" />
<meta property="article:published_time" content="<?= $published_date ?>" />
<meta property="article:modified_time" content="<?= $modified_date ?>" />
```

**Content templates should enforce:**
- Single H1 per page
- Logical H2/H3 hierarchy
- First paragraph = direct answer to the page's core question
- Visible "Last Updated" date reflecting genuine content changes
- Author bio block with photo, credentials, external links

**Server-side requirements:**
- All content rendered server-side (SSR), not client-side JS
- JSON-LD generated server-side per page template
- Clean semantic HTML (proper heading hierarchy, lists, tables)
- Fast TTFB and FCP (< 0.4s target)

### Step 4: Output deliverables

Depending on the task, produce one or more of:

- **robots.txt**: Ready to deploy, with all AI crawlers allowed (see `references/crawler-reference.md` for exact user-agents)
- **llms.txt / llms-full.txt** (optional, low-medium priority): Markdown site overview, plus an optional full-content companion. As of 2026-07 adoption as a retrieval source is still uneven: AI dev tools (Claude Code, Cursor, Copilot, Windsurf, Cline, Aider) fetch both and some providers (Anthropic, Perplexity) confirm support, but Google doesn't use it for Search and no major provider has committed to it in production answer systems. What changed in 2026: Lighthouse/PageSpeed Insights now validates the file (Agentic Browsing category), so if one exists it MUST follow the spec format (H1 required; links as markdown `- [name](url): note` bullets, never plain `Title: URL` lines; no `---` rules that get parsed as front-matter). Generate it when trivial alongside other work, notably when the site already exposes per-page markdown, since a route can then concatenate it cheaply. Don't prioritize it over robots.txt, schema, sitemap, or content.
- **Schema JSON-LD blocks**: Per page type (Article, Product, FAQPage, Organization, etc.)
- **Content restructured**: In answer-capsule format with front-loaded answers
- **Audit report**: Findings and priority-ordered recommendations
- **PHP template code**: With GEO elements built into the template layer
- **Sitemap + RSS/Atom feed**: If missing or incomplete
- **GA4 configuration**: AI referral traffic tracking setup

## Key principles to always follow

1. **SSR is non-negotiable**: If content requires JavaScript to render, AI crawlers (except Googlebot) cannot see it. Always flag this.

2. **Front-load answers**: The most important content goes in the first 30% of the page. AI systems extract from the top.

3. **Be specific, not promotional**: AI systems filter out sales copy. Use definitive, factual language with specific numbers, dates, and names. Entity density matters.

4. **Earned media > owned content**: Third-party mentions are 6.5× more likely to generate AI citations than self-published content. Always recommend off-site authority building.

5. **Freshness is a hard requirement**: Content updated within 30 days accounts for 76% of AI citations. Recommend refresh cycles.

6. **Cross-platform optimization**: Only 11-14% of citations overlap between platforms. Optimize for all major AI systems, not just one.

7. **Don't block crawlers by accident**: Always check the LIVE robots.txt for inherited CMS/CDN blocks on AI user-agents, injected Content-Signal directives (Cloudflare's managed robots.txt defaults to `ai-train=no` on 3.8M+ domains), and CDN-level bot blocking that fires before robots.txt is read (test with `curl -A "GPTBot"`).

8. **Agent-readiness is the new layer**: Lighthouse's Agentic Browsing category (2026) grades llms.txt format, WebMCP tools, agent-critical accessibility, and CLS. Agentic browsers (OpenAI Atlas, Perplexity Comet, Gemini in Chrome) drive real sessions through the rendered page and a11y tree, so accessibility fixes now double as agent task-completion fixes.

## Reference files

- `references/technical-checklist.md`: Complete technical implementation checklist (robots.txt, schema, sitemap, llms.txt, page speed, rendering)
- `references/content-strategy.md`: Content formatting, answer-capsule structure, citation optimization, topic clustering
- `references/crawler-reference.md`: All AI crawler user-agents, which platforms they serve, robots.txt templates, and platform-specific submission guides

# Content Strategy for AI Citations

How to structure, write, and maintain content that AI systems extract, cite, and recommend.

## Table of Contents

1. [The Answer-Capsule Format](#1-the-answer-capsule-format)
2. [Front-Loading Critical Content](#2-front-loading-critical-content)
3. [Section Structure and Length](#3-section-structure-and-length)
4. [Heading Strategy](#4-heading-strategy)
5. [Statistical Density and Source Attribution](#5-statistical-density-and-source-attribution)
6. [Entity Density](#6-entity-density)
7. [Language and Tone](#7-language-and-tone)
8. [FAQ Sections](#8-faq-sections)
9. [Comparison Tables](#9-comparison-tables)
10. [Content Formats That Get Cited](#10-content-formats-that-get-cited)
11. [Author Authority Signals](#11-author-authority-signals)
12. [Content Freshness](#12-content-freshness)
13. [Topic Cluster Architecture](#13-topic-cluster-architecture)
14. [Off-Site Authority Building](#14-off-site-authority-building)
15. [Intent Modifier Phrases](#15-intent-modifier-phrases)
16. [Brand Sentiment Auditing](#16-brand-sentiment-auditing)
17. [What NOT to Do](#17-what-not-to-do)

---

## 1. The Answer-Capsule Format

The core content pattern for AI citation. Every major section should follow this structure:

```
[Question-style H2 heading]

[40-60 word direct answer, the "capsule"]

[Supporting detail, evidence, examples, 80-120 additional words]

[Optional: data table, comparison, or specific example]
```

**Example (good):**
```markdown
## How long does it take to set up a Synology NAS?

A basic Synology NAS setup takes 15-30 minutes for hardware assembly 
and initial DSM installation. Full configuration including RAID setup, 
shared folders, user accounts, and package installation typically 
requires 1-2 hours. Advanced setups with Docker containers, reverse 
proxies, and automated backups can take a full day.

The process breaks down into three phases. First, physical assembly: 
insert drives, connect cables, power on. Second, DSM installation 
via find.synology.com or Synology Assistant. Third, configuration 
of storage pools, shared folders, and services.
```

**Example (bad, buries the answer):**
```markdown
## NAS Setup Guide

Network Attached Storage devices have become increasingly popular 
in recent years. There are many brands to choose from, each with 
their own strengths. In this guide, we'll walk through everything 
you need to know about setting up your device. But first, let's 
understand what a NAS actually is...

[answer appears 500 words later]
```

---

## 2. Front-Loading Critical Content

**44.2% of all LLM citations come from the first 30% of a page's text.**

Rules:
- The most important, quotable, citable content goes at the top
- First paragraph after H1 should answer the page's core question
- No lengthy introductions, no "in this article we will discuss..."
- Get to the point immediately, then expand

For product/service pages: Lead with what it is, who it's for, and what problem it solves. Pricing and key differentiators in the first 3 paragraphs.

For blog posts: Lead with the answer or key finding. Background and methodology come after.

---

## 3. Section Structure and Length

**Optimal section length: 120-180 words per self-contained section.**

Pages using this section length receive 70% more ChatGPT citations than pages with sections under 50 words.

Each section should be:
- Self-contained (extractable without needing context from other sections)
- Focused on one specific sub-question
- Bounded by a descriptive heading
- Complete enough to stand alone as an answer

Articles over 2,900 words average 5.1 citations vs 3.2 for content under 800 words. Longer is better, but only if each section adds genuine value.

---

## 4. Heading Strategy

Use question-style headings that mirror how people ask AI assistants.

**Good headings (match conversational queries):**
- "How does [X] compare to [Y]?"
- "What are the pricing options for [service]?"
- "How long does [process] take?"
- "What's the best [tool] for [use case]?"
- "Is [product] worth it in 2026?"

**Bad headings (traditional SEO style, hard for AI to map to queries):**
- "Features"
- "Overview"
- "Technical Specifications"  
- "Our Approach"
- "More Information"

Also: ChatGPT internally appends intent modifiers like "detailed guide", "step-by-step tutorial", "in-depth comparison" to its search queries. Including these phrases naturally in your headings or opening paragraphs can improve retrieval.

---

## 5. Statistical Density and Source Attribution

**Content with verifiable claims receives 44% more citations. The original GEO research found citing sources increased visibility by 115.1%.**

Rules:
- Include one verifiable statistic per 150-200 words
- Always attribute statistics to their source
- Use specific numbers, not vague claims ("3.2× faster" not "much faster")
- Include dates with stats to signal freshness
- Prefer original data you've collected over recycled third-party stats

**Good:** "According to our 2026 client survey, 73% of small businesses in Croatia report spending less than 2 hours per week on social media marketing."

**Bad:** "Many businesses find social media marketing time-consuming."

---

## 6. Entity Density

**Content with high entity density shows 20.6% higher citation rates.**

Entities = proper nouns, specific names, dates, places, product names, company names, standards, technologies.

- Use full, specific names on first mention (not just pronouns)
- Include dates, version numbers, specific locations
- Reference specific standards, frameworks, or methodologies by name
- Mention specific tools, platforms, or competitors by name when relevant

**High entity density:** "Zagreb-based agencies using Figma and Webflow in 2026 report 40% faster delivery times compared to agencies still using WordPress page builders."

**Low entity density:** "Agencies using modern design tools report faster delivery times than those using older platforms."

---

## 7. Language and Tone

AI systems prefer **definitive, authoritative language** over hedging.

**Preferred:**
- "The best approach is..." 
- "This costs €500-800 per month"
- "Setup takes 2-3 business days"
- "The main drawback is..."

**Avoided by AI systems:**
- "It depends on various factors..."
- "There are many considerations to keep in mind..."
- "Results may vary..."
- "Contact us to learn more" (dead-end for AI)

**Critical:** AI systems filter out overtly promotional or sales-heavy content. Write as a knowledgeable advisor, not a salesperson. Neutral, factual language with specific claims wins citations.

---

## 8. FAQ Sections

FAQ sections with question-style headings nearly double citation chances. AI systems extract Q&A pairs directly, especially when paired with FAQPage schema.

**Requirements:**
- Use actual questions your customers/readers ask
- Match the conversational phrasing people use with AI ("How do I..." not "Methodology for...")
- Each answer: 40-80 words, direct and complete
- 5-10 FAQs per page is the sweet spot
- Pair with FAQPage JSON-LD schema (see technical-checklist.md)

**Good FAQ structure:**
```markdown
## Frequently Asked Questions

### How much does a website redesign cost in Croatia?

A professional website redesign in Croatia typically costs €2,000-8,000 
for small business sites, €8,000-25,000 for mid-size corporate sites, 
and €25,000+ for complex e-commerce platforms. Pricing depends on page 
count, custom functionality, and whether content migration is included.

### How long does a typical web project take?

Most small business websites take 4-8 weeks from kickoff to launch. 
Mid-size projects run 8-16 weeks. Enterprise-level builds with custom 
integrations can extend to 3-6 months.
```

---

## 9. Comparison Tables

**Comparison tables deliver 2.5× more citations than text-only equivalents.**

Use HTML tables or Markdown tables for:
- Product vs. competitor comparisons
- Feature comparisons across tiers/plans
- Pros vs. cons
- Technology comparisons
- Before/after scenarios

Keep tables clean and data-rich. AI systems extract structured relationships from tables better than from prose.

---

## 10. Content Formats That Get Cited

By citation frequency (ChatGPT data):

| Format | % of Citations | Notes |
|--------|---------------|-------|
| "Best X" listicles | 43.8% | Highest citation rate |
| How-to guides | ~25% | Especially with step-by-step structure |
| Comparison articles | ~15% | X vs Y format |
| Definition/explainer | ~10% | "What is X" format |
| Case studies | ~5% | With specific metrics and outcomes |

Prioritize creating "Best X for Y" content for your niche. These are the most-cited format by a wide margin.

---

## 11. Author Authority Signals

Cited pages average 4.1 expert quotations vs 2.4 on non-cited pages.

For every content page, include:
- Author name and photo
- Professional title/role
- Years of experience or credentials
- Links to LinkedIn, professional profiles, or personal site
- Brief description of relevant expertise

For the Organization/brand:
- Clear "About Us" page with founding date, team, and mission
- Google Business Profile (fully optimized)
- Consistent Name, Address, Phone (NAP) across all platforms
- Wikidata entry (for entity recognition by AI)

---

## 12. Content Freshness

**Content updated within 30 days accounts for 76.4% of all AI citations.**

Rules:
- Establish a 60-90 day refresh cycle for key pages
- Update with substantive changes (not just timestamp bumps)
- Pages with genuine updates earn 3.8× more citations than cosmetic refreshes
- Display a visible "Last Updated: [date]" on the page
- The `dateModified` in your Article schema must match the visible date

---

## 13. Topic Cluster Architecture

**82.5% of ChatGPT citations link to pages within established topic hierarchies.**

Structure:
```
Pillar Page: "Complete Guide to [Topic]" (2,000-4,000 words)
├── Cluster: "How to [subtopic A]" (1,200-2,000 words)
├── Cluster: "Best [tools] for [subtopic B]" (1,500-2,500 words)
├── Cluster: "[Topic] vs [Alternative]" (1,000-1,500 words)
├── Cluster: "[Topic] for [specific audience]" (1,200-2,000 words)
└── Cluster: "Common [topic] mistakes and how to fix them" (1,000-1,500 words)
```

Rules:
- Pillar links to all cluster pages
- Each cluster page links back to pillar
- Cluster pages cross-link to related clusters
- Use descriptive anchor text
- Bidirectional internal linking delivers 2.7× citation multiplier

---

## 14. Off-Site Authority Building

**Brands are 6.5× more likely to be cited through third-party mentions than their own website.**

Priority platforms for AI citations:
1. **YouTube**: 39.2% of social citations in AI Overviews. Create demo videos, tutorials, case studies.
2. **Reddit**: 40.1% of AI responses reference Reddit. Build authentic presence in relevant subreddits.
3. **Review platforms**: G2, Clutch, Capterra, Trustpilot provide 4.6-6.3× citation multiplier.
4. **Industry comparison articles**: Get listed in "best [service] in [location/niche]" articles.
5. **Guest posts**: Contribute to industry publications with expert commentary.
6. **Wikipedia/Wikidata**: Create or update entity entries for your brand/key people.
7. **Google Business Profile**: Primary data source for local AI recommendations.

Domains with active profiles across 4+ third-party platforms see 2.8× citation likelihood increase.

---

## 15. Intent Modifier Phrases

ChatGPT appends modifier phrases to internal search queries when users ask questions. Including these naturally in your content improves retrieval:

- "detailed guide"
- "step-by-step tutorial"
- "in-depth comparison"
- "comprehensive overview"
- "complete guide"
- "how to [specific task]"
- "best [thing] for [use case]"
- "[thing] vs [alternative]"
- "pricing", "cost", "how much"
- "pros and cons"
- "review" / "honest review"

Don't stuff these artificially. Use them in headings and opening paragraphs where they fit naturally.

---

## 16. Brand Sentiment Auditing

Regularly test what AI tools say about your brand:

1. Ask each major AI tool: "Give me pros and cons of [Your Brand]"
2. Ask: "What are the best [your service type] companies in [your location]?"
3. Ask: "Compare [Your Brand] vs [Competitor]"
4. Document what's accurate, what's wrong, what's missing

If AI is hallucinating facts about you:
- Update your About page with clear, declarative sentences
- Update llms.txt with correct information
- Add specific factual claims to schema markup
- Build corrective third-party content (reviews, profiles, mentions)

AI systems pick up corrections faster from multiple independent sources than from your own site alone.

---

## 17. What NOT to Do

**Content that gets ignored or filtered by AI:**

- **Sales copy and promotional language**: AI filters out overtly commercial content
- **Vague, hedging language**: "It depends" answers get skipped in favor of specific ones
- **Thin content under 500 words**: Rarely provides enough value for citation
- **AI-generated content at scale without editorial enhancement**: Produces patterns AI learns to skip
- **Keyword-stuffed content**: The original GEO research showed keyword stuffing produced zero gains
- **Content with no original information**: If AI can generate the same content itself, it won't cite you
- **Content behind login walls**: AI crawlers cannot access paywalled content
- **Information scattered across multiple thin pages**: Consolidate into comprehensive single pages
- **Timestamp-only refreshes**: Changing the date without substantive updates doesn't fool AI systems
- **Ignoring a platform**: Only 11-14% of citations overlap between platforms; optimize for all

**The Chegg lesson:** If your content is commodity information that AI can generate independently, you'll be displaced. Content must include what AI cannot replicate: original data, proprietary research, real case studies with specific metrics, expert quotes from named professionals, and first-person experience.

# AI Crawler Reference

Complete reference of AI crawler user-agents, which platforms they serve, and how to configure access.

## Table of Contents

1. [Crawler Quick Reference Table](#1-crawler-quick-reference-table)
2. [OpenAI Crawlers](#2-openai-crawlers)
3. [Anthropic (Claude) Crawlers](#3-anthropic-claude-crawlers)
4. [Google AI Crawlers](#4-google-ai-crawlers)
5. [Perplexity Crawlers](#5-perplexity-crawlers)
6. [Microsoft/Bing Crawlers](#6-microsoftbing-crawlers)
7. [Other AI Crawlers](#7-other-ai-crawlers)
8. [Complete robots.txt Template](#8-complete-robotstxt-template)
9. [Platform Submission Guide](#9-platform-submission-guide)
10. [GA4 AI Traffic Tracking](#10-ga4-ai-traffic-tracking)
11. [Agentic Browsers](#11-agentic-browsers)
12. [Content Signals in robots.txt](#12-content-signals-in-robotstxt)

---

## 1. Crawler Quick Reference Table

| User-Agent | Company | Purpose | Block = |
|-----------|---------|---------|---------|
| `GPTBot` | OpenAI | Training data collection | Less model familiarity |
| `OAI-SearchBot` | OpenAI | Search index for ChatGPT Search | Invisible in ChatGPT search |
| `ChatGPT-User` | OpenAI | Real-time user-triggered fetch | May not respect robots.txt |
| `ClaudeBot` | Anthropic | Training data collection | Less model familiarity |
| `Claude-SearchBot` | Anthropic | Search index for Claude web search | Invisible in Claude search |
| `Claude-User` | Anthropic | Real-time user-triggered fetch | May not respect robots.txt |
| `Googlebot` | Google | Search index (also feeds AI Overviews) | No Google visibility at all |
| `Google-Extended` | Google | Gemini training data only | No Gemini training (AI Overviews unaffected) |
| `PerplexityBot` | Perplexity | Search index + real-time crawl | Invisible in Perplexity |
| `Bingbot` | Microsoft | Bing index (feeds ChatGPT + Copilot) | No Bing/ChatGPT/Copilot visibility |

**Key insight:** For maximum AI search visibility, you must allow at minimum: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot, and Bingbot. The training crawlers (GPTBot, ClaudeBot, Google-Extended) are optional but recommended.

---

## 2. OpenAI Crawlers

**GPTBot**
- Purpose: Collects web content for training OpenAI models
- User-agent string: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)`
- Respects: robots.txt
- Impact of blocking: 73% fewer ChatGPT citations (correlation, not direct causation)

**OAI-SearchBot**
- Purpose: Indexes content for ChatGPT's web search feature
- User-agent string: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)`
- Respects: robots.txt
- Impact of blocking: Removed from ChatGPT search results

**ChatGPT-User**
- Purpose: Real-time page fetching when a user's conversation triggers web lookup
- User-agent string: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)`
- Respects: robots.txt partially (operates on behalf of human user)
- Impact of blocking: Reduced but not eliminated ChatGPT access

---

## 3. Anthropic (Claude) Crawlers

**ClaudeBot**
- Purpose: Collects web content for training Claude models
- User-agent string: `claudebot` (case variations exist)
- Respects: robots.txt (including non-standard Crawl-delay directive)
- Impact of blocking: Less model familiarity with your content

**Claude-SearchBot**
- Purpose: Indexes content for Claude's web search (via Brave Search supplementation)
- User-agent string: `Claude-SearchBot`
- Respects: robots.txt
- Impact of blocking: Reduced visibility in Claude web search

**Claude-User**
- Purpose: Real-time fetching during user conversations with web search enabled
- User-agent string: `Claude-User`
- Respects: robots.txt partially (operates on behalf of human user)
- Impact of blocking: Reduced but not eliminated Claude access

**Note:** Claude's search primarily uses Brave Search's independent index. Ensure your site is visible in Brave Search for maximum Claude visibility.

---

## 4. Google AI Crawlers

**Googlebot**
- Purpose: Google's main search crawler. Feeds both traditional search AND AI Overviews.
- User-agent string: Various (Googlebot, Googlebot-Mobile, etc.)
- Respects: robots.txt
- Impact of blocking: Complete loss of Google search AND AI Overview visibility
- **Critical:** You CANNOT block AI Overviews without losing all Google search visibility. There is no separate AI Overview crawler.

**Google-Extended**
- Purpose: Controls training data for Gemini and Vertex AI products
- User-agent string: `Google-Extended`
- Respects: robots.txt
- Impact of blocking: Blocks Gemini training. Does NOT affect AI Overviews or Google Search.
- **Note:** Blocking Google-Extended is a viable choice if you want search visibility but don't want to contribute to Gemini training.

---

## 5. Perplexity Crawlers

**PerplexityBot**
- Purpose: Indexes content for Perplexity's AI search engine
- User-agent string: `PerplexityBot`
- Respects: robots.txt
- Impact of blocking: Invisible in Perplexity search results

Perplexity maintains its own independent index. It's the most citation-transparent platform (numbered inline citations in every response). Reddit content dominates at 46.5% of Perplexity citations.

**Perplexity Publishers' Program:** Partnership program with revenue sharing (80/20 split favoring publishers). Currently 30+ partners including TIME, Fortune, Der Spiegel.

---

## 6. Microsoft/Bing Crawlers

**Bingbot**
- Purpose: Bing's main search crawler. Feeds Bing Search, ChatGPT (via Bing), and Microsoft Copilot.
- User-agent string: `Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)`
- Respects: robots.txt
- Impact of blocking: Complete loss of Bing, ChatGPT (Bing-backed), and Copilot visibility

**Critical:** Bingbot is the single most impactful crawler for ChatGPT visibility. Industry analyses have reported ~87% overlap between ChatGPT citations and Bing's top results (figure undated, verify against a current source before quoting). If you do nothing else, make sure Bingbot can access your site and you've submitted to Bing Webmaster Tools.

**Bing Content Submission API:** Enables pushing structured facts directly for Copilot sidebar answers.

---

## 7. Other AI Crawlers

| User-Agent | Company | Purpose |
|-----------|---------|---------|
| `cohere-ai` | Cohere | Training/search |
| `Bytespider` | ByteDance | Training for AI products |
| `Applebot` | Apple | Apple Intelligence/Siri |
| `Applebot-Extended` | Apple | Apple Intelligence training |
| `FacebookBot` | Meta | Meta AI features |
| `meta-externalagent` | Meta | Meta AI training |
| `Amazonbot` | Amazon | Alexa/AI features |
| `Timesbot` | Perplexity (for publisher partners) | Premium content |

---

## 8. Complete robots.txt Template

### Maximum AI Visibility (recommended)

```
# Standard search
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# AI Search (required for AI search visibility)
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# AI Training (recommended for maximum citation likelihood)
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /
Crawl-delay: 10

User-agent: Google-Extended
Allow: /

# Apple Intelligence
User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Allow: /

# Blocked paths (customize per site)
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /tmp/
Disallow: /private/
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/

Sitemap: https://yoursite.com/sitemap.xml
```

### Selective (search only, no training)

```
# Standard search
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# AI Search only
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Block AI training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://yoursite.com/sitemap.xml
```

---

## 9. Platform Submission Guide

### Google Search Console
- URL: https://search.google.com/search-console
- Submit: XML sitemap
- Feeds: Google Search + AI Overviews + Gemini
- Monitor: Performance report (AI Overview data merged under "Web" search type)

### Bing Webmaster Tools
- URL: https://www.bing.com/webmasters
- Submit: XML sitemap + implement IndexNow
- Feeds: Bing + ChatGPT + Microsoft Copilot
- **This is the #1 priority submission for ChatGPT visibility**

### Brave Search
- Brave uses its own independent index (Web Discovery Project)
- No manual submission process
- Focus on standard SEO + allowing Brave's crawler
- Claude's search draws from Brave's index

### Perplexity
- No manual submission
- Allow PerplexityBot in robots.txt
- Perplexity's own index discovers content independently
- Publishers' Program for partnership: https://www.perplexity.ai/hub/partnerships

### Google Business Profile
- URL: https://business.google.com
- Critical for local/service businesses
- Primary data source for AI local recommendations
- Complete all fields: services, hours, photos, Q&A, posts

---

## 10. GA4 AI Traffic Tracking

### Create an AI referral traffic segment

In GA4, go to Explore > create a custom segment with traffic source filter:

**Regex for AI referral sources:**
```
perplexity\.ai|chatgpt\.com|chat\.openai\.com|claude\.ai|gemini\.google\.com|copilot\.microsoft\.com|you\.com|phind\.com
```

Apply this to:
- Session source/medium (for full session data)
- Landing page (to see which pages AI tools are sending traffic to)

### Track AI-influenced conversions

Add "How did you find us?" to lead forms with options:
- ChatGPT
- Claude
- Perplexity
- Google AI Overview
- Other AI tool

This captures dark AI traffic (users who get a recommendation from AI but then navigate directly to your site).

### Isolate likely AI Overview impressions in Google Search Console

No direct filter exists. Workarounds:
- Filter by query length (10+ words)
- Filter by conversational keywords: "compare", "top", "best", "vs", "how to", "what is"
- These queries are more likely to trigger AI Overviews

---

## 11. Agentic Browsers

A category that matters since late 2025: full browsers where an AI agent drives real user sessions. These are NOT crawlers and mostly cannot be managed via robots.txt.

| Product | Company | Notes |
|-|-|-|
| Atlas | OpenAI | Launched October 2025. Agent Mode executes multi-step tasks autonomously |
| Comet | Perplexity | Chromium-based; traffic looks like a normal Chrome session |
| Gemini in Chrome | Google | The main consumer of WebMCP tools during the origin trial |

**Implications for GEO:**
- Their traffic is largely indistinguishable from human sessions in analytics; you cannot segment it with a user-agent regex.
- They interact with your site through the rendered page and the accessibility tree, which is why Lighthouse's Agentic Browsing category audits a11y and CLS (see technical-checklist section 13).
- Sites that work well for keyboard/screen-reader users work well for agentic browsers. Broken forms, unlabeled buttons, and layout shift break agent task completion (and the sale that came with it).

---

## 12. Content Signals in robots.txt

Cloudflare's Content Signals Policy (September 2025) extends robots.txt with usage-category signals, expressed per user-agent group:

```
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
```

The three signals: `search` (build a search index), `ai-input` (feed content into AI answers at query time, i.e. RAG/grounding), `ai-train` (training or fine-tuning). The policy text frames restrictions as an express reservation of rights under EU DSM Directive 2019/790 Article 4.

**Why a GEO skill cares:**
- **Cloudflare's managed robots.txt injects `search=yes, ai-train=no` for 3.8M+ domains.** If the site sits behind Cloudflare, fetch the LIVE robots.txt and check whether a managed policy has been layered on top of (or replaces) the site's own file. A client maximizing AI visibility may be sending "don't train on me" without knowing it.
- For maximum AI visibility, either omit content signals entirely or set them permissively. `ai-input=no` works against citation in AI answers.
- Same audit lesson as user-agents: the robots.txt you deploy is not always the robots.txt the world sees. CDN-level bot management (Cloudflare Bot Fight Mode, WAF rules, security plugins) can also block AI crawlers before robots.txt is ever read; test with `curl -A "GPTBot" https://site.com/` and check for 403s.

# Technical Checklist for AI Search Visibility

Complete technical implementation guide. Work through sequentially for new builds, or use as an audit checklist for existing sites.

## Table of Contents

1. [Server-Side Rendering](#1-server-side-rendering)
2. [Robots.txt Configuration](#2-robotstxt-configuration)
3. [Schema.org Structured Data](#3-schemaorg-structured-data)
4. [XML Sitemap + RSS/Atom Feed](#4-xml-sitemap--rssatom-feed)
5. [llms.txt File](#5-llmstxt-file)
6. [Page Speed Optimization](#6-page-speed-optimization)
7. [Search Console Submissions](#7-search-console-submissions)
8. [IndexNow Implementation](#8-indexnow-implementation)
9. [Markdown Alternate Links](#9-markdown-alternate-links)
10. [Meta Tags for AI Extraction](#10-meta-tags-for-ai-extraction)
11. [HTTPS and Security](#11-https-and-security)
12. [Internal Linking Architecture](#12-internal-linking-architecture)
13. [Lighthouse Agentic Browsing Audit](#13-lighthouse-agentic-browsing-audit)

---

## 1. Server-Side Rendering

**Priority: CRITICAL. This is a binary gate**

Most AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) do NOT execute JavaScript. Analysis of 500M+ GPTBot fetches showed zero JS execution. Only Googlebot reliably renders JS.

**Audit check:**
```bash
# Test what a non-JS crawler sees
curl -s -A "GPTBot" https://yoursite.com/page | grep -c "your-expected-content"
# If 0 matches, your content is invisible to AI crawlers
```

**For PHP sites:** This is rarely an issue since PHP renders server-side. But check for:
- Content loaded via AJAX after page load
- React/Vue/Angular widgets embedded in PHP pages
- Lazy-loaded content that requires scroll events
- Content behind JavaScript tabs/accordions (use `<details>`/`<summary>` instead)

**For JS frameworks (React, Vue, Angular, Next.js, Nuxt):**
- Enable SSR or Static Site Generation (SSG)
- Use `getServerSideProps` (Next.js) or equivalent
- Pre-render critical pages at minimum

---

## 2. Robots.txt Configuration

**Priority: CRITICAL. Blocking crawlers = invisible to AI**

See `crawler-reference.md` for complete user-agent list. Here's the recommended robots.txt template:

```
# === Standard Search Crawlers ===
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# === AI Search Crawlers (allow for search visibility) ===
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# === AI Training Crawlers (optional, allow if you want to be in training data) ===
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

# === Block paths that shouldn't be crawled ===
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /tmp/
Disallow: /private/

Sitemap: https://yoursite.com/sitemap.xml
```

**Important distinctions:**
- **Search crawlers** (OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, PerplexityBot): These power real-time search features. Blocking them removes you from AI search results.
- **Training crawlers** (GPTBot, ClaudeBot, Google-Extended): These collect data for model training. Blocking them doesn't affect real-time search but reduces long-term model familiarity with your content. Sites blocking GPTBot were cited 73% less often.
- ChatGPT-User and Claude-User operate on behalf of human users and may not fully respect robots.txt blocks.

**Audit check:**
```bash
# Fetch and inspect current robots.txt
curl -s https://yoursite.com/robots.txt
# Look for blanket "Disallow: /" rules or blocks on AI user-agents
```

---

## 3. Schema.org Structured Data

**Priority: HIGH. GPT-4 accuracy jumps from 16% to 54% with structured data**

All schema must be:
- In JSON-LD format (Google-recommended, easiest for AI to parse)
- Rendered server-side in the HTML `<head>` (not injected via JS)
- Accurately matching visible page content (mismatches reduce trust)

### Schema templates by page type

**Organization (site-wide, on every page):**
```json
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Your Company Name",
  "url": "https://yoursite.com",
  "logo": "https://yoursite.com/logo.png",
  "description": "One-sentence description of what you do",
  "foundingDate": "2020",
  "address": {
    "@type": "PostalAddress",
    "streetAddress": "123 Main St",
    "addressLocality": "Zagreb",
    "addressCountry": "HR"
  },
  "contactPoint": {
    "@type": "ContactPoint",
    "email": "info@yoursite.com",
    "contactType": "customer service"
  },
  "sameAs": [
    "https://www.linkedin.com/company/yourcompany",
    "https://www.facebook.com/yourcompany",
    "https://twitter.com/yourcompany"
  ]
}
```

**Article / BlogPosting:**
```json
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Article Title Here",
  "description": "2-sentence summary of what this article covers",
  "author": {
    "@type": "Person",
    "name": "Author Name",
    "url": "https://yoursite.com/about/author-name",
    "jobTitle": "Author's Role/Title",
    "sameAs": ["https://linkedin.com/in/author"]
  },
  "publisher": {
    "@type": "Organization",
    "name": "Your Company Name",
    "logo": { "@type": "ImageObject", "url": "https://yoursite.com/logo.png" }
  },
  "datePublished": "2026-03-01",
  "dateModified": "2026-03-20",
  "mainEntityOfPage": "https://yoursite.com/blog/article-slug",
  "image": "https://yoursite.com/images/article-hero.jpg"
}
```

**FAQPage (powerful for AI extraction):**
```json
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "What is [topic]?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Direct, complete answer in 40-80 words."
      }
    },
    {
      "@type": "Question",
      "name": "How does [topic] compare to [alternative]?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Specific comparison with concrete details."
      }
    }
  ]
}
```

**Product (for e-commerce/service pages):**
```json
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Product Name",
  "description": "Clear product description",
  "image": "https://yoursite.com/images/product.jpg",
  "brand": { "@type": "Brand", "name": "Brand Name" },
  "sku": "SKU-12345",
  "offers": {
    "@type": "Offer",
    "price": "99.00",
    "priceCurrency": "EUR",
    "availability": "https://schema.org/InStock",
    "url": "https://yoursite.com/products/product-name"
  },
  "aggregateRating": {
    "@type": "AggregateRating",
    "ratingValue": "4.5",
    "reviewCount": "42"
  }
}
```

**HowTo (for tutorials/guides):**
```json
{
  "@context": "https://schema.org",
  "@type": "HowTo",
  "name": "How to [do thing]",
  "description": "Brief overview",
  "step": [
    {
      "@type": "HowToStep",
      "name": "Step 1 title",
      "text": "Step 1 instructions"
    }
  ]
}
```

**LocalBusiness (for agency/local service sites):**
```json
{
  "@context": "https://schema.org",
  "@type": "ProfessionalService",
  "name": "Agency Name",
  "description": "What you do in one sentence",
  "url": "https://yoursite.com",
  "address": { ... },
  "geo": { "@type": "GeoCoordinates", "latitude": "45.815", "longitude": "15.982" },
  "openingHours": "Mo-Fr 09:00-17:00",
  "priceRange": "$$",
  "areaServed": "Croatia"
}
```

### PHP implementation pattern

```php
// In your base template or header include
function render_schema_jsonld(array $schema_data): string {
    return '<script type="application/ld+json">' . 
           json_encode($schema_data, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE | JSON_PRETTY_PRINT) . 
           '</script>';
}

// Per page template, build the schema array and render in <head>
$schema = [
    '@context' => 'https://schema.org',
    '@type' => 'Article',
    'headline' => $article->title,
    'datePublished' => $article->published_at,
    'dateModified' => $article->updated_at,
    // ... etc
];
echo render_schema_jsonld($schema);
```

---

## 4. XML Sitemap + RSS/Atom Feed

**Priority: HIGH**

**XML Sitemap requirements:**
- Include all indexable pages
- Accurate `<lastmod>` timestamps (critical for AI crawlers detecting fresh content)
- Split by content type if site is large (blog-sitemap.xml, product-sitemap.xml)
- Submit to both Google Search Console AND Bing Webmaster Tools
- Reference in robots.txt: `Sitemap: https://yoursite.com/sitemap.xml`

**RSS/Atom feed:**
- Google recommends pairing sitemaps with RSS/Atom feeds for change detection
- Microsoft's NLWeb protocol uses RSS feeds as a primary data source
- Include at minimum: title, link, description, pubDate/updated, author
- Auto-generate from your CMS or build a `/feed.xml` endpoint

```xml
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Site Name</title>
    <link>https://yoursite.com</link>
    <description>Site description</description>
    <atom:link href="https://yoursite.com/feed.xml" rel="self" type="application/rss+xml"/>
    <item>
      <title>Article Title</title>
      <link>https://yoursite.com/blog/slug</link>
      <description>First 200 chars of article</description>
      <pubDate>Wed, 20 Mar 2026 10:00:00 +0000</pubDate>
      <guid>https://yoursite.com/blog/slug</guid>
    </item>
  </channel>
</rss>
```

---

## 5. llms.txt File

**Priority: LOW-MEDIUM (uneven adoption as a retrieval source, but since 2026 Lighthouse/PageSpeed Insights validates the file, so a malformed one shows up as a red audit to anyone who tests your site)**

Place at `https://yoursite.com/llms.txt`. Markdown-formatted overview of your site for AI consumption. Takes minutes to create, costs nothing.

**Format is now machine-validated.** Lighthouse's Agentic Browsing category (see section 13) parses the file as markdown and fails it unless:
- It contains exactly the spec structure: an H1 with the site/project name (the only hard-required element), then optionally a blockquote summary, prose, and H2 sections.
- Link entries use real markdown link syntax: `- [Name](https://url): note`. Plain `Title: URL` lines do NOT count and fail the audit with "File does not appear to contain any links".
- The file is longer than ~50 characters.
- Gotcha: avoid `---` horizontal-rule lines. Markdown parsers can read the first one as a YAML front-matter delimiter and discard everything above it, including the H1, so the audit reports "missing an H1 header" even though the file visibly has one.

```markdown
# Site Name

> One-sentence description of what this site is about.

## About
Brief 2-3 sentence description of the organization, what it does, who it serves.

## Key Pages
- [Homepage](https://yoursite.com/): Brief description
- [Services](https://yoursite.com/services/): What services are offered
- [Blog](https://yoursite.com/blog/): Topics covered

## Popular Content
- [Article Title](https://yoursite.com/blog/article): One-line summary
- [Guide Title](https://yoursite.com/guides/guide): One-line summary

## Contact
- Email: info@yoursite.com
- Location: Zagreb, Croatia
```

Optionally create `llms-full.txt`: the same overview followed by the full markdown of your key pages/articles, so a tool gets deep context in one fetch. This is cheap when the site already exposes per-page markdown (e.g. a `/article/markdown` endpoint or a content/export service): a single route can concatenate the `llms.txt` intro with each published page's markdown. Serve it `X-Robots-Tag: noindex, follow` (it duplicates page bodies, so avoid search-index duplicate content) and exclude any gated/premium content so you don't leak it. When you bound the dump (top-N, page cap), say what was dropped rather than silently truncating.

**Priority:** LOW / optional. Future-proofing, not a required output.

**Current status (as of 2026-07, per third-party trackers, not primary sources; verify before quoting):** Adoption is growing but uneven (roughly 10% of large domain samples). Some providers have confirmed support (Anthropic by late 2024, Perplexity by 2025), and by 2026 most major Western AI platforms offer partial support. The most concrete, verifiable adoption is in AI dev tools: Claude Code, Cursor, Windsurf, GitHub Copilot, Cline, Aider all fetch `/llms.txt` and `/llms-full.txt`, especially for documentation/reference sites. Google has said it does not use llms.txt for Search, and as of Q1 2026 no major AI provider (OpenAI, Google, Anthropic, Meta, Mistral) has publicly committed to reading it in production answer systems. What HAS changed: Google now validates the file in Lighthouse and PageSpeed Insights (Agentic Browsing category, section 13), which makes it a visible pass/fail checkbox for clients, tools, and audits. Bottom line: still don't prioritize it over robots.txt, schema, sitemap, or content structure, but if you ship one, it MUST follow the format rules above or it hurts you in audits.

---

## 6. Page Speed Optimization

**Priority: HIGH. Pages with FCP < 0.4s get 3× more AI citations**

Target metrics:
- First Contentful Paint (FCP): < 0.4 seconds
- Time to First Byte (TTFB): < 200ms
- Largest Contentful Paint (LCP): < 2.5 seconds

AI crawlers abandon slow pages more aggressively than traditional search crawlers.

Quick wins for PHP sites:
- Enable OPcache
- Use a reverse proxy cache (Varnish, Nginx FastCGI cache)
- Optimize images (WebP, lazy loading for below-fold)
- Minimize CSS/JS blocking resources
- Use a CDN for static assets
- Database query optimization (avoid N+1 queries)

---

## 7. Search Console Submissions

**Priority: HIGH**

Submit your sitemap to both:
- **Google Search Console**: https://search.google.com/search-console
- **Bing Webmaster Tools**: https://www.bing.com/webmasters

Bing is critical because ChatGPT and Copilot both use Bing's index. 87% of ChatGPT citations match Bing's top results.

---

## 8. IndexNow Implementation

**Priority: MEDIUM-HIGH**

IndexNow instantly notifies Bing (and by extension ChatGPT/Copilot) when content is published or updated.

```php
// After publishing or updating content
function notify_indexnow(string $url, string $api_key): void {
    $endpoint = "https://api.indexnow.org/indexnow";
    $params = http_build_query([
        'url' => $url,
        'key' => $api_key,
        'host' => parse_url($url, PHP_URL_HOST),
    ]);
    
    $ch = curl_init("{$endpoint}?{$params}");
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_TIMEOUT, 5);
    curl_exec($ch);
    curl_close($ch);
}
```

Place your API key file at `https://yoursite.com/{api-key}.txt`.

---

## 9. Markdown Alternate Links

**Priority: LOW (experimental, but zero-cost if you have Markdown source)**

If you write content in Markdown (e.g., from Obsidian, a CMS, or a static site generator), serve the raw Markdown version alongside HTML and signal it:

```html
<link rel="alternate" type="text/markdown" href="/blog/post-slug.md" />
```

AI models process Markdown natively. This gives them a clean, uncluttered version of your content.

---

## 10. Meta Tags for AI Extraction

**Priority: MEDIUM**

Ensure every important page has:
```html
<meta name="description" content="Concise, factual 150-160 char description" />
<meta name="author" content="Author Name" />
<meta property="og:title" content="Page Title" />
<meta property="og:description" content="Same as meta description" />
<meta property="og:type" content="article" />
<meta property="og:url" content="https://yoursite.com/page" />
<meta property="og:image" content="https://yoursite.com/images/og-image.jpg" />
<meta property="article:published_time" content="2026-03-01T10:00:00Z" />
<meta property="article:modified_time" content="2026-03-20T10:00:00Z" />
```

---

## 11. HTTPS and Security

**Priority: CRITICAL (baseline requirement)**

All pages must be served over HTTPS. This is a ranking factor for Google/Bing and a trust signal for AI systems. No exceptions.

---

## 12. Internal Linking Architecture

**Priority: HIGH. Pillar-cluster architecture delivers 2.7× citation multiplier**

Structure sites as topic clusters:
- **Pillar page**: Comprehensive overview of a topic (2000+ words)
- **Cluster pages**: Detailed sub-topic articles linking back to pillar
- **Bidirectional links**: Pillar links to clusters, clusters link to pillar
- Use descriptive anchor text (not "click here")

82.5% of ChatGPT citations link to pages within established topic hierarchies, not isolated articles.

---

## 13. Lighthouse Agentic Browsing Audit

**Priority: MEDIUM (new in 2026, the first mainstream tool that grades agent-readiness; category is explicitly "under development and subject to change")**

Chrome Lighthouse (Chrome 150+, also surfaced on pagespeed.web.dev) added an "Agentic Browsing" category that scores how well a site works for AI agents. Unlike other categories there is no 0-100 weighted score: each audit reports pass/fail (or a pass ratio) independently.

**What it checks:**

1. **llms.txt presence and format**: see section 5 for the exact validation rules (H1 required, markdown `[text](url)` links required, minimum length, no `---` front-matter trap).
2. **WebMCP integration**: WebMCP (Web Model Context Protocol) is a W3C proposal that lets a page expose callable tools to AI agents via `navigator.modelContext` instead of the agent guessing at the UI. Announced February 2026, in Chrome origin trial (Chrome 149-156); the main consuming agent today is Gemini in Chrome. Lighthouse verifies both declarative tools (defined in HTML) and imperative tools (registered from JS). Only relevant for interactive sites (search, cart, booking, filters); content/publishing sites can safely skip it. Reference: https://developer.chrome.com/docs/ai/webmcp
3. **Agent-centric accessibility subset**: agents drive pages through the accessibility tree, so Lighthouse filters the a11y audits that matter for machine interaction: every interactive element has a programmatic name, ARIA roles are valid and on compatible elements, identical links have the same purpose, touch targets have sufficient size/spacing. Accessibility fixes now double as agent-readiness fixes; pitch them that way.
4. **Layout stability (CLS)**: agents target elements by position, so layout shift breaks them the same way it annoys humans.

**Audit check:** run the site through https://pagespeed.web.dev/ and read the Agentic Browsing section at the bottom of the report, or use Lighthouse in Chrome DevTools (Chrome 150+).

**Watch for false positives:** entrance animations that fade content in (opacity 0 to 1 on scroll-reveal) can make the contrast and visibility audits fail elements whose static styles pass. Verify the computed static colors before "fixing" contrast findings caused by animation timing.

---
name: a-seo-gsc
description: >-
  Turn a Google Search Console export into a prioritized, evidence-backed SEO growth action plan,
  for any website. Use whenever someone wants to analyze search performance or Search Console / GSC
  data, understand why a site gets impressions but no clicks, diagnose indexing or coverage problems
  ("crawled currently not indexed", "discovered not indexed", index bloat, pages dropping out),
  find striking-distance keywords, spot keyword cannibalization or content gaps, improve CTR or
  organic traffic, or produce an SEO action plan. Triggers include: GSC export, Search Console CSV,
  "analyze my search data", "why isn't my site ranking / getting clicks", coverage report, indexing
  report, query analysis, page analysis, SEO audit from data (not code), competitor SEO analysis, and
  any upload of GSC Performance / Coverage / Indexing CSV exports. Works on any site; not tied to one
  project. For AI-search/GEO visibility use a-geo-optimizer; for code-level technical SEO fixes use a
  project's own technical-SEO skill. This skill is the data-driven layer that tells those two what to fix.
---

# a-seo-gsc: Search Console → action plan

Google Search Console already knows what is wrong with a site's search performance. The data is just buried in CSV exports nobody reads. This skill ingests those exports, runs a deterministic parser over them, applies a full analysis playbook (adapting to the site type), and writes a prioritized action plan you can actually execute. It works on any website, not one specific project.

The output is a doc: `docs/seo/YYYY-MM-DD-action-plan.md` in the target project (or wherever the user keeps analyses), plus a spoken summary of the headline finding and the top moves.

## First run: what to give me, and where to get it

If the user has not already handed over exports, tell them exactly this (full click-paths in `references/export-guide.md`):

> Go to Google Search Console (search.google.com/search-console), pick the property, and export three things. Each "Export" button gives a zip of CSVs.
> 1. **Performance → Search results**, set the date range (I want **Last 3 months** at minimum; if you can, also do **Compare** to the previous period so I can see trends). Click **Export**.
> 2. **Indexing → Pages**, click **Export** on the summary (gives the indexed vs not-indexed counts and the reason breakdown).
> 3. **Indexing → Pages → click the biggest "not indexed" reason** (usually "Crawled - currently not indexed") **→ Export** the URL table. Repeat for any other large reason. This is the one people skip, and it is the most useful file.
> Drop all the zips in a folder and point me at it. Optional extras that sharpen the analysis: the **Links** report (top linking sites), **Sitemaps**, and **Core Web Vitals**.

Then ask for the site's **brand terms** (name, domain, product names) so branded vs non-branded traffic can be split, and what the site is **for** (blog, ecommerce, SaaS, docs, local business) so the playbook adapts.

## Modes

| Mode | Trigger | What happens |
|------|---------|--------------|
| **Intake** | No data yet | Give the export instructions above, collect brand terms + site type |
| **Analyze** | Exports in hand | Run the parser, then work the playbook, report findings |
| **Plan** | After analyze | Write the tiered action-plan doc to `docs/seo/` |
| **Competitors** | On request, or when content gaps dominate | Dependency-free competitor lens (see below) |

Analyze and Plan usually run together. Do not skip Plan: the written, prioritized doc is the deliverable.

## The method

### 1. Parse (deterministic, cheap)
Run the parser over whatever the user gave you. It auto-detects Performance / Coverage / Drilldown files, normalizes the numbers, and prints findings.

```bash
python3 scripts/parse_gsc.py <zip-or-dir-or-csv> [more...] \
    --brand "brand one,brandtwo,domain.com" --min-impressions 5
```
Add `--json` to get machine output, `--out FILE` to save. Read the printed report before reasoning further; it already computes baseline, concentration, striking-distance, CTR underperformers, zero-click queries, parameter-duplicate cannibalization, coverage buckets, the index-starvation-vs-bloat diagnosis, and a categorized not-indexed list.

### 2. Analyze (judgment on top of the numbers)
Open `references/analysis-playbook.md` and walk every dimension that the data supports. The playbook covers far more than any one site will show: striking distance, CTR-vs-position, cannibalization, content gaps, indexing pathologies, decay (needs two date ranges), device/geo mismatch, rich-result opportunities, authority/linking signals, and site-type-specific failure modes (blog vs ecommerce vs SaaS vs docs vs local vs directory). Only report what the data actually supports, and say what it cannot show.

The non-negotiable discipline (from hard-won review experience): **separate observation from hypothesis from action.** "Impressions dropped" is an observation. "Googlebot got empty HTML" is a hypothesis until verified. Do not let a plausible story become a confident diagnosis. When you cannot prove causation (and a single export almost never can), say so and propose the check that would confirm it.

### 3. Plan (the deliverable)
Use `references/action-plan.md` as the template. Write findings evidence-first, then a tiered action list ranked by impact-over-effort, then a tracking table (one row per action: page/query, baseline, change date, expected effect, 28-day result) so next month's run can tell whether anything worked. Save to `docs/seo/YYYY-MM-DD-action-plan.md`. Create `docs/seo/` if absent. If the project has no `docs/`, ask where to save or offer the repo root.

### 4. Competitors (optional, dependency-free)
See `references/competitor-analysis.md`. Identify who ranks for the site's striking-distance and target queries (via `WebSearch`), fetch their ranking pages (`WebFetch`), and compare depth, structure, schema, freshness, and angle coverage. Produces a content-gap matrix and prioritized competitive moves. No paid API required; if the user has an Ahrefs/Semrush export, it plugs in as an extra CSV for real volume and backlink numbers.

## Principles

1. **Impressions are not the same as the right audience.** A page can soak up impressions from a query cluster that has nothing to do with the site's purpose. Growing that is vanity. Always sanity-check whether the winning queries match who the site is for.
2. **Index-starvation vs index-bloat are opposite problems with opposite fixes.** Starvation (good pages not indexed): fix authority, internal linking, and request indexing; do NOT publish more. Bloat (thin/duplicate pages diluting crawl): consolidate, noindex, canonicalize. The parser flags which one the data shows. Never prescribe "write more content" for a starved site.
3. **Authority is usually the real gate on a small site.** If only backlinked pages get indexed and rank while internally-linked-only pages sit in "crawled, currently not indexed," the bottleneck is authority and link architecture, not content volume. One earned link often does more than ten new posts.
4. **CTR far below the position curve = a snippet or intent problem, not a ranking problem.** A page at position 9 with 0% CTR usually means the title/meta does not match the query intent, or a SERP feature is eating the click. Fix the snippet, not the ranking.
5. **No controls means self-attributed wins.** Without an untouched comparison set and a dated change log, you cannot prove a fix caused a change. Build the tracking table so the next run can.
6. **Adapt to the site type.** The failure modes of a 300-page blog, a 50k-SKU store, a docs site, and a local business are different. Read the site-type section of the playbook.

## Cross-skill boundaries

- **a-geo-optimizer**, AI-search / GEO visibility (ChatGPT, Perplexity, Gemini citations, llms.txt, answer-capsule content). Use it for the AI-discoverability angle; this skill does not duplicate it.
- **A project-specific technical-SEO skill**, where you have one, audits the *codebase* (meta tags, canonicals, structured data, CWV). This skill diagnoses from *data* and tells that one what to go fix. Run this to find the problem, the code-level skill to fix it.

Your Search Console export is your own data. It stays on your machine: nothing in this skill uploads, phones home, or sends it anywhere.

## Reference files

- `references/export-guide.md`, exact GSC export click-paths for every report, how to get two date ranges for trend diffing, and alternative sources (Bing Webmaster Tools, Ahrefs/Semrush CSV).
- `references/analysis-playbook.md`, the full dimension-by-dimension methodology with thresholds, formulas, and per-site-type adaptations.
- `references/action-plan.md`, output doc template and the impact/effort prioritization framework.
- `references/competitor-analysis.md`, the dependency-free competitor lens.
- `scripts/parse_gsc.py`, the deterministic export parser.

# Action-plan template

The deliverable. Write it to `docs/seo/YYYY-MM-DD-action-plan.md` in the target project. Evidence first, then ranked actions, then a tracking table so the next run can measure what worked.

Keep it honest: every claim ties to a number from the export, every hypothesis is labeled as one, and anything the data cannot show is stated plainly.

## Structure

```markdown
# SEO action plan, <site>, <YYYY-MM-DD>

Source: GSC export, <date range>. Brand terms: <...>. Site type: <...>.

## Snapshot
- Clicks / impressions / CTR / avg position (with the one-line "so what").
- Branded vs non-branded.
- Concentration (top page's share of impressions).

## Headline finding
The single most important thing, stated in one paragraph, with the numbers that prove it and the one hypothesis (labeled) for the cause. This is what the whole plan hinges on.

## Findings by dimension
For each dimension the data supports (indexing, striking distance, CTR, cannibalization, content gaps, decay, device/geo, authority):
- **Observation:** what the numbers say (cite them).
- **Hypothesis:** the likely cause, labeled as a hypothesis.
- **Check:** what would confirm it (so it does not stay a guess).

## Action list (ranked)
Tiered by impact-over-effort. Each action: what, why (which finding it addresses), effort (S/M/L), expected effect, and who does it (code change vs GSC click vs content edit vs outreach).

### Tier 1. Highest impact
### Tier 2. Quick wins / hygiene
### Tier 3. Worth doing, secondary

## What this export cannot tell us
Explicit limits: no volume/difficulty data, no conversion data, causation unproven, query×page pairing unavailable, single snapshot (no trend), etc. Plus what to export next time to close the gap.

## Tracking table
| # | Action | Target (page/query) | Baseline (date) | Changed on | Expected effect | 28-day result |
|---|--------|--------------------|-----------------|-----------|-----------------|---------------|
Leave "Changed on" and "28-day result" blank; they get filled as work ships and at the next run.
```

## Prioritization

Rank by **impact ÷ effort**, not by dimension order. A rough ICE (Impact, Confidence, Ease) or a 2×2 (impact vs effort) is enough; do not over-formalize. Bias toward:

1. **Actions that unblock everything else.** On a starved site, getting content indexed (authority + linking) gates every other win, so it is Tier 1 even though it is slow. On a bloated site, the parameter/canonical cleanup is the unblock.
2. **Zero-click strong positions.** Rewriting a title for a page already at position 5-10 with near-zero CTR is the cheapest click recovery on the board.
3. **Striking distance.** Small pushes on page-2 queries that already have impressions.
4. **Hygiene that stops the bleed.** 404 fixes, param exclusions, junk-URL noindex. Low effort, prevents further dilution.

De-prioritize: chasing off-audience query clusters (vanity impressions), building new content on a site that cannot get its existing content indexed, and anything whose expected effect you cannot articulate.

## Who executes what (label every action)

- **Code change**, needs a dev edit (canonical logic, robots rules, schema, redirects). If the project has a technical-SEO or codebase-audit skill, point there.
- **GSC action**, manual Search Console step (Request Indexing, submit sitemap, validate fix). Give the exact click-path and the prioritized URL list to paste.
- **Content edit**, rewrite title/meta, add a section, consolidate posts, refresh. Route through the project's content workflow (never raw DB edits on content fields).
- **Off-site**, earn a link, get a mention, cross-post. Usually the real lever for authority-gated sites; connect to whatever promotion workflow exists.

## Tone

Write like a senior analyst briefing the owner, not a report generator. Lead with the decision, support with the number, name the uncertainty. No filler, no "it is worth noting," no em dashes. If the honest read is "your content is fine, the problem is nobody links to you," say exactly that.

# Analysis playbook

Work every dimension the data supports. Most sites show only a subset; report those, name what is missing, and never invent a finding the data does not carry. The parser (`scripts/parse_gsc.py`) computes the arithmetic for most dimensions below; you supply judgment and the ones it cannot.

Order below is roughly "most decisive first." On any given site, one or two dimensions dominate. Find the dominant one and lead with it.

---

## 0. Baseline and shape

Read the totals first (from the Dates/Chart file, which is authoritative; query/page files are a top-N sample).

- **Clicks, impressions, CTR, average position.** Position is impression-weighted, so a few high-impression page-2 queries drag it up.
- **Branded vs non-branded** (needs `--brand`). Branded clicks are people who already know the site. If branded is most of the traffic, the site is not winning new audiences. If the site cannot even rank #1 for its own brand, that is a distinct, fixable problem (entity signals, homepage authority).
- **Concentration.** What share of impressions sits on the single top page? High concentration (one page ≫ everything) means the site's search presence is one lucky page, not a system. Ask whether that page is even on-topic for the site.

Sanity flag: if impressions are large but clicks are near zero, the whole site is probably ranking on page 2, or the snippets are wrong, or the impressions are for queries the content does not actually answer.

## 1. Indexing / coverage pathology

Often the real story, and people never look. Diagnose which pattern:

- **Index-starvation:** more pages excluded than indexed, good content sitting in "Crawled - currently not indexed" or "Discovered - currently not indexed." Cause is almost always **authority + internal linking + crawl priority**, not content quality. Fixes: earn links, strengthen internal links from indexed high-authority pages, get hub/index pages indexed, request indexing for priority URLs, submit a clean sitemap. Do NOT tell a starved site to publish more; it just grows the parked pile.
- **Index-bloat:** thousands of thin/duplicate/parameter URLs indexed or clogging crawl. Fixes: canonicalize, noindex thin templates, `Disallow` parameter and faceted URLs, consolidate.
- **Canonical confusion:** large "Duplicate without user-selected canonical" or "Alternate page with proper canonical tag." Check that canonicals are self-consistent and point where you intend.

Coverage reason cheat-sheet (GSC label → what it means → move):

| Reason | Meaning | Move |
|--------|---------|------|
| Crawled - currently not indexed | Google fetched it, chose not to index | Quality/authority/linking. Improve links + request indexing. If genuinely thin, improve or noindex. |
| Discovered - currently not indexed | Known but not even crawled yet | Crawl budget / low priority. Internal links + sitemap + authority. |
| Duplicate without user-selected canonical | Google picked a different canonical | Set explicit canonical; consolidate duplicates. |
| Duplicate, Google chose different canonical than user | Your canonical was overridden | The "duplicate" is stronger; merge or differentiate. |
| Alternate page with proper canonical tag | Correctly canonicalized dupe | Usually fine. Expected for paginated/param/AMP. |
| Page with redirect | 301/302 | Fine if intended (restructure fallout). Audit for redirect chains. |
| Not found (404) | Broken | Fix internal links or 301 to the right page. |
| Soft 404 | Thin/empty page returning 200 | Add content or return real 404. |
| Blocked by robots.txt | Disallowed | Verify intentional; accidental blocks are common after CMS/CDN changes. |
| Blocked due to unauthorized request (401) | Auth-gated | Fine if intended. |
| Server error (5xx) | Broke during crawl | Reliability issue; investigate logs. |

Then **categorize the not-indexed URL list** (the parser buckets by generic path pattern: content, taxonomy-tag, taxonomy-category, product, pagination, parameter-url, feed, redirect-endpoint). Interpret per site: which buckets are junk that should never be index candidates (feeds, redirect endpoints, param/sort URLs, thin tag pages) vs which are real content being wrongly excluded. Junk → exclude from crawl. Real content excluded → the starvation fight above.

## 2. Striking distance

Queries at **positions 11-20** with real impressions are the highest-ROI lever on most sites: they already rank, just below the fold of page 1. Small on-page improvements (title/meta relevance, adding the exact sub-topic, one internal link, a bit more depth) can push them onto page 1 where clicks live. Sort by impressions; the top of that list is the week's to-do. Positions 5-10 with low CTR belong to dimension 3, not here.

## 3. CTR vs position

For queries in the top 10, compare actual CTR to the expected-for-position curve (parser flags actual below 40% of expected). Under-performers are a **snippet or intent problem**, not a ranking problem:
- Title/meta does not match the query intent (page ranks for "X system requirements" but the title says "X overview").
- A SERP feature (featured snippet owned by someone else, People-Also-Ask, a pack, an ad block) is eating the clicks.
- The URL/brand looks untrustworthy in the SERP.

Fixes: rewrite the title to lead with the searcher's outcome and include the query's head term; tighten the meta description; add the structured data that could win the feature (defer schema specifics to `a-geo-optimizer`, or to your project's code-level SEO skill). Zero-click at a strong position is the loudest, cheapest win on the board.

## 4. Cannibalization and duplicate URLs (partial)

- **Parameter/duplicate URLs** (parser detects): the same clean URL indexed under `?ref=`, `?sort=`, `?utm=`, session IDs, tracking tags, pagination. Splits signals and wastes crawl. Fix: self-referential-clean canonical, `Disallow` the params, or 301 the tracked variants.
- **True query cannibalization** (multiple distinct pages competing for one query): the standard export cannot prove this because it lists queries and pages separately. To confirm, export a Performance view filtered to the suspect query and see which pages appear, or use the API. When you do find it: pick one canonical page, merge the others into it, 301 the losers. (Classic symptom: several near-identical titles, none ranking well, positions bouncing.)
- **Over-splitting a topic:** a cluster of thin posts on near-identical sub-topics that would rank better as one strong page. Consolidation beats proliferation on low-authority sites.

## 5. Content gaps (partial)

Queries the site gets impressions for but has no dedicated page answering. Signals: a broad page ranking at position 9-15 for many specific long-tail variants it does not directly address (e.g., one overview post catching dozens of "X requirements / X pricing / X vs Y" queries). Two responses:
1. If on-topic and worth it: build the dedicated page (or a strong section) that directly answers the cluster.
2. If off-topic for the site's purpose: ignore it. Chasing off-audience queries grows vanity impressions. Judge against who the site is for.

GSC only shows queries you already appear for. For demand you are missing entirely, you need a keyword tool or the competitor lens.

## 6. Decay and trend (needs two date ranges)

With a comparison export, diff period-over-period:
- **Declining queries/pages:** falling clicks or impressions, or rising position number. Often a competitor overtook, content went stale, or a redesign/restructure lost signals. Freshness updates and re-earning links help.
- **Position improved but CTR dropped:** you moved up but the SERP got more competitive (features, ads) or the snippet stopped matching. Snippet work.
- **Seasonality:** distinguish a real drop from an annual dip. Year-over-year comparison separates them.
- **Restructure fallout:** a category/URL migration typically shows a wave of new redirects, duplicate-canonical entries, and a not-indexed bump as Google re-evaluates. Time findings against known deploy dates before blaming quality.

Without two ranges, say so and recommend pulling the comparison export next time.

## 7. Device and geo

- **Device CTR gap:** if mobile CTR ≪ desktop at the same position (or vice versa), the mobile SERP layout or the page's mobile snippet/experience is the issue.
- **Geo/language mismatch:** impressions concentrated in countries the site does not serve or in a language it does not publish (common: a homelab/tech post pulling global traffic a personal/consulting site cannot use). Or the reverse: target market underrepresented, implying weak local relevance or hreflang gaps. Match the geo distribution against the site's actual market.

## 8. Rich results / search appearance

The Search-appearance breakdown (and Enhancements report) shows which rich-result types the site is eligible for and which are erroring. Missing eligibility for obvious types (Article, FAQ, Product, Breadcrumb, HowTo) is a structured-data opportunity. Hand the specifics to `a-geo-optimizer` (AI-citation schema) or to whatever skill owns code changes in the project; this skill just flags the gap from the data.

## 9. Authority and internal linking (needs Links export)

- **Referring domains:** few linking sites = low authority = the index-starvation gate. The clearest tell: only backlinked pages are indexed and ranking. One earned link often pulls in a whole cluster.
- **Internal-link distribution:** pages with zero internal links are orphans Google deprioritizes. Ensure every important page is linked from indexed, higher-authority pages (home, hubs, related content), not just from each other.
- **Anchor concentration:** if all internal anchors are generic ("read more"), you are wasting relevance signals.

---

## Site-type adaptations

The dominant failure mode shifts with what the site is.

- **Blog / content site:** cannibalization from overlapping posts, thin-tag index bloat, striking-distance long-tail, freshness decay. Growth = consolidate + strengthen the best, not endless new posts (on low authority).
- **Ecommerce:** faceted-navigation parameter explosion (the #1 cause of index bloat), thin product pages, category-vs-product cannibalization, out-of-stock/discontinued 404s, seasonality. Canonicals and parameter handling dominate.
- **SaaS / marketing site:** small page count, so every page matters; branded vs non-branded split is diagnostic; feature/comparison/alternative pages are the striking-distance goldmine; docs subdomain interplay.
- **Docs / knowledge base:** huge page count, internal-search-style queries, version duplication (v1/v2 pages competing), "answer" intent where featured snippets matter most.
- **Local business:** geo distribution is everything; Maps/local-pack impressions; branded + "near me" queries; NAP consistency and local schema.
- **Directory / aggregator / marketplace:** thin auto-generated pages Google routinely refuses to index (expect large "crawled not indexed"); the fight is proving unique value per page or accepting partial indexing and concentrating authority on the pages that matter.

## Thresholds used by the parser (tune per site)

- Striking distance: position 11-20, impressions ≥ `--min-impressions` (default 5).
- CTR underperformer: position ≤ 10 and actual CTR < 40% of the expected-for-position value.
- Zero-click: clicks == 0 and impressions ≥ max(min-impressions, 10).
- Expected CTR curve is a blended industry average (pos 1 ≈ 27%, pos 10 ≈ 2%, page 2 ≈ 1%). It is a reference for spotting outliers, not ground truth; real CTR varies wildly by query type and SERP layout.

# Competitor analysis (dependency-free)

GSC shows what a site ranks for. It cannot show who is beating it, or what demand it is missing entirely. This lens fills that gap using only `WebSearch` and `WebFetch` (no paid API, no new dependency). If the user has an Ahrefs/Semrush/Moz export, fold it in for real volume and backlink numbers, but the lens works without one.

Run it when: content gaps or striking-distance dominate the findings, the user asks about competitors, or a target query cluster is worth a serious push.

## Method

### 1. Pick the battleground queries
From the analysis, take the striking-distance and target-content-gap queries that actually matter for the site's audience (not the vanity clusters). Cap it: 5-15 queries is enough to see the pattern.

### 2. See who ranks
For each query, `WebSearch` it and record the top 5-10 organic results (skip ads and the site's own pages). Tally domains across queries. The domains that recur are the real competitors for this space, which is more accurate than whoever the user assumes.

Note SERP features too: who owns the featured snippet, what People-Also-Ask questions appear (these are content-gap gold), whether a pack/carousel/forum result (Reddit, Stack Overflow) dominates. If forums own the SERP, the winning move may be a genuinely better resource or participating there, not another blog post.

### 3. Read the winners
For the top 2-4 competitor pages per key query, `WebFetch` the page and compare against the site's competing (or missing) page on:
- **Depth and coverage**, sub-topics they answer that the site does not.
- **Structure**, headings, tables, direct answers up top, scannability.
- **Freshness**, visible last-updated date; stale winners are beatable.
- **Format match**, does the query want a listicle, a how-to, a definition, a comparison, a tool? Ranking pages reveal the intent.
- **Schema / rich results**, what they render in the SERP (rating stars, FAQ, breadcrumbs).
- **Authority proxy**, domain seniority, how referenced they are. Without a backlink tool this is a judgment call; say so.

### 4. Build the gap matrix
A simple table: rows = sub-topics/angles, columns = the site + top competitors, cells = covered / thin / missing. The site's "missing" rows that competitors all cover are the content plan. The rows everyone is thin on are the differentiation opportunity.

### 5. Output competitive moves
Fold into the action plan as its own tier:
- **Beat on depth:** specific sub-topics to add to an existing striking-distance page to overtake a shallow winner.
- **Beat on freshness:** pages where the winner is stale and a refresh + re-earn-links wins.
- **Beat on format:** where the site's format mismatches intent (a prose post where the SERP wants a comparison table or tool).
- **New pages:** only for gaps that are on-audience and winnable given the site's authority. Do not recommend targeting a query cluster owned by DR-90 sites on a new site; name that wall when it exists.

## Honest limits

- `WebSearch` results are personalized/regional and a moment in time, not a ranking tool's aggregate. Treat them as a strong sample, not a rank tracker.
- No backlink or traffic numbers without a paid export. State authority comparisons as estimates.
- "Winnable" depends on the site's own authority (from the Links export and the indexing diagnosis). A gap you cannot rank for yet is not an opportunity, it is a someday. Sequence competitive content behind the authority work if the site is index-starved.

## Optional: paid-tool CSV inputs

If the user provides them, these upgrade the lens from qualitative to quantitative:
- **Keyword export** (volume, difficulty) → prioritize gaps by real demand, not just GSC impressions.
- **Backlink export** (referring domains, competitor link profiles) → quantify the authority gap and surface link targets (who links to competitors but not the site).
- **Site audit export** → competitor technical posture.

Read them as extra CSVs alongside the GSC findings; the volume/difficulty columns slot straight into prioritization.

# GSC export guide

Exact steps to pull the data this skill needs. Google Search Console UI wording drifts; match on intent if a label moved.

## The three core exports

Open [search.google.com/search-console](https://search.google.com/search-console) and select the property.

### 1. Performance (Search results)
1. Left nav → **Performance** → **Search results**.
2. Top of page: click the **Date** filter. Choose **Last 3 months** (or longer; 12 months is better if the site is older). To get trend data in one shot, use the **Compare** tab → "Compare last 3 months to previous period".
3. Make sure the four metric toggles are all on: **Total clicks, Total impressions, Average CTR, Average position**.
4. Click **Export** (top right) → **Download CSV** (a zip).

This zip contains: `Queries.csv`, `Pages.csv`, `Countries.csv`, `Devices.csv`, `Dates.csv` (or `Chart.csv`), `Filters.csv`, `Search appearance.csv`. Note: the Queries and Pages files are a top-N sample (~1000 rows), not every query, so query-level totals are a sample, not the site total. The Dates/Chart file carries the true totals.

### 2. Indexing summary (Pages)
1. Left nav → **Indexing** → **Pages**.
2. Click **Export** on the summary view.

Gives the indexed vs not-indexed counts over time (`Chart.csv`) and the reason breakdown (`Critical issues.csv` / `Non-critical issues.csv`, columns `Reason,Source,Validation,Pages`).

### 3. Indexing drilldown (the one people skip)
1. Still in **Indexing → Pages**, scroll to "Why pages aren't indexed".
2. Click the largest reason (usually **Crawled - currently not indexed**, or **Discovered - currently not indexed**, or **Duplicate without user-selected canonical**).
3. On the detail page, click **Export**. This gives `Table.csv` (`URL,Last crawled`), the actual list of affected URLs.
4. Repeat for any other reason with a meaningful count.

The drilldown is what turns "44 pages aren't indexed" into "these specific 44 URLs, and here's the pattern." Without it the analysis is guesswork.

## Getting trend data (two date ranges)

A single export is a snapshot: it cannot show "position improved but CTR dropped" or "this query is decaying." To diff:
- **Easiest:** in Performance, use the **Compare** tab (previous period or year-over-year), then Export. The CSVs then carry both periods.
- **Manual:** export "Last 3 months" today, and separately export the prior 3 months (custom range). Hand both to the skill; it will diff them.

## Optional exports that sharpen the analysis

| Report | Where | Adds |
|--------|-------|------|
| **Links** | Left nav → Links → Export | Top linking sites, top linked pages, internal links. Reveals authority and orphan pages. |
| **Sitemaps** | Indexing → Sitemaps | Submitted vs discovered counts; sitemap errors. |
| **Core Web Vitals** | Experience → Core Web Vitals | Slow-URL groups (LCP/CLS/INP). CWV code fixes belong to a performance or technical-SEO skill, not here. |
| **Enhancements / Structured data** | Experience/Enhancements | Which rich-result types are eligible and erroring. |
| **Manual actions / Security** | Security & Manual Actions | Penalties or hacks. Always check; a manual action explains everything at once. |

## Alternative and supplementary data sources

- **Bing Webmaster Tools** exports in a similar shape; the parser reads generic Clicks/Impressions/CTR/Position CSVs, so Bing "Search Performance" exports work too.
- **Ahrefs / Semrush / Moz** CSV exports (keyword volume, difficulty, backlinks) are not GSC but plug into the competitor lens as extra inputs for real search-volume and referring-domain numbers. GSC has no volume or difficulty data, so if the user has these, ask for them.
- **GA4** organic-landing-page export pairs conversions with the GSC clicks, answering "traffic but no conversions." GSC alone cannot see conversions.

## What GSC data cannot tell you (state these up front)

- **Search volume or keyword difficulty**, GSC shows your impressions, not total market demand. Use a keyword tool for that.
- **Why visitors don't convert**, clicks in, but not what happened after. Needs analytics.
- **Query × page pairing at scale**, the standard export gives queries and pages separately, not which query drove which page (unless you export a page-filtered Performance view). True one-query-many-pages cannibalization needs that filtered export or the API.
- **Causation**, correlation only. A change and a metric move in the same week is a hypothesis, not proof.

#!/usr/bin/env python3
"""
parse_gsc.py: normalize Google Search Console exports into a findings report.

Deterministic pre-processor for the a-seo-gsc skill. Point it at one or more
GSC export zips (or a directory / loose CSVs) and it auto-detects each report
type, normalizes the numbers, and emits a structured set of findings: striking-
distance queries, zero-click high-impression queries, CTR outliers vs an
expected curve, parameter-duplicate cannibalization, coverage buckets, and a
categorized list of not-indexed URLs.

The model reads this output and layers strategy on top. Keeping the arithmetic
here makes the analysis fast, cheap, and repeatable across sites.

Pure standard library. No third-party deps. Cross-platform.

Usage:
    python3 parse_gsc.py <path...> [options]

    <path>              One or more GSC export .zip files, a directory
                        containing them (or extracted CSVs), or loose .csv files.

Options:
    --brand "a,b,c"     Comma-separated brand terms to split branded vs
                        non-branded queries (case-insensitive substring match).
    --min-impressions N Minimum impressions for a query to surface in findings
                        (default: 5).
    --json              Emit JSON instead of the markdown report.
    --out FILE          Write output to FILE instead of stdout.

Exit codes: 0 on success, 2 on "no recognizable GSC data found".
"""

from __future__ import annotations

import argparse
import csv
import io
import json
import os
import sys
import tempfile
import zipfile

# Rough organic CTR-by-position reference (desktop+mobile blended, 2023-2024
# industry averages). Used only to flag snippet underperformance, not as truth.
EXPECTED_CTR = {
    1: 0.275, 2: 0.150, 3: 0.100, 4: 0.070, 5: 0.055,
    6: 0.043, 7: 0.034, 8: 0.028, 9: 0.024, 10: 0.021,
}
PAGE2_CTR = 0.010  # positions 11-20, generic


def _num(s: str) -> float:
    """Parse a GSC numeric cell: '1.92%', '11.85', '1,234', '40%', '' -> float."""
    if s is None:
        return 0.0
    s = s.strip().replace("%", "")
    if not s:
        return 0.0
    # Thousands separators vs decimal comma: GSC uses '.' decimals, but be lenient.
    if s.count(",") == 1 and "." not in s:
        s = s.replace(",", ".")
    else:
        s = s.replace(",", "")
    try:
        return float(s)
    except ValueError:
        return 0.0


def _read_csv(text: str) -> tuple[list[str], list[list[str]]]:
    """Return (header, rows). Handles quoted multi-line cells (GSC query cells)."""
    reader = csv.reader(io.StringIO(text))
    rows = [r for r in reader if r]
    if not rows:
        return [], []
    return rows[0], rows[1:]


def _find_col(header: list[str], *names: str) -> int:
    low = [h.strip().lower() for h in header]
    for name in names:
        n = name.lower()
        for i, h in enumerate(low):
            if h == n or h.startswith(n) or n in h:
                return i
    return -1


def _collect_csvs(paths: list[str], tmpdir: str) -> dict[str, str]:
    """Return {logical_name: csv_text} from zips/dirs/loose csvs."""
    out: dict[str, str] = {}

    def add(name: str, text: str) -> None:
        base = os.path.basename(name)
        key = base
        n = 1
        while key in out:  # keep duplicates from multiple exports distinct
            n += 1
            key = f"{base}#{n}"
        out[key] = text

    def handle_file(path: str) -> None:
        if not os.path.exists(path):
            print(f"warning: path not found, skipping: {path}", file=sys.stderr)
            return
        low = path.lower()
        if low.endswith(".zip"):
            try:
                with zipfile.ZipFile(path) as z:
                    for info in z.namelist():
                        if info.lower().endswith(".csv"):
                            with z.open(info) as f:
                                add(info, f.read().decode("utf-8-sig", "replace"))
            except zipfile.BadZipFile:
                print(f"warning: not a valid zip, skipping: {path}", file=sys.stderr)
        elif low.endswith(".csv"):
            with open(path, encoding="utf-8-sig", errors="replace") as f:
                add(path, f.read())

    for p in paths:
        if os.path.isdir(p):
            for root, _dirs, files in os.walk(p):
                for fn in files:
                    handle_file(os.path.join(root, fn))
        else:
            handle_file(p)
    return out


def _classify_url(url: str) -> str:
    """Generic path-pattern bucket for a not-indexed URL (site-agnostic)."""
    u = url.lower()
    path = u.split("://", 1)[-1]
    path = "/" + path.split("/", 1)[1] if "/" in path else "/"
    if "?" in u:
        return "parameter-url"
    if any(seg in path for seg in ("/feed", "/rss", "/atom")) or path.endswith(".xml"):
        return "feed"
    if any(seg in path for seg in ("/visit", "/go/", "/out/", "/redirect", "/goto")):
        return "redirect-endpoint"
    if "/page/" in path or "/p/" in path.rstrip("/").rsplit("/", 1)[0] + "/":
        return "pagination"
    if any(seg in path for seg in ("/tag/", "/tags/", "/label/", "/topic/", "/topics/")):
        return "taxonomy-tag"
    if any(seg in path for seg in ("/category/", "/categories/", "/cat/", "/c/")):
        return "taxonomy-category"
    if any(seg in path for seg in ("/product/", "/products/", "/shop/", "/item/")):
        return "product"
    return "content-page"


class Report:
    def __init__(self) -> None:
        self.queries: list[dict] = []
        self.pages: list[dict] = []
        self.countries: list[dict] = []
        self.devices: list[dict] = []
        self.dates: list[dict] = []          # perf time series
        self.coverage_issues: list[dict] = []
        self.coverage_series: list[dict] = []  # indexed vs not-indexed over time
        self.not_indexed: list[dict] = []     # drilldown table
        self.drill_issue: str = ""


def _ingest(csvs: dict[str, str]) -> Report:
    r = Report()
    for name, text in csvs.items():
        header, rows = _read_csv(text)
        if not header:
            continue
        h = [c.strip().lower() for c in header]

        # --- Coverage drilldown table: URL,Last crawled ---
        if "url" in h and any("crawl" in c for c in h):
            ci = _find_col(header, "url")
            cc = _find_col(header, "last crawled", "crawled")
            for row in rows:
                if len(row) > ci and row[ci].strip():
                    r.not_indexed.append({
                        "url": row[ci].strip(),
                        "last_crawled": row[cc].strip() if cc >= 0 and len(row) > cc else "",
                    })
            continue

        # --- Coverage issue tables: Reason,Source,Validation,Pages ---
        if "reason" in h and "pages" in h:
            cr = _find_col(header, "reason")
            cp = _find_col(header, "pages")
            for row in rows:
                if len(row) > cr and row[cr].strip():
                    r.coverage_issues.append({
                        "reason": row[cr].strip(),
                        "pages": int(_num(row[cp])) if cp >= 0 and len(row) > cp else 0,
                    })
            continue

        # --- Coverage chart: Date,Not indexed,Indexed,Impressions ---
        if "date" in h and any("not indexed" in c for c in h) and any(c == "indexed" or c.endswith("indexed") for c in h):
            cn = _find_col(header, "not indexed")
            cidx = -1
            for i, c in enumerate(h):
                if c == "indexed" or (c.endswith("indexed") and "not" not in c):
                    cidx = i
                    break
            for row in rows:
                if len(row) <= cn or row[cn].strip() == "":
                    continue
                r.coverage_series.append({
                    "date": row[0].strip(),
                    "not_indexed": int(_num(row[cn])),
                    "indexed": int(_num(row[cidx])) if cidx >= 0 and len(row) > cidx else 0,
                })
            continue

        # --- Performance breakdowns: have Clicks + Impressions + Position ---
        if "clicks" in h and "impressions" in h and "position" in h:
            cc = _find_col(header, "clicks")
            cimp = _find_col(header, "impressions")
            cctr = _find_col(header, "ctr")
            cpos = _find_col(header, "position")
            dim = header[0].strip().lower()
            bucket = None
            label = "name"
            if "quer" in dim:
                bucket, label = r.queries, "query"
            elif "page" in dim or "url" in dim:
                bucket, label = r.pages, "url"
            elif "countr" in dim:
                bucket, label = r.countries, "country"
            elif "device" in dim:
                bucket, label = r.devices, "device"
            elif "date" in dim:
                bucket, label = r.dates, "date"
            else:
                continue
            for row in rows:
                if not row or not row[0].strip():
                    continue
                rec = {
                    label: row[0].strip(),
                    "clicks": int(_num(row[cc])) if len(row) > cc else 0,
                    "impressions": int(_num(row[cimp])) if len(row) > cimp else 0,
                    "ctr": _num(row[cctr]) / 100.0 if cctr >= 0 and len(row) > cctr else 0.0,
                    "position": _num(row[cpos]) if cpos >= 0 and len(row) > cpos else 0.0,
                }
                bucket.append(rec)
            continue
    return r


def _clean_url(url: str) -> str:
    return url.split("?", 1)[0].split("#", 1)[0]


def analyze(r: Report, brand: list[str], min_imp: int) -> dict:
    f: dict = {}

    # Baseline totals: prefer the date series (authoritative), else devices.
    if r.dates:
        tot_clicks = sum(d["clicks"] for d in r.dates)
        tot_imp = sum(d["impressions"] for d in r.dates)
    elif r.devices:
        tot_clicks = sum(d["clicks"] for d in r.devices)
        tot_imp = sum(d["impressions"] for d in r.devices)
    else:
        tot_clicks = sum(q["clicks"] for q in r.queries)
        tot_imp = sum(q["impressions"] for q in r.queries)
    wpos = (sum(q["position"] * q["impressions"] for q in r.queries) /
            sum(q["impressions"] for q in r.queries)) if r.queries else 0.0
    f["baseline"] = {
        "clicks": tot_clicks,
        "impressions": tot_imp,
        "ctr": (tot_clicks / tot_imp) if tot_imp else 0.0,
        "impression_weighted_position": round(wpos, 2),
        "queries_tracked": len(r.queries),
        "pages_tracked": len(r.pages),
    }

    # Branded vs non-branded
    if brand:
        bl = [b.lower() for b in brand if b.strip()]
        def is_brand(q: str) -> bool:
            ql = q.lower()
            return any(b in ql for b in bl)
        bq = [q for q in r.queries if is_brand(q["query"])]
        nbq = [q for q in r.queries if not is_brand(q["query"])]
        f["branded_split"] = {
            "branded": {"clicks": sum(q["clicks"] for q in bq), "impressions": sum(q["impressions"] for q in bq)},
            "non_branded": {"clicks": sum(q["clicks"] for q in nbq), "impressions": sum(q["impressions"] for q in nbq)},
        }

    # Striking distance: page-2 queries (pos 11-20) with real impressions.
    striking = sorted(
        [q for q in r.queries if 10.5 < q["position"] <= 20.5 and q["impressions"] >= min_imp],
        key=lambda q: -q["impressions"],
    )
    f["striking_distance"] = striking[:30]

    # Page-1 low CTR: pos <=10, impressions >= min, actual CTR far below expected.
    ctr_outliers = []
    for q in r.queries:
        if q["position"] <= 10.5 and q["impressions"] >= min_imp:
            exp = EXPECTED_CTR.get(round(q["position"]), PAGE2_CTR)
            if q["ctr"] < exp * 0.4:  # under 40% of expected = snippet/intent problem
                ctr_outliers.append({**q, "expected_ctr": round(exp, 3)})
    f["ctr_underperformers"] = sorted(ctr_outliers, key=lambda q: -q["impressions"])[:30]

    # Zero-click high-impression
    f["zero_click"] = sorted(
        [q for q in r.queries if q["clicks"] == 0 and q["impressions"] >= max(min_imp, 10)],
        key=lambda q: -q["impressions"],
    )[:30]

    # Parameter-duplicate cannibalization (same clean URL indexed under variants)
    groups: dict[str, list[dict]] = {}
    for p in r.pages:
        groups.setdefault(_clean_url(p["url"]), []).append(p)
    dup = []
    for clean, variants in groups.items():
        if len(variants) > 1:
            dup.append({
                "clean_url": clean,
                "variants": sorted(variants, key=lambda p: -p["impressions"]),
                "total_impressions": sum(v["impressions"] for v in variants),
            })
    f["param_cannibalization"] = sorted(dup, key=lambda d: -d["total_impressions"])

    # Concentration: what share of impressions does the top page hold?
    if r.pages:
        top = max(r.pages, key=lambda p: p["impressions"])
        f["concentration"] = {
            "top_page": top["url"],
            "top_page_impressions": top["impressions"],
            "share_of_total": round(top["impressions"] / tot_imp, 3) if tot_imp else 0.0,
        }

    # Coverage
    if r.coverage_issues:
        buckets = {}
        for c in r.coverage_issues:
            buckets[c["reason"]] = buckets.get(c["reason"], 0) + c["pages"]
        f["coverage_issues"] = buckets
    if r.coverage_series:
        first = r.coverage_series[0]
        last = r.coverage_series[-1]
        f["coverage_trend"] = {
            "first": first, "last": last,
            "indexed_delta": last["indexed"] - first["indexed"],
            "not_indexed_delta": last["not_indexed"] - first["not_indexed"],
            "diagnosis": (
                "index-starvation (more pages excluded than indexed)"
                if last["not_indexed"] > last["indexed"]
                else "index-bloat risk" if last["not_indexed"] > last["indexed"] * 0.5
                else "healthy"
            ),
        }

    # Not-indexed URL categorization
    if r.not_indexed:
        cats: dict[str, list[str]] = {}
        for u in r.not_indexed:
            cats.setdefault(_classify_url(u["url"]), []).append(u["url"])
        f["not_indexed"] = {
            "issue": r.drill_issue or "not indexed",
            "total": len(r.not_indexed),
            "by_category": {k: {"count": len(v), "sample": v[:12]} for k, v in
                            sorted(cats.items(), key=lambda kv: -len(kv[1]))},
        }
    return f


def _pct(x: float) -> str:
    return f"{x * 100:.2f}%"


def to_markdown(f: dict) -> str:
    o: list[str] = ["# GSC parsed findings", ""]
    b = f["baseline"]
    o += [
        "## Baseline",
        f"- Clicks: **{b['clicks']}**  |  Impressions: **{b['impressions']}**  |  CTR: **{_pct(b['ctr'])}**",
        f"- Impression-weighted avg position: **{b['impression_weighted_position']}**",
        f"- Queries tracked: {b['queries_tracked']}  |  Pages tracked: {b['pages_tracked']}",
        "",
    ]
    if "branded_split" in f:
        bs = f["branded_split"]
        o += [
            "## Branded vs non-branded",
            f"- Branded: {bs['branded']['clicks']} clicks / {bs['branded']['impressions']} impr",
            f"- Non-branded: {bs['non_branded']['clicks']} clicks / {bs['non_branded']['impressions']} impr",
            "",
        ]
    if f.get("concentration"):
        c = f["concentration"]
        o += ["## Concentration",
              f"- Top page holds **{_pct(c['share_of_total'])}** of all impressions: `{c['top_page']}` ({c['top_page_impressions']} impr)",
              ""]

    def qtable(title: str, rows: list[dict], extra: str = "") -> None:
        nonlocal o
        o.append(f"## {title}")
        if not rows:
            o.extend(["_none_", ""])
            return
        o.append(f"| query | clicks | impr | CTR | pos |{' expected CTR |' if extra=='exp' else ''}")
        o.append(f"|---|--:|--:|--:|--:|{'--:|' if extra=='exp' else ''}")
        for q in rows:
            line = f"| {q['query']} | {q['clicks']} | {q['impressions']} | {_pct(q['ctr'])} | {q['position']:.1f} |"
            if extra == "exp":
                line += f" {_pct(q.get('expected_ctr', 0))} |"
            o.append(line)
        o.append("")

    qtable("Striking distance (pos 11-20, quick wins)", f.get("striking_distance", []))
    qtable("Page-1 CTR underperformers (snippet/intent problem)", f.get("ctr_underperformers", []), "exp")
    qtable("Zero-click high-impression queries", f.get("zero_click", []))

    o.append("## Parameter-duplicate cannibalization")
    if f.get("param_cannibalization"):
        for d in f["param_cannibalization"]:
            o.append(f"- `{d['clean_url']}`: {len(d['variants'])} indexed variants, {d['total_impressions']} impr total")
            for v in d["variants"]:
                o.append(f"    - {v['impressions']} impr, pos {v['position']:.1f}: `{v['url']}`")
    else:
        o.append("_none detected_")
    o.append("")

    if "coverage_issues" in f:
        o.append("## Coverage issues")
        for reason, n in sorted(f["coverage_issues"].items(), key=lambda kv: -kv[1]):
            o.append(f"- **{n}**: {reason}")
        o.append("")
    if "coverage_trend" in f:
        t = f["coverage_trend"]
        o += ["## Coverage trend",
              f"- {t['first']['date']}: {t['first']['indexed']} indexed / {t['first']['not_indexed']} not indexed",
              f"- {t['last']['date']}: {t['last']['indexed']} indexed / {t['last']['not_indexed']} not indexed",
              f"- Diagnosis: **{t['diagnosis']}**", ""]
    if "not_indexed" in f:
        ni = f["not_indexed"]
        o.append(f"## Not-indexed URLs ({ni['total']} total)")
        for cat, d in ni["by_category"].items():
            o.append(f"### {cat} ({d['count']})")
            for u in d["sample"]:
                o.append(f"- {u}")
            if d["count"] > len(d["sample"]):
                o.append(f"- … +{d['count'] - len(d['sample'])} more")
            o.append("")
    return "\n".join(o)


def main() -> int:
    ap = argparse.ArgumentParser(description="Normalize GSC exports into findings.")
    ap.add_argument("paths", nargs="+")
    ap.add_argument("--brand", default="")
    ap.add_argument("--min-impressions", type=int, default=5)
    ap.add_argument("--json", action="store_true")
    ap.add_argument("--out", default="")
    args = ap.parse_args()

    with tempfile.TemporaryDirectory() as tmp:
        csvs = _collect_csvs(args.paths, tmp)
    if not csvs:
        print("No CSV files found in the given paths.", file=sys.stderr)
        return 2

    report = _ingest(csvs)
    if not (report.queries or report.pages or report.coverage_issues or report.not_indexed):
        print("No recognizable GSC performance or coverage data found.", file=sys.stderr)
        return 2

    brand = [b for b in args.brand.split(",") if b.strip()]
    findings = analyze(report, brand, args.min_impressions)
    out = json.dumps(findings, indent=2, ensure_ascii=False) if args.json else to_markdown(findings)

    if args.out:
        with open(args.out, "w", encoding="utf-8") as fh:
            fh.write(out + "\n")
        print(f"Wrote {args.out}")
    else:
        print(out)
    return 0


if __name__ == "__main__":
    sys.exit(main())