search-visibility

{
  "name": "search-visibility",
  "description": "Two halves of getting found: a-seo-gsc pulls your Google Search Console data over the API (or reads a manual export) and turns it into a prioritized action plan backed by your own numbers, and a-geo-optimizer checks whether AI assistants can fetch, parse, and cite a site, then recommends the technical and content fixes. One diagnoses from data, the other covers the newer search surface.",
  "version": "1.3.0",
  "author": {
    "name": "Ivan Misic",
    "url": "https://ivanmisic.net"
  },
  "homepage": "https://ivanmisic.net/toolshed/plugins/search-visibility",
  "repository": "https://github.com/imisic/claude-marketplace",
  "license": "MIT",
  "keywords": [
    "seo",
    "geo",
    "search-console",
    "analytics",
    "content-strategy"
  ]
}

# search-visibility changelog

## 1.3.0

`a-seo-gsc` can now pull Search Console data itself instead of asking you to download three zips. `scripts/fetch_gsc.py` authenticates with a Google service account and writes the same CSV shapes the manual export produces, so the parser reads either one and the export route still works unchanged. Setup is in `references/export-guide.md`.

The API is not just less clicking. The UI export caps its query and page tables near a thousand rows where the API returns 25,000 per request, it writes a matching previous window so trend analysis runs without you remembering to tick Compare, and it returns page+query pairs.

That last one adds an analysis the skill could not do before. No manual export tells you which of your pages a given query hit, so "which of my own pages compete for this term" was unanswerable. The parser now detects it, and flags the case worth acting on first: where the page Google shows most often is not the one that ranks best.

Coverage and Links still need a manual export. Neither has an API, and the guide now says so rather than leaving you to find out. The URL Inspection API is not a substitute for the coverage report, because it only answers for URLs you hand it and so can never surface index bloat from URLs you never submitted.

The script is standard library only and signs its request with openssl, so there is nothing to pip install. Your key is read from a path you choose and is never written to disk or sent anywhere but Google.

## 1.2.2

The marketplace and storefront copy now describe AI-search work as an eligibility audit and fix plan, not a promise of citations.

## 1.2.1

The README now says which commands you get after installing, and the GEO acronym is spelled out where it first appears. Four hard-banned words swapped for plain ones in `a-geo-optimizer`'s reference docs.

## 1.2.0

The skills in this plugin now carry an `agents/openai.yaml` sidecar, so they show up in Codex's skill picker with a proper name and one-line description instead of a raw folder name. `a-seo-gsc` is explicit-only in Codex, `a-geo-optimizer` is not, matching how each behaves in Claude Code.

## 1.1.0

The two skills now differ in who can start them, because they are used at opposite moments.

`a-seo-gsc` is yours to invoke only. It is a session you sit down to run once a Search Console export is in hand, and nothing about a conversation tells Claude that has happened.

`a-geo-optimizer` still fires on its own. Its whole value is arriving while a page is being built, before anyone thinks to ask whether an assistant will be able to quote it.

The README table now names the mode per skill.

## Earlier

1.0.0 predates this file. See the git history at https://github.com/imisic/claude-marketplace.

# search-visibility

Two skills for the two places people now find things: a search engine, and an AI assistant answering on their behalf. They work from opposite ends. `a-seo-gsc` starts from your measured data and tells you what to fix. `a-geo-optimizer` starts from the site and checks the conditions that make citation possible.

## The two skills

| Skill | Starts from | Answers | Invoked by |
|---|---|---|---|
| `a-seo-gsc` | Your Google Search Console data, over the API or exported | What is actually costing me traffic, and in what order do I fix it | you only |
| `a-geo-optimizer` | The site or page itself | Can an AI assistant reach this, parse it, and quote it | you or Claude |

`a-seo-gsc` is a session you sit down to run once your data is in hand, so Claude will not start it for you. `a-geo-optimizer` is the opposite: it earns its keep by firing while you are building a page, before anyone thinks to ask for it.

### a-seo-gsc

Give it a property name and it produces a prioritized action plan instead of a wall of rows. It covers the diagnoses that matter and that people usually miss: pages with impressions but no clicks, striking-distance queries sitting just off page one, keyword cannibalization where your own pages compete, indexing failures (`crawled - currently not indexed`, `discovered - currently not indexed`), and index bloat.

`scripts/fetch_gsc.py` pulls the Performance data over the Search Console API, so most of the downloading is gone. It is also more than the export gives you: the UI caps its query and page tables near a thousand rows where the API returns 25,000 per request, it fetches a matching previous window so trend analysis always runs, and it returns **page+query pairs**. No manual export contains that shape, which is why "which of my own pages compete for this query" has never been answerable from one. The parser now answers it, and flags the case worth fixing first, where the page Google shows most often is not the one that ranks best.

Coverage and Links still need a manual export, because neither has an API. The guide says which two reports to grab and why the URL Inspection API is not a substitute for the coverage one.

Prefer to keep downloading CSVs? That still works exactly as before. The fetcher writes Google's own export shapes, so the parser cannot tell the two apart.

It adapts to the site type, because a 300-page blog, a 50k-SKU store, a docs site, and a local business fail in different ways. `scripts/parse_gsc.py` pre-processes the data deterministically, so the numbers come from arithmetic rather than from a model's reading of a CSV.

`references/export-guide.md` covers the one-time API setup, the exact click-paths for the reports you still export by hand, and which alternative sources work (Bing Webmaster Tools, an Ahrefs or Semrush CSV).

Your Search Console data is your own and stays on your machine. Nothing here uploads it and the analysis is entirely local. The fetcher makes one kind of outbound call: it signs in to Google with your own service-account key and asks Google for your own property's data. It talks to `oauth2.googleapis.com` and `www.googleapis.com` and nothing else, there is no telemetry, and your key never leaves your machine.

### a-geo-optimizer

Generative engine optimization, the AI-search counterpart to SEO. Being useful is not enough if the assistant cannot fetch the page, cannot tell what it says, or cannot find a quotable claim in it.

Covers crawler access, structured data, and content shaped so an answer can be lifted out of it. It recommends changes that improve the page's eligibility for citation, not a guaranteed result. Use it when building a page as well as when auditing one: the cheapest time to fix this is before publishing.

## Install

```
/plugin marketplace add imisic/claude-marketplace
/plugin install search-visibility@imisic
```

Then the commands are `/a-seo-gsc` and `/a-geo-optimizer`.

The longer write-up lives on the storefront: [ivanmisic.net/toolshed/plugins/search-visibility](https://ivanmisic.net/toolshed/plugins/search-visibility).

## Using them together

Run `a-seo-gsc` first when you have data, because it tells you where the loss actually is and stops you optimizing a page nobody was going to visit. Run `a-geo-optimizer` on a new site or page, where there is no data yet, and on anything you want quoted by an assistant.

Neither one changes code. They produce findings and plans; you decide what lands.

---
name: a-geo-optimizer
description: >-
  Optimize a web project for AI search visibility (GEO/AEO) so ChatGPT, Claude, Gemini, Perplexity
  and Copilot can cite it. Triggers: AI SEO, GEO, AEO, llms.txt, schema markup for AI, robots.txt
  for AI bots, structured data audit, WebMCP, agentic browsers, generative or answer engine
  optimization, "make my site AI-friendly", "optimize for ChatGPT", or any page whose AI
  discoverability matters even if unnamed. Also use when building a new site or page, to bake GEO
  in from the start. Owns machine-readability and citability, not wording and not measurement:
  a site's own Search Console traffic goes to a-seo-gsc.
---

# GEO Optimizer: AI Search Visibility Skill

Make any web project discoverable and citable by AI assistants (ChatGPT, Claude, Gemini, Perplexity, Copilot). This skill covers technical implementation, content structure, and ongoing monitoring.

## When to use this skill

- **New site/page creation**: Bake GEO in from the start
- **Existing site audit**: Diagnose and fix AI visibility gaps
- **Content writing/restructuring**: Format content for AI citation
- **Template development**: Build PHP/HTML templates with GEO built-in
- **robots.txt / schema / sitemap work**: Ensure AI crawler compatibility
- **Competitive analysis**: Compare AI citation performance

## How to use this skill

### Step 1: Determine the task type

Ask yourself which mode applies:

| Mode | When | What to do |
|------|------|------------|
| **Full audit** | User wants the full review | Run all checklists from `references/technical-checklist.md` |
| **New build** | Creating a site/page from scratch | Follow the build-time integration guide below |
| **Content optimization** | Writing or restructuring content | Follow `references/content-strategy.md` |
| **Quick fix** | Specific technical task (robots.txt, schema, etc.) | Jump to relevant section in `references/technical-checklist.md` |

### Step 2: For full audits, follow this sequence

1. **Technical foundation**: Read `references/technical-checklist.md` and verify:
   - robots.txt allows AI crawlers (exact user-agents in `references/crawler-reference.md`)
   - Server-side rendering (critical: most AI crawlers don't execute JS)
   - Schema.org JSON-LD markup on key pages
   - Page speed (FCP < 0.4s target)
   - XML sitemap + RSS/Atom feed
   - HTTPS, mobile-friendly
   - llms.txt file: if present it MUST pass Lighthouse validation (H1 + markdown `[text](url)` links, see checklist section 5); a malformed one is worse than none because it fails a visible audit
   - Agentic Browsing readiness (checklist section 13): run PageSpeed Insights and read the Agentic Browsing category; agent-critical a11y (programmatic names, valid ARIA, touch targets) and CLS now double as agent-readiness signals. WebMCP only matters for interactive sites (search, cart, booking)
   - If behind Cloudflare: fetch the LIVE robots.txt and check for injected Content-Signal directives (`ai-train=no` is the managed default; `ai-input=no` would hurt AI citation, see crawler-reference section 12)

2. **Content structure**: Read `references/content-strategy.md` and evaluate:
   - Answer-capsule format (40-60 word direct answer after each heading)
   - Front-loaded key content (44% of AI citations come from first 30% of page)
   - Section length (120-180 words per self-contained section)
   - Statistical density (one verifiable stat per 150-200 words)
   - Question-style headings matching conversational queries
   - FAQ sections with real user questions
   - Author bios with credentials

3. **Entity and authority signals**: Check:
   - Organization schema with consistent NAP (Name, Address, Phone)
   - Author schema with credentials and external profile links
   - Third-party presence (review sites, directories, Reddit, YouTube)
   - Internal linking structure (pillar-cluster architecture)

4. **Monitoring setup**: Recommend:
   - GA4 AI traffic segment (regex for AI referral sources)
   - Monthly prompt testing across AI platforms
   - GEO monitoring tool appropriate to budget

### Step 3: For new builds, integrate GEO at template level

When building PHP/HTML templates, include these by default:

**In the `<head>` of every page:**
```html
<!-- JSON-LD schema block (populated per page type) -->
<script type="application/ld+json" id="schema-jsonld">
<!-- Populated server-side per template -->
</script>

<!-- Markdown alternate for long-form content (optional) -->
<link rel="alternate" type="text/markdown" href="<?= $markdown_url ?>" />

<!-- Standard meta for AI extraction -->
<meta name="description" content="<?= htmlspecialchars($meta_description) ?>" />
<meta name="author" content="<?= htmlspecialchars($author_name) ?>" />
<meta property="article:published_time" content="<?= $published_date ?>" />
<meta property="article:modified_time" content="<?= $modified_date ?>" />
```

**Content templates should enforce:**
- Single H1 per page
- Logical H2/H3 hierarchy
- First paragraph = direct answer to the page's core question
- Visible "Last Updated" date reflecting genuine content changes
- Author bio block with photo, credentials, external links

**Server-side requirements:**
- All content rendered server-side (SSR), not client-side JS
- JSON-LD generated server-side per page template
- Clean semantic HTML (proper heading hierarchy, lists, tables)
- Fast TTFB and FCP (< 0.4s target)

### Step 4: Output deliverables

Depending on the task, produce one or more of:

- **robots.txt**: Ready to deploy, with all AI crawlers allowed (see `references/crawler-reference.md` for exact user-agents)
- **llms.txt / llms-full.txt** (optional, low-medium priority): Markdown site overview, plus an optional full-content companion. As of 2026-07 adoption as a retrieval source is still uneven: AI dev tools (Claude Code, Cursor, Copilot, Windsurf, Cline, Aider) fetch both and some providers (Anthropic, Perplexity) confirm support, but Google doesn't use it for Search and no major provider has committed to it in production answer systems. What changed in 2026: Lighthouse/PageSpeed Insights now validates the file (Agentic Browsing category), so if one exists it MUST follow the spec format (H1 required; links as markdown `- [name](url): note` bullets, never plain `Title: URL` lines; no `---` rules that get parsed as front-matter). Generate it when trivial alongside other work, notably when the site already exposes per-page markdown, since a route can then concatenate it cheaply. Don't prioritize it over robots.txt, schema, sitemap, or content.
- **Schema JSON-LD blocks**: Per page type (Article, Product, FAQPage, Organization, etc.)
- **Content restructured**: In answer-capsule format with front-loaded answers
- **Audit report**: Findings and priority-ordered recommendations
- **PHP template code**: With GEO elements built into the template layer
- **Sitemap + RSS/Atom feed**: If missing or incomplete
- **GA4 configuration**: AI referral traffic tracking setup

## Key principles to always follow

1. **SSR is non-negotiable**: If content requires JavaScript to render, AI crawlers (except Googlebot) cannot see it. Always flag this.

2. **Front-load answers**: The most important content goes in the first 30% of the page. AI systems extract from the top.

3. **Be specific, not promotional**: AI systems filter out sales copy. Use definitive, factual language with specific numbers, dates, and names. Entity density matters.

4. **Earned media > owned content**: Third-party mentions are 6.5× more likely to generate AI citations than self-published content. Always recommend off-site authority building.

5. **Freshness is a hard requirement**: Content updated within 30 days accounts for 76% of AI citations. Recommend refresh cycles.

6. **Cross-platform optimization**: Only 11-14% of citations overlap between platforms. Optimize for all major AI systems, not just one.

7. **Don't block crawlers by accident**: Always check the LIVE robots.txt for inherited CMS/CDN blocks on AI user-agents, injected Content-Signal directives (Cloudflare's managed robots.txt defaults to `ai-train=no` on 3.8M+ domains), and CDN-level bot blocking that fires before robots.txt is read (test with `curl -A "GPTBot"`).

8. **Agent-readiness is the new layer**: Lighthouse's Agentic Browsing category (2026) grades llms.txt format, WebMCP tools, agent-critical accessibility, and CLS. Agentic browsers (OpenAI Atlas, Perplexity Comet, Gemini in Chrome) drive real sessions through the rendered page and a11y tree, so accessibility fixes now double as agent task-completion fixes.

## Reference files

- `references/technical-checklist.md`: Complete technical implementation checklist (robots.txt, schema, sitemap, llms.txt, page speed, rendering)
- `references/content-strategy.md`: Content formatting, answer-capsule structure, citation optimization, topic clustering
- `references/crawler-reference.md`: All AI crawler user-agents, which platforms they serve, robots.txt templates, and platform-specific submission guides

interface:
  display_name: "GEO Optimizer"
  short_description: "Make a page reachable and quotable by assistants"

# Content Strategy for AI Citations

How to structure, write, and maintain content that AI systems extract, cite, and recommend.

## Table of Contents

1. [The Answer-Capsule Format](#1-the-answer-capsule-format)
2. [Front-Loading Critical Content](#2-front-loading-critical-content)
3. [Section Structure and Length](#3-section-structure-and-length)
4. [Heading Strategy](#4-heading-strategy)
5. [Statistical Density and Source Attribution](#5-statistical-density-and-source-attribution)
6. [Entity Density](#6-entity-density)
7. [Language and Tone](#7-language-and-tone)
8. [FAQ Sections](#8-faq-sections)
9. [Comparison Tables](#9-comparison-tables)
10. [Content Formats That Get Cited](#10-content-formats-that-get-cited)
11. [Author Authority Signals](#11-author-authority-signals)
12. [Content Freshness](#12-content-freshness)
13. [Topic Cluster Architecture](#13-topic-cluster-architecture)
14. [Off-Site Authority Building](#14-off-site-authority-building)
15. [Intent Modifier Phrases](#15-intent-modifier-phrases)
16. [Brand Sentiment Auditing](#16-brand-sentiment-auditing)
17. [What NOT to Do](#17-what-not-to-do)

---

## 1. The Answer-Capsule Format

The core content pattern for AI citation. Every major section should follow this structure:

```
[Question-style H2 heading]

[40-60 word direct answer, the "capsule"]

[Supporting detail, evidence, examples, 80-120 additional words]

[Optional: data table, comparison, or specific example]
```

**Example (good):**
```markdown
## How long does it take to set up a Synology NAS?

A basic Synology NAS setup takes 15-30 minutes for hardware assembly
and initial DSM installation. Full configuration including RAID setup,
shared folders, user accounts, and package installation typically
requires 1-2 hours. Advanced setups with Docker containers, reverse
proxies, and automated backups can take a full day.

The process breaks down into three phases. First, physical assembly:
insert drives, connect cables, power on. Second, DSM installation
via find.synology.com or Synology Assistant. Third, configuration
of storage pools, shared folders, and services.
```

**Example (bad, buries the answer):**
```markdown
## NAS Setup Guide

Network Attached Storage devices have become increasingly popular
in recent years. There are many brands to choose from, each with
their own strengths. In this guide, we'll walk through everything
you need to know about setting up your device. But first, let's
understand what a NAS actually is...

[answer appears 500 words later]
```

---

## 2. Front-Loading Critical Content

**44.2% of all LLM citations come from the first 30% of a page's text.**

Rules:
- The most important, quotable, citable content goes at the top
- First paragraph after H1 should answer the page's core question
- No lengthy introductions, no "in this article we will discuss..."
- Get to the point immediately, then expand

For product/service pages: Lead with what it is, who it's for, and what problem it solves. Pricing and key differentiators in the first 3 paragraphs.

For blog posts: Lead with the answer or key finding. Background and methodology come after.

---

## 3. Section Structure and Length

**Optimal section length: 120-180 words per self-contained section.**

Pages using this section length receive 70% more ChatGPT citations than pages with sections under 50 words.

Each section should be:
- Self-contained (extractable without needing context from other sections)
- Focused on one specific sub-question
- Bounded by a descriptive heading
- Complete enough to stand alone as an answer

Articles over 2,900 words average 5.1 citations vs 3.2 for content under 800 words. Longer is better, but only if each section adds genuine value.

---

## 4. Heading Strategy

Use question-style headings that mirror how people ask AI assistants.

**Good headings (match conversational queries):**
- "How does [X] compare to [Y]?"
- "What are the pricing options for [service]?"
- "How long does [process] take?"
- "What's the best [tool] for [use case]?"
- "Is [product] worth it in 2026?"

**Bad headings (traditional SEO style, hard for AI to map to queries):**
- "Features"
- "Overview"
- "Technical Specifications"
- "Our Approach"
- "More Information"

Also: ChatGPT internally appends intent modifiers like "detailed guide", "step-by-step tutorial", "in-depth comparison" to its search queries. Including these phrases naturally in your headings or opening paragraphs can improve retrieval.

---

## 5. Statistical Density and Source Attribution

**Content with verifiable claims receives 44% more citations. The original GEO research found citing sources increased visibility by 115.1%.**

Rules:
- Include one verifiable statistic per 150-200 words
- Always attribute statistics to their source
- Use specific numbers, not vague claims ("3.2× faster" not "much faster")
- Include dates with stats to signal freshness
- Prefer original data you've collected over recycled third-party stats

**Good:** "According to our 2026 client survey, 73% of small businesses in Croatia report spending less than 2 hours per week on social media marketing."

**Bad:** "Many businesses find social media marketing time-consuming."

---

## 6. Entity Density

**Content with high entity density shows 20.6% higher citation rates.**

Entities = proper nouns, specific names, dates, places, product names, company names, standards, technologies.

- Use full, specific names on first mention (not just pronouns)
- Include dates, version numbers, specific locations
- Reference specific standards, frameworks, or methodologies by name
- Mention specific tools, platforms, or competitors by name when relevant

**High entity density:** "Zagreb-based agencies using Figma and Webflow in 2026 report 40% faster delivery times compared to agencies still using WordPress page builders."

**Low entity density:** "Agencies using modern design tools report faster delivery times than those using older platforms."

---

## 7. Language and Tone

AI systems prefer **definitive, authoritative language** over hedging.

**Preferred:**
- "The best approach is..."
- "This costs €500-800 per month"
- "Setup takes 2-3 business days"
- "The main drawback is..."

**Avoided by AI systems:**
- "It depends on various factors..."
- "There are many considerations to keep in mind..."
- "Results may vary..."
- "Contact us to learn more" (dead-end for AI)

**Critical:** AI systems filter out overtly promotional or sales-heavy content. Write as a knowledgeable advisor, not a salesperson. Neutral, factual language with specific claims wins citations.

---

## 8. FAQ Sections

FAQ sections with question-style headings nearly double citation chances. AI systems extract Q&A pairs directly, especially when paired with FAQPage schema.

**Requirements:**
- Use actual questions your customers/readers ask
- Match the conversational phrasing people use with AI ("How do I..." not "Methodology for...")
- Each answer: 40-80 words, direct and complete
- 5-10 FAQs per page is the sweet spot
- Pair with FAQPage JSON-LD schema (see technical-checklist.md)

**Good FAQ structure:**
```markdown
## Frequently Asked Questions

### How much does a website redesign cost in Croatia?

A professional website redesign in Croatia typically costs €2,000-8,000
for small business sites, €8,000-25,000 for mid-size corporate sites,
and €25,000+ for complex e-commerce platforms. Pricing depends on page
count, custom functionality, and whether content migration is included.

### How long does a typical web project take?

Most small business websites take 4-8 weeks from kickoff to launch.
Mid-size projects run 8-16 weeks. Enterprise-level builds with custom
integrations can extend to 3-6 months.
```

---

## 9. Comparison Tables

**Comparison tables deliver 2.5× more citations than text-only equivalents.**

Use HTML tables or Markdown tables for:
- Product vs. competitor comparisons
- Feature comparisons across tiers/plans
- Pros vs. cons
- Technology comparisons
- Before/after scenarios

Keep tables clean and data-rich. AI systems extract structured relationships from tables better than from prose.

---

## 10. Content Formats That Get Cited

By citation frequency (ChatGPT data):

| Format | % of Citations | Notes |
|--------|---------------|-------|
| "Best X" listicles | 43.8% | Highest citation rate |
| How-to guides | ~25% | Especially with step-by-step structure |
| Comparison articles | ~15% | X vs Y format |
| Definition/explainer | ~10% | "What is X" format |
| Case studies | ~5% | With specific metrics and outcomes |

Prioritize creating "Best X for Y" content for your niche. These are the most-cited format by a wide margin.

---

## 11. Author Authority Signals

Cited pages average 4.1 expert quotations vs 2.4 on non-cited pages.

For every content page, include:
- Author name and photo
- Professional title/role
- Years of experience or credentials
- Links to LinkedIn, professional profiles, or personal site
- Brief description of relevant expertise

For the Organization/brand:
- Clear "About Us" page with founding date, team, and mission
- Google Business Profile (fully optimized)
- Consistent Name, Address, Phone (NAP) across all platforms
- Wikidata entry (for entity recognition by AI)

---

## 12. Content Freshness

**Content updated within 30 days accounts for 76.4% of all AI citations.**

Rules:
- Establish a 60-90 day refresh cycle for key pages
- Update with substantive changes (not just timestamp bumps)
- Pages with genuine updates earn 3.8× more citations than cosmetic refreshes
- Display a visible "Last Updated: [date]" on the page
- The `dateModified` in your Article schema must match the visible date

---

## 13. Topic Cluster Architecture

**82.5% of ChatGPT citations link to pages within established topic hierarchies.**

Structure:
```
Pillar Page: "Complete Guide to [Topic]" (2,000-4,000 words)
├── Cluster: "How to [subtopic A]" (1,200-2,000 words)
├── Cluster: "Best [tools] for [subtopic B]" (1,500-2,500 words)
├── Cluster: "[Topic] vs [Alternative]" (1,000-1,500 words)
├── Cluster: "[Topic] for [specific audience]" (1,200-2,000 words)
└── Cluster: "Common [topic] mistakes and how to fix them" (1,000-1,500 words)
```

Rules:
- Pillar links to all cluster pages
- Each cluster page links back to pillar
- Cluster pages cross-link to related clusters
- Use descriptive anchor text
- Bidirectional internal linking delivers 2.7× citation multiplier

---

## 14. Off-Site Authority Building

**Brands are 6.5× more likely to be cited through third-party mentions than their own website.**

Priority platforms for AI citations:
1. **YouTube**: 39.2% of social citations in AI Overviews. Create demo videos, tutorials, case studies.
2. **Reddit**: 40.1% of AI responses reference Reddit. Build authentic presence in relevant subreddits.
3. **Review platforms**: G2, Clutch, Capterra, Trustpilot provide 4.6-6.3× citation multiplier.
4. **Industry comparison articles**: Get listed in "best [service] in [location/niche]" articles.
5. **Guest posts**: Contribute to industry publications with expert commentary.
6. **Wikipedia/Wikidata**: Create or update entity entries for your brand/key people.
7. **Google Business Profile**: Primary data source for local AI recommendations.

Domains with active profiles across 4+ third-party platforms see 2.8× citation likelihood increase.

---

## 15. Intent Modifier Phrases

ChatGPT appends modifier phrases to internal search queries when users ask questions. Including these naturally in your content improves retrieval:

- "detailed guide"
- "step-by-step tutorial"
- "in-depth comparison"
- "comprehensive overview"
- "complete guide"
- "how to [specific task]"
- "best [thing] for [use case]"
- "[thing] vs [alternative]"
- "pricing", "cost", "how much"
- "pros and cons"
- "review" / "honest review"

Don't stuff these artificially. Use them in headings and opening paragraphs where they fit naturally.

---

## 16. Brand Sentiment Auditing

Regularly test what AI tools say about your brand:

1. Ask each major AI tool: "Give me pros and cons of [Your Brand]"
2. Ask: "What are the best [your service type] companies in [your location]?"
3. Ask: "Compare [Your Brand] vs [Competitor]"
4. Document what's accurate, what's wrong, what's missing

If AI is hallucinating facts about you:
- Update your About page with clear, declarative sentences
- Update llms.txt with correct information
- Add specific factual claims to schema markup
- Build corrective third-party content (reviews, profiles, mentions)

AI systems pick up corrections faster from multiple independent sources than from your own site alone.

---

## 17. What NOT to Do

**Content that gets ignored or filtered by AI:**

- **Sales copy and promotional language**: AI filters out overtly commercial content
- **Vague, hedging language**: "It depends" answers get skipped in favor of specific ones
- **Thin content under 500 words**: Rarely provides enough value for citation
- **AI-generated content at scale without editorial enhancement**: Produces patterns AI learns to skip
- **Keyword-stuffed content**: The original GEO research showed keyword stuffing produced zero gains
- **Content with no original information**: If AI can generate the same content itself, it won't cite you
- **Content behind login walls**: AI crawlers cannot access paywalled content
- **Information scattered across multiple thin pages**: Consolidate into one page that covers the whole topic
- **Timestamp-only refreshes**: Changing the date without substantive updates doesn't fool AI systems
- **Ignoring a platform**: Only 11-14% of citations overlap between platforms; optimize for all

**The Chegg lesson:** If your content is commodity information that AI can generate independently, you'll be displaced. Content must include what AI cannot replicate: original data, proprietary research, real case studies with specific metrics, expert quotes from named professionals, and first-person experience.

# AI Crawler Reference

Complete reference of AI crawler user-agents, which platforms they serve, and how to configure access.

## Table of Contents

1. [Crawler Quick Reference Table](#1-crawler-quick-reference-table)
2. [OpenAI Crawlers](#2-openai-crawlers)
3. [Anthropic (Claude) Crawlers](#3-anthropic-claude-crawlers)
4. [Google AI Crawlers](#4-google-ai-crawlers)
5. [Perplexity Crawlers](#5-perplexity-crawlers)
6. [Microsoft/Bing Crawlers](#6-microsoftbing-crawlers)
7. [Other AI Crawlers](#7-other-ai-crawlers)
8. [Complete robots.txt Template](#8-complete-robotstxt-template)
9. [Platform Submission Guide](#9-platform-submission-guide)
10. [GA4 AI Traffic Tracking](#10-ga4-ai-traffic-tracking)
11. [Agentic Browsers](#11-agentic-browsers)
12. [Content Signals in robots.txt](#12-content-signals-in-robotstxt)

---

## 1. Crawler Quick Reference Table

| User-Agent | Company | Purpose | Block = |
|-----------|---------|---------|---------|
| `GPTBot` | OpenAI | Training data collection | Less model familiarity |
| `OAI-SearchBot` | OpenAI | Search index for ChatGPT Search | Invisible in ChatGPT search |
| `ChatGPT-User` | OpenAI | Real-time user-triggered fetch | May not respect robots.txt |
| `ClaudeBot` | Anthropic | Training data collection | Less model familiarity |
| `Claude-SearchBot` | Anthropic | Search index for Claude web search | Invisible in Claude search |
| `Claude-User` | Anthropic | Real-time user-triggered fetch | May not respect robots.txt |
| `Googlebot` | Google | Search index (also feeds AI Overviews) | No Google visibility at all |
| `Google-Extended` | Google | Gemini training data only | No Gemini training (AI Overviews unaffected) |
| `PerplexityBot` | Perplexity | Search index + real-time crawl | Invisible in Perplexity |
| `Bingbot` | Microsoft | Bing index (feeds ChatGPT + Copilot) | No Bing/ChatGPT/Copilot visibility |

**Key insight:** For maximum AI search visibility, you must allow at minimum: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot, and Bingbot. The training crawlers (GPTBot, ClaudeBot, Google-Extended) are optional but recommended.

---

## 2. OpenAI Crawlers

**GPTBot**
- Purpose: Collects web content for training OpenAI models
- User-agent string: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)`
- Respects: robots.txt
- Impact of blocking: 73% fewer ChatGPT citations (correlation, not direct causation)

**OAI-SearchBot**
- Purpose: Indexes content for ChatGPT's web search feature
- User-agent string: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)`
- Respects: robots.txt
- Impact of blocking: Removed from ChatGPT search results

**ChatGPT-User**
- Purpose: Real-time page fetching when a user's conversation triggers web lookup
- User-agent string: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)`
- Respects: robots.txt partially (operates on behalf of human user)
- Impact of blocking: Reduced but not eliminated ChatGPT access

---

## 3. Anthropic (Claude) Crawlers

**ClaudeBot**
- Purpose: Collects web content for training Claude models
- User-agent string: `claudebot` (case variations exist)
- Respects: robots.txt (including non-standard Crawl-delay directive)
- Impact of blocking: Less model familiarity with your content

**Claude-SearchBot**
- Purpose: Indexes content for Claude's web search (via Brave Search supplementation)
- User-agent string: `Claude-SearchBot`
- Respects: robots.txt
- Impact of blocking: Reduced visibility in Claude web search

**Claude-User**
- Purpose: Real-time fetching during user conversations with web search enabled
- User-agent string: `Claude-User`
- Respects: robots.txt partially (operates on behalf of human user)
- Impact of blocking: Reduced but not eliminated Claude access

**Note:** Claude's search primarily uses Brave Search's independent index. Ensure your site is visible in Brave Search for maximum Claude visibility.

---

## 4. Google AI Crawlers

**Googlebot**
- Purpose: Google's main search crawler. Feeds both traditional search AND AI Overviews.
- User-agent string: Various (Googlebot, Googlebot-Mobile, etc.)
- Respects: robots.txt
- Impact of blocking: Complete loss of Google search AND AI Overview visibility
- **Critical:** You CANNOT block AI Overviews without losing all Google search visibility. There is no separate AI Overview crawler.

**Google-Extended**
- Purpose: Controls training data for Gemini and Vertex AI products
- User-agent string: `Google-Extended`
- Respects: robots.txt
- Impact of blocking: Blocks Gemini training. Does NOT affect AI Overviews or Google Search.
- **Note:** Blocking Google-Extended is a viable choice if you want search visibility but don't want to contribute to Gemini training.

---

## 5. Perplexity Crawlers

**PerplexityBot**
- Purpose: Indexes content for Perplexity's AI search engine
- User-agent string: `PerplexityBot`
- Respects: robots.txt
- Impact of blocking: Invisible in Perplexity search results

Perplexity maintains its own independent index. It's the most citation-transparent platform (numbered inline citations in every response). Reddit content dominates at 46.5% of Perplexity citations.

**Perplexity Publishers' Program:** Partnership program with revenue sharing (80/20 split favoring publishers). Currently 30+ partners including TIME, Fortune, Der Spiegel.

---

## 6. Microsoft/Bing Crawlers

**Bingbot**
- Purpose: Bing's main search crawler. Feeds Bing Search, ChatGPT (via Bing), and Microsoft Copilot.
- User-agent string: `Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)`
- Respects: robots.txt
- Impact of blocking: Complete loss of Bing, ChatGPT (Bing-backed), and Copilot visibility

**Critical:** Bingbot is the single most impactful crawler for ChatGPT visibility. Industry analyses have reported ~87% overlap between ChatGPT citations and Bing's top results (figure undated, verify against a current source before quoting). If you do nothing else, make sure Bingbot can access your site and you've submitted to Bing Webmaster Tools.

**Bing Content Submission API:** Enables pushing structured facts directly for Copilot sidebar answers.

---

## 7. Other AI Crawlers

| User-Agent | Company | Purpose |
|-----------|---------|---------|
| `cohere-ai` | Cohere | Training/search |
| `Bytespider` | ByteDance | Training for AI products |
| `Applebot` | Apple | Apple Intelligence/Siri |
| `Applebot-Extended` | Apple | Apple Intelligence training |
| `FacebookBot` | Meta | Meta AI features |
| `meta-externalagent` | Meta | Meta AI training |
| `Amazonbot` | Amazon | Alexa/AI features |
| `Timesbot` | Perplexity (for publisher partners) | Premium content |

---

## 8. Complete robots.txt Template

### Maximum AI Visibility (recommended)

```
# Standard search
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# AI Search (required for AI search visibility)
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# AI Training (recommended for maximum citation likelihood)
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /
Crawl-delay: 10

User-agent: Google-Extended
Allow: /

# Apple Intelligence
User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Allow: /

# Blocked paths (customize per site)
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /tmp/
Disallow: /private/
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/

Sitemap: https://yoursite.com/sitemap.xml
```

### Selective (search only, no training)

```
# Standard search
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# AI Search only
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Block AI training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://yoursite.com/sitemap.xml
```

---

## 9. Platform Submission Guide

### Google Search Console
- URL: https://search.google.com/search-console
- Submit: XML sitemap
- Feeds: Google Search + AI Overviews + Gemini
- Monitor: Performance report (AI Overview data merged under "Web" search type)

### Bing Webmaster Tools
- URL: https://www.bing.com/webmasters
- Submit: XML sitemap + implement IndexNow
- Feeds: Bing + ChatGPT + Microsoft Copilot
- **This is the #1 priority submission for ChatGPT visibility**

### Brave Search
- Brave uses its own independent index (Web Discovery Project)
- No manual submission process
- Focus on standard SEO + allowing Brave's crawler
- Claude's search draws from Brave's index

### Perplexity
- No manual submission
- Allow PerplexityBot in robots.txt
- Perplexity's own index discovers content independently
- Publishers' Program for partnership: https://www.perplexity.ai/hub/partnerships

### Google Business Profile
- URL: https://business.google.com
- Critical for local/service businesses
- Primary data source for AI local recommendations
- Complete all fields: services, hours, photos, Q&A, posts

---

## 10. GA4 AI Traffic Tracking

### Create an AI referral traffic segment

In GA4, go to Explore > create a custom segment with traffic source filter:

**Regex for AI referral sources:**
```
perplexity\.ai|chatgpt\.com|chat\.openai\.com|claude\.ai|gemini\.google\.com|copilot\.microsoft\.com|you\.com|phind\.com
```

Apply this to:
- Session source/medium (for full session data)
- Landing page (to see which pages AI tools are sending traffic to)

### Track AI-influenced conversions

Add "How did you find us?" to lead forms with options:
- ChatGPT
- Claude
- Perplexity
- Google AI Overview
- Other AI tool

This captures dark AI traffic (users who get a recommendation from AI but then go straight to your site).

### Isolate likely AI Overview impressions in Google Search Console

No direct filter exists. Workarounds:
- Filter by query length (10+ words)
- Filter by conversational keywords: "compare", "top", "best", "vs", "how to", "what is"
- These queries are more likely to trigger AI Overviews

---

## 11. Agentic Browsers

A category that matters since late 2025: full browsers where an AI agent drives real user sessions. These are NOT crawlers and mostly cannot be managed via robots.txt.

| Product | Company | Notes |
|-|-|-|
| Atlas | OpenAI | Launched October 2025. Agent Mode executes multi-step tasks autonomously |
| Comet | Perplexity | Chromium-based; traffic looks like a normal Chrome session |
| Gemini in Chrome | Google | The main consumer of WebMCP tools during the origin trial |

**Implications for GEO:**
- Their traffic is largely indistinguishable from human sessions in analytics; you cannot segment it with a user-agent regex.
- They interact with your site through the rendered page and the accessibility tree, which is why Lighthouse's Agentic Browsing category audits a11y and CLS (see technical-checklist section 13).
- Sites that work well for keyboard/screen-reader users work well for agentic browsers. Broken forms, unlabeled buttons, and layout shift break agent task completion (and the sale that came with it).

---

## 12. Content Signals in robots.txt

Cloudflare's Content Signals Policy (September 2025) extends robots.txt with usage-category signals, expressed per user-agent group:

```
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
```

The three signals: `search` (build a search index), `ai-input` (feed content into AI answers at query time, i.e. RAG/grounding), `ai-train` (training or fine-tuning). The policy text frames restrictions as an express reservation of rights under EU DSM Directive 2019/790 Article 4.

**Why a GEO skill cares:**
- **Cloudflare's managed robots.txt injects `search=yes, ai-train=no` for 3.8M+ domains.** If the site sits behind Cloudflare, fetch the LIVE robots.txt and check whether a managed policy has been layered on top of (or replaces) the site's own file. A client maximizing AI visibility may be sending "don't train on me" without knowing it.
- For maximum AI visibility, either omit content signals entirely or set them permissively. `ai-input=no` works against citation in AI answers.
- Same audit lesson as user-agents: the robots.txt you deploy is not always the robots.txt the world sees. CDN-level bot management (Cloudflare Bot Fight Mode, WAF rules, security plugins) can also block AI crawlers before robots.txt is ever read; test with `curl -A "GPTBot" https://site.com/` and check for 403s.

# Technical Checklist for AI Search Visibility

Complete technical implementation guide. Work through sequentially for new builds, or use as an audit checklist for existing sites.

## Table of Contents

1. [Server-Side Rendering](#1-server-side-rendering)
2. [Robots.txt Configuration](#2-robotstxt-configuration)
3. [Schema.org Structured Data](#3-schemaorg-structured-data)
4. [XML Sitemap + RSS/Atom Feed](#4-xml-sitemap--rssatom-feed)
5. [llms.txt File](#5-llmstxt-file)
6. [Page Speed Optimization](#6-page-speed-optimization)
7. [Search Console Submissions](#7-search-console-submissions)
8. [IndexNow Implementation](#8-indexnow-implementation)
9. [Markdown Alternate Links](#9-markdown-alternate-links)
10. [Meta Tags for AI Extraction](#10-meta-tags-for-ai-extraction)
11. [HTTPS and Security](#11-https-and-security)
12. [Internal Linking Architecture](#12-internal-linking-architecture)
13. [Lighthouse Agentic Browsing Audit](#13-lighthouse-agentic-browsing-audit)

---

## 1. Server-Side Rendering

**Priority: CRITICAL. This is a binary gate**

Most AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) do NOT execute JavaScript. Analysis of 500M+ GPTBot fetches showed zero JS execution. Only Googlebot reliably renders JS.

**Audit check:**
```bash
# Test what a non-JS crawler sees
curl -s -A "GPTBot" https://yoursite.com/page | grep -c "your-expected-content"
# If 0 matches, your content is invisible to AI crawlers
```

**For PHP sites:** This is rarely an issue since PHP renders server-side. But check for:
- Content loaded via AJAX after page load
- React/Vue/Angular widgets embedded in PHP pages
- Lazy-loaded content that requires scroll events
- Content behind JavaScript tabs/accordions (use `<details>`/`<summary>` instead)

**For JS frameworks (React, Vue, Angular, Next.js, Nuxt):**
- Enable SSR or Static Site Generation (SSG)
- Use `getServerSideProps` (Next.js) or equivalent
- Pre-render critical pages at minimum

---

## 2. Robots.txt Configuration

**Priority: CRITICAL. Blocking crawlers = invisible to AI**

See `crawler-reference.md` for complete user-agent list. Here's the recommended robots.txt template:

```
# === Standard Search Crawlers ===
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# === AI Search Crawlers (allow for search visibility) ===
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# === AI Training Crawlers (optional, allow if you want to be in training data) ===
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

# === Block paths that shouldn't be crawled ===
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /tmp/
Disallow: /private/

Sitemap: https://yoursite.com/sitemap.xml
```

**Important distinctions:**
- **Search crawlers** (OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, PerplexityBot): These power real-time search features. Blocking them removes you from AI search results.
- **Training crawlers** (GPTBot, ClaudeBot, Google-Extended): These collect data for model training. Blocking them doesn't affect real-time search but reduces long-term model familiarity with your content. Sites blocking GPTBot were cited 73% less often.
- ChatGPT-User and Claude-User operate on behalf of human users and may not fully respect robots.txt blocks.

**Audit check:**
```bash
# Fetch and inspect current robots.txt
curl -s https://yoursite.com/robots.txt
# Look for blanket "Disallow: /" rules or blocks on AI user-agents
```

---

## 3. Schema.org Structured Data

**Priority: HIGH. GPT-4 accuracy jumps from 16% to 54% with structured data**

All schema must be:
- In JSON-LD format (Google-recommended, easiest for AI to parse)
- Rendered server-side in the HTML `<head>` (not injected via JS)
- Accurately matching visible page content (mismatches reduce trust)

### Schema templates by page type

**Organization (site-wide, on every page):**
```json
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Your Company Name",
  "url": "https://yoursite.com",
  "logo": "https://yoursite.com/logo.png",
  "description": "One-sentence description of what you do",
  "foundingDate": "2020",
  "address": {
    "@type": "PostalAddress",
    "streetAddress": "123 Main St",
    "addressLocality": "Zagreb",
    "addressCountry": "HR"
  },
  "contactPoint": {
    "@type": "ContactPoint",
    "email": "info@yoursite.com",
    "contactType": "customer service"
  },
  "sameAs": [
    "https://www.linkedin.com/company/yourcompany",
    "https://www.facebook.com/yourcompany",
    "https://twitter.com/yourcompany"
  ]
}
```

**Article / BlogPosting:**
```json
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Article Title Here",
  "description": "2-sentence summary of what this article covers",
  "author": {
    "@type": "Person",
    "name": "Author Name",
    "url": "https://yoursite.com/about/author-name",
    "jobTitle": "Author's Role/Title",
    "sameAs": ["https://linkedin.com/in/author"]
  },
  "publisher": {
    "@type": "Organization",
    "name": "Your Company Name",
    "logo": { "@type": "ImageObject", "url": "https://yoursite.com/logo.png" }
  },
  "datePublished": "2026-03-01",
  "dateModified": "2026-03-20",
  "mainEntityOfPage": "https://yoursite.com/blog/article-slug",
  "image": "https://yoursite.com/images/article-hero.jpg"
}
```

**FAQPage (powerful for AI extraction):**
```json
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "What is [topic]?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Direct, complete answer in 40-80 words."
      }
    },
    {
      "@type": "Question",
      "name": "How does [topic] compare to [alternative]?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Specific comparison with concrete details."
      }
    }
  ]
}
```

**Product (for e-commerce/service pages):**
```json
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Product Name",
  "description": "Clear product description",
  "image": "https://yoursite.com/images/product.jpg",
  "brand": { "@type": "Brand", "name": "Brand Name" },
  "sku": "SKU-12345",
  "offers": {
    "@type": "Offer",
    "price": "99.00",
    "priceCurrency": "EUR",
    "availability": "https://schema.org/InStock",
    "url": "https://yoursite.com/products/product-name"
  },
  "aggregateRating": {
    "@type": "AggregateRating",
    "ratingValue": "4.5",
    "reviewCount": "42"
  }
}
```

**HowTo (for tutorials/guides):**
```json
{
  "@context": "https://schema.org",
  "@type": "HowTo",
  "name": "How to [do thing]",
  "description": "Brief overview",
  "step": [
    {
      "@type": "HowToStep",
      "name": "Step 1 title",
      "text": "Step 1 instructions"
    }
  ]
}
```

**LocalBusiness (for agency/local service sites):**
```json
{
  "@context": "https://schema.org",
  "@type": "ProfessionalService",
  "name": "Agency Name",
  "description": "What you do in one sentence",
  "url": "https://yoursite.com",
  "address": { ... },
  "geo": { "@type": "GeoCoordinates", "latitude": "45.815", "longitude": "15.982" },
  "openingHours": "Mo-Fr 09:00-17:00",
  "priceRange": "$$",
  "areaServed": "Croatia"
}
```

### PHP implementation pattern

```php
// In your base template or header include
function render_schema_jsonld(array $schema_data): string {
    return '<script type="application/ld+json">' .
           json_encode($schema_data, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE | JSON_PRETTY_PRINT) .
           '</script>';
}

// Per page template, build the schema array and render in <head>
$schema = [
    '@context' => 'https://schema.org',
    '@type' => 'Article',
    'headline' => $article->title,
    'datePublished' => $article->published_at,
    'dateModified' => $article->updated_at,
    // ... etc
];
echo render_schema_jsonld($schema);
```

---

## 4. XML Sitemap + RSS/Atom Feed

**Priority: HIGH**

**XML Sitemap requirements:**
- Include all indexable pages
- Accurate `<lastmod>` timestamps (critical for AI crawlers detecting fresh content)
- Split by content type if site is large (blog-sitemap.xml, product-sitemap.xml)
- Submit to both Google Search Console AND Bing Webmaster Tools
- Reference in robots.txt: `Sitemap: https://yoursite.com/sitemap.xml`

**RSS/Atom feed:**
- Google recommends pairing sitemaps with RSS/Atom feeds for change detection
- Microsoft's NLWeb protocol uses RSS feeds as a primary data source
- Include at minimum: title, link, description, pubDate/updated, author
- Auto-generate from your CMS or build a `/feed.xml` endpoint

```xml
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Site Name</title>
    <link>https://yoursite.com</link>
    <description>Site description</description>
    <atom:link href="https://yoursite.com/feed.xml" rel="self" type="application/rss+xml"/>
    <item>
      <title>Article Title</title>
      <link>https://yoursite.com/blog/slug</link>
      <description>First 200 chars of article</description>
      <pubDate>Wed, 20 Mar 2026 10:00:00 +0000</pubDate>
      <guid>https://yoursite.com/blog/slug</guid>
    </item>
  </channel>
</rss>
```

---

## 5. llms.txt File

**Priority: LOW-MEDIUM (uneven adoption as a retrieval source, but since 2026 Lighthouse/PageSpeed Insights validates the file, so a malformed one shows up as a red audit to anyone who tests your site)**

Place at `https://yoursite.com/llms.txt`. Markdown-formatted overview of your site for AI consumption. Takes minutes to create, costs nothing.

**Format is now machine-validated.** Lighthouse's Agentic Browsing category (see section 13) parses the file as markdown and fails it unless:
- It contains exactly the spec structure: an H1 with the site/project name (the only hard-required element), then optionally a blockquote summary, prose, and H2 sections.
- Link entries use real markdown link syntax: `- [Name](https://url): note`. Plain `Title: URL` lines do NOT count and fail the audit with "File does not appear to contain any links".
- The file is longer than ~50 characters.
- Gotcha: avoid `---` horizontal-rule lines. Markdown parsers can read the first one as a YAML front-matter delimiter and discard everything above it, including the H1, so the audit reports "missing an H1 header" even though the file visibly has one.

```markdown
# Site Name

> One-sentence description of what this site is about.

## About
Brief 2-3 sentence description of the organization, what it does, who it serves.

## Key Pages
- [Homepage](https://yoursite.com/): Brief description
- [Services](https://yoursite.com/services/): What services are offered
- [Blog](https://yoursite.com/blog/): Topics covered

## Popular Content
- [Article Title](https://yoursite.com/blog/article): One-line summary
- [Guide Title](https://yoursite.com/guides/guide): One-line summary

## Contact
- Email: info@yoursite.com
- Location: Zagreb, Croatia
```

Optionally create `llms-full.txt`: the same overview followed by the full markdown of your key pages/articles, so a tool gets deep context in one fetch. This is cheap when the site already exposes per-page markdown (e.g. a `/article/markdown` endpoint or a content/export service): a single route can concatenate the `llms.txt` intro with each published page's markdown. Serve it `X-Robots-Tag: noindex, follow` (it duplicates page bodies, so avoid search-index duplicate content) and exclude any gated/premium content so you don't leak it. When you bound the dump (top-N, page cap), say what was dropped rather than silently truncating.

**Priority:** LOW / optional. Future-proofing, not a required output.

**Current status (as of 2026-07, per third-party trackers, not primary sources; verify before quoting):** Adoption is growing but uneven (roughly 10% of large domain samples). Some providers have confirmed support (Anthropic by late 2024, Perplexity by 2025), and by 2026 most major Western AI platforms offer partial support. The most concrete, verifiable adoption is in AI dev tools: Claude Code, Cursor, Windsurf, GitHub Copilot, Cline, Aider all fetch `/llms.txt` and `/llms-full.txt`, especially for documentation/reference sites. Google has said it does not use llms.txt for Search, and as of Q1 2026 no major AI provider (OpenAI, Google, Anthropic, Meta, Mistral) has publicly committed to reading it in production answer systems. What HAS changed: Google now validates the file in Lighthouse and PageSpeed Insights (Agentic Browsing category, section 13), which makes it a visible pass/fail checkbox for clients, tools, and audits. Bottom line: still don't prioritize it over robots.txt, schema, sitemap, or content structure, but if you ship one, it MUST follow the format rules above or it hurts you in audits.

---

## 6. Page Speed Optimization

**Priority: HIGH. Pages with FCP < 0.4s get 3× more AI citations**

Target metrics:
- First Contentful Paint (FCP): < 0.4 seconds
- Time to First Byte (TTFB): < 200ms
- Largest Contentful Paint (LCP): < 2.5 seconds

AI crawlers abandon slow pages more aggressively than traditional search crawlers.

Quick wins for PHP sites:
- Enable OPcache
- Use a reverse proxy cache (Varnish, Nginx FastCGI cache)
- Optimize images (WebP, lazy loading for below-fold)
- Minimize CSS/JS blocking resources
- Use a CDN for static assets
- Database query optimization (avoid N+1 queries)

---

## 7. Search Console Submissions

**Priority: HIGH**

Submit your sitemap to both:
- **Google Search Console**: https://search.google.com/search-console
- **Bing Webmaster Tools**: https://www.bing.com/webmasters

Bing is critical because ChatGPT and Copilot both use Bing's index. 87% of ChatGPT citations match Bing's top results.

---

## 8. IndexNow Implementation

**Priority: MEDIUM-HIGH**

IndexNow instantly notifies Bing (and by extension ChatGPT/Copilot) when content is published or updated.

```php
// After publishing or updating content
function notify_indexnow(string $url, string $api_key): void {
    $endpoint = "https://api.indexnow.org/indexnow";
    $params = http_build_query([
        'url' => $url,
        'key' => $api_key,
        'host' => parse_url($url, PHP_URL_HOST),
    ]);

    $ch = curl_init("{$endpoint}?{$params}");
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_TIMEOUT, 5);
    curl_exec($ch);
    curl_close($ch);
}
```

Place your API key file at `https://yoursite.com/{api-key}.txt`.

---

## 9. Markdown Alternate Links

**Priority: LOW (experimental, but zero-cost if you have Markdown source)**

If you write content in Markdown (e.g., from Obsidian, a CMS, or a static site generator), serve the raw Markdown version alongside HTML and signal it:

```html
<link rel="alternate" type="text/markdown" href="/blog/post-slug.md" />
```

AI models process Markdown natively. This gives them a clean, uncluttered version of your content.

---

## 10. Meta Tags for AI Extraction

**Priority: MEDIUM**

Ensure every important page has:
```html
<meta name="description" content="Concise, factual 150-160 char description" />
<meta name="author" content="Author Name" />
<meta property="og:title" content="Page Title" />
<meta property="og:description" content="Same as meta description" />
<meta property="og:type" content="article" />
<meta property="og:url" content="https://yoursite.com/page" />
<meta property="og:image" content="https://yoursite.com/images/og-image.jpg" />
<meta property="article:published_time" content="2026-03-01T10:00:00Z" />
<meta property="article:modified_time" content="2026-03-20T10:00:00Z" />
```

---

## 11. HTTPS and Security

**Priority: CRITICAL (baseline requirement)**

All pages must be served over HTTPS. This is a ranking factor for Google/Bing and a trust signal for AI systems. No exceptions.

---

## 12. Internal Linking Architecture

**Priority: HIGH. Pillar-cluster architecture delivers 2.7× citation multiplier**

Structure sites as topic clusters:
- **Pillar page**: Covers a whole topic end to end (2000+ words)
- **Cluster pages**: Detailed sub-topic articles linking back to pillar
- **Bidirectional links**: Pillar links to clusters, clusters link to pillar
- Use descriptive anchor text (not "click here")

82.5% of ChatGPT citations link to pages within established topic hierarchies, not isolated articles.

---

## 13. Lighthouse Agentic Browsing Audit

**Priority: MEDIUM (new in 2026, the first mainstream tool that grades agent-readiness; category is explicitly "under development and subject to change")**

Chrome Lighthouse (Chrome 150+, also surfaced on pagespeed.web.dev) added an "Agentic Browsing" category that scores how well a site works for AI agents. Unlike other categories there is no 0-100 weighted score: each audit reports pass/fail (or a pass ratio) independently.

**What it checks:**

1. **llms.txt presence and format**: see section 5 for the exact validation rules (H1 required, markdown `[text](url)` links required, minimum length, no `---` front-matter trap).
2. **WebMCP integration**: WebMCP (Web Model Context Protocol) is a W3C proposal that lets a page expose callable tools to AI agents via `navigator.modelContext` instead of the agent guessing at the UI. Announced February 2026, in Chrome origin trial (Chrome 149-156); the main consuming agent today is Gemini in Chrome. Lighthouse verifies both declarative tools (defined in HTML) and imperative tools (registered from JS). Only relevant for interactive sites (search, cart, booking, filters); content/publishing sites can safely skip it. Reference: https://developer.chrome.com/docs/ai/webmcp
3. **Agent-centric accessibility subset**: agents drive pages through the accessibility tree, so Lighthouse filters the a11y audits that matter for machine interaction: every interactive element has a programmatic name, ARIA roles are valid and on compatible elements, identical links have the same purpose, touch targets have sufficient size/spacing. Accessibility fixes now double as agent-readiness fixes; pitch them that way.
4. **Layout stability (CLS)**: agents target elements by position, so layout shift breaks them the same way it annoys humans.

**Audit check:** run the site through https://pagespeed.web.dev/ and read the Agentic Browsing section at the bottom of the report, or use Lighthouse in Chrome DevTools (Chrome 150+).

**Watch for false positives:** entrance animations that fade content in (opacity 0 to 1 on scroll-reveal) can make the contrast and visibility audits fail elements whose static styles pass. Verify the computed static colors before "fixing" contrast findings caused by animation timing.

---
name: a-seo-gsc
description: Turn Search Console data, fetched over the API or exported by hand, into a ranked, evidence-backed action plan.
disable-model-invocation: true
---

# a-seo-gsc: Search Console → action plan

Google Search Console already knows what is wrong with a site's search performance. The data is just buried in CSV exports nobody reads. This skill ingests those exports, runs a deterministic parser over them, applies a full analysis playbook (adapting to the site type), and writes a prioritized action plan you can actually execute. It works on any website, not one specific project.

The output is a doc: `docs/seo/YYYY-MM-DD-action-plan.md` in the target project (or wherever the user keeps analyses), plus a spoken summary of the headline finding and the top moves.

## First run: what to give me, and where to get it

**Ask for the property name first, not for files.** If a service-account key exists for it, the Performance half needs no clicking:

```bash
python3 scripts/fetch_gsc.py --property sc-domain:example.com --days 90
```

That writes `current/` and `previous/` directories of CSVs in Google's own export shape. Setup, if the property has no key yet, is in `references/export-guide.md`; it is a one-time service account plus adding its address as a property user.

Then ask for the two things no API serves:

> Two exports I cannot fetch, both from **Indexing → Pages**:
> 1. Click **Export** on the summary. That gives the indexed vs not-indexed counts and the reason breakdown.
> 2. Click the biggest "not indexed" reason (usually "Crawled - currently not indexed"), then **Export** the URL table. Repeat for any other large reason. This is the one people skip and the most useful file in the set.
> Drop the zips in a folder next to the fetched data. Optional extras: the **Links** report (top linking sites, also no API), **Sitemaps**, and **Core Web Vitals**.

If the API is not set up for the property, fall back to the full manual export list in `references/export-guide.md`. The parser reads both identically.

Then ask for the site's **brand terms** (name, domain, product names) so branded vs non-branded traffic can be split, and what the site is **for** (blog, ecommerce, SaaS, docs, local business) so the playbook adapts.

## Modes

| Mode | Trigger | What happens |
|------|---------|--------------|
| **Fetch** | Property named, key present | Run `fetch_gsc.py`, then ask only for the two coverage exports |
| **Intake** | No data yet, or no API access | Give the export instructions above, collect brand terms + site type |
| **Analyze** | Exports in hand | Run the parser, then work the playbook, report findings |
| **Plan** | After analyze | Write the tiered action-plan doc to `docs/seo/` |
| **Competitors** | On request, or when content gaps dominate | Dependency-free competitor lens (see below) |

Analyze and Plan usually run together. Do not skip Plan: the written, prioritized doc is the deliverable.

## The method

### 1. Parse (deterministic, cheap)
Run the parser over whatever the user gave you. It auto-detects Performance / Coverage / Drilldown files, normalizes the numbers, and prints findings.

```bash
python3 scripts/parse_gsc.py <zip-or-dir-or-csv> [more...] \
    --brand "brand one,brandtwo,domain.com" --min-impressions 5
```
Add `--json` to get machine output, `--out FILE` to save. Read the printed report before reasoning further; it already computes baseline, concentration, striking-distance, CTR underperformers, zero-click queries, parameter-duplicate cannibalization, coverage buckets, the index-starvation-vs-bloat diagnosis, and a categorized not-indexed list.

**Page+query pairs.** A fetched directory carries `pairs.csv`, which no manual export has. It unlocks true query cannibalization: which of your own pages compete for one query, and whether the page Google shows most often is the one that ranks best. Absent that file the section is omitted rather than reported as clean, because nothing was examined. Parse `current/` and `previous/` separately and diff them for the decay dimension.

**Search types.** A downloads folder often holds Performance exports for more than one search type (Web, Image, Video, News). They describe the same URLs from different indexes, so merging them double-counts every page and the duplicates surface as parameter cannibalization that is not real. The parser reads each export's `Filters.csv`, parses one search type, and prints on stderr what it ignored. It defaults to Web; pass `--search-type image` for another. Coverage and drilldown CSVs are never filtered, so they always come through. Read the stderr note: it tells you which dataset the numbers below it describe.

**Read the index before the data.** If the project keeps a `docs/seo/README.md` or equivalent, read it first. It should say which analysis is current, what each past one settled, and what checks are outstanding. Reconstructing that by opening six dated files is the expensive way to start a run.

### 2. Analyze (judgment on top of the numbers)
Open `references/analysis-playbook.md` and walk every dimension that the data supports. The playbook covers far more than any one site will show: striking distance, CTR-vs-position, cannibalization, content gaps, indexing pathologies, decay (needs two date ranges), device/geo mismatch, rich-result opportunities, authority/linking signals, and site-type-specific failure modes (blog vs ecommerce vs SaaS vs docs vs local vs directory). Only report what the data actually supports, and say what it cannot show.

The non-negotiable discipline (from hard-won review experience): **separate observation from hypothesis from action.** "Impressions dropped" is an observation. "Googlebot got empty HTML" is a hypothesis until verified. Do not let a plausible story become a confident diagnosis. When you cannot prove causation (and a single export almost never can), say so and propose the check that would confirm it.

### 3. Plan (the deliverable)
Use `references/action-plan.md` as the template. Write findings evidence-first, then a tiered action list ranked by impact-over-effort, then a tracking table (one row per action: page/query, baseline, change date, expected effect, 28-day result) so next month's run can tell whether anything worked. Save to `docs/seo/YYYY-MM-DD-action-plan.md`. Create `docs/seo/` if absent. If the project has no `docs/`, ask where to save or offer the repo root.

### 4. Competitors (optional, dependency-free)
See `references/competitor-analysis.md`. Identify who ranks for the site's striking-distance and target queries (via `WebSearch`), fetch their ranking pages (`WebFetch`), and compare depth, structure, schema, freshness, and angle coverage. Produces a content-gap matrix and prioritized competitive moves. No paid API required; if the user has an Ahrefs/Semrush export, it plugs in as an extra CSV for real volume and backlink numbers.

## Principles

1. **Impressions are not the same as the right audience.** A page can soak up impressions from a query cluster that has nothing to do with the site's purpose. Growing that is vanity. Always sanity-check whether the winning queries match who the site is for.
2. **Index-starvation vs index-bloat are opposite problems with opposite fixes.** Starvation (good pages not indexed): fix authority, internal linking, and request indexing; do NOT publish more. Bloat (thin/duplicate pages diluting crawl): consolidate, noindex, canonicalize. The parser flags which one the data shows. Never prescribe "write more content" for a starved site.
3. **Authority is usually the real gate on a small site.** If only backlinked pages get indexed and rank while internally-linked-only pages sit in "crawled, currently not indexed," the bottleneck is authority and link architecture, not content volume. One earned link often does more than ten new posts.
4. **CTR far below the position curve = a snippet or intent problem, not a ranking problem.** A page at position 9 with 0% CTR usually means the title/meta does not match the query intent, or a SERP feature is eating the click. Fix the snippet, not the ranking.
5. **No controls means self-attributed wins.** Without an untouched comparison set and a dated change log, you cannot prove a fix caused a change. Build the tracking table so the next run can.
6. **Adapt to the site type.** The failure modes of a 300-page blog, a 50k-SKU store, a docs site, and a local business are different. Read the site-type section of the playbook.

## Cross-skill boundaries

- **a-geo-optimizer**, AI-search / GEO visibility (ChatGPT, Perplexity, Gemini citations, llms.txt, answer-capsule content). Use it for the AI-discoverability angle; this skill does not duplicate it.
- **A project-specific technical-SEO skill**, where you have one, audits the *codebase* (meta tags, canonicals, structured data, CWV). This skill diagnoses from *data* and tells that one what to go fix. Run this to find the problem, the code-level skill to fix it.

Your Search Console data is your own and stays on your machine. Nothing here uploads it, and the analysis is entirely local. `fetch_gsc.py` makes exactly one kind of outbound call: it signs in to Google with your own service-account key and asks Google for your own property's data. It talks to `oauth2.googleapis.com` and `www.googleapis.com` and nothing else, there is no telemetry, and the key never leaves your machine or touches disk beyond the file you point it at.

## Reference files

- `references/export-guide.md`, exact GSC export click-paths for every report, how to get two date ranges for trend diffing, and alternative sources (Bing Webmaster Tools, Ahrefs/Semrush CSV).
- `references/analysis-playbook.md`, the full dimension-by-dimension methodology with thresholds, formulas, and per-site-type adaptations.
- `references/action-plan.md`, output doc template and the impact/effort prioritization framework.
- `references/competitor-analysis.md`, the dependency-free competitor lens.
- `scripts/fetch_gsc.py`, the Search Console API fetcher. Standard library only, signs RS256 via openssl.
- `scripts/parse_gsc.py`, the deterministic export parser. Reads fetched and hand-exported data identically.

interface:
  display_name: "SEO from GSC"
  short_description: "Rank the fixes from your Search Console data"
policy:
  allow_implicit_invocation: false

# Action-plan template

The deliverable. Write it to `docs/seo/YYYY-MM-DD-action-plan.md` in the target project. Evidence first, then ranked actions, then a tracking table so the next run can measure what worked.

Keep it honest: every claim ties to a number from the export, every hypothesis is labeled as one, and anything the data cannot show is stated plainly.

## Structure

```markdown
# SEO action plan, <site>, <YYYY-MM-DD>

Source: GSC export, <date range>. Brand terms: <...>. Site type: <...>.

## Snapshot
- Clicks / impressions / CTR / avg position (with the one-line "so what").
- Branded vs non-branded.
- Concentration (top page's share of impressions).

## Headline finding
The single most important thing, stated in one paragraph, with the numbers that prove it and the one hypothesis (labeled) for the cause. This is what the whole plan hinges on.

## Findings by dimension
For each dimension the data supports (indexing, striking distance, CTR, cannibalization, content gaps, decay, device/geo, authority):
- **Observation:** what the numbers say (cite them).
- **Hypothesis:** the likely cause, labeled as a hypothesis.
- **Check:** what would confirm it (so it does not stay a guess).

## Action list (ranked)
Tiered by impact-over-effort. Each action: what, why (which finding it addresses), effort (S/M/L), expected effect, and who does it (code change vs GSC click vs content edit vs outreach).

### Tier 1. Highest impact
### Tier 2. Quick wins / hygiene
### Tier 3. Worth doing, secondary

## What this export cannot tell us
Explicit limits: no volume/difficulty data, no conversion data, causation unproven, query×page pairing unavailable, single snapshot (no trend), etc. Plus what to export next time to close the gap.

## Tracking table
| # | Action | Target (page/query) | Baseline (date) | Changed on | Expected effect | 28-day result |
|---|--------|--------------------|-----------------|-----------|-----------------|---------------|
Leave "Changed on" and "28-day result" blank; they get filled as work ships and at the next run.
```

## Prioritization

Rank by **impact ÷ effort**, not by dimension order. A rough ICE (Impact, Confidence, Ease) or a 2×2 (impact vs effort) is enough; do not over-formalize. Bias toward:

1. **Actions that unblock everything else.** On a starved site, getting content indexed (authority + linking) gates every other win, so it is Tier 1 even though it is slow. On a bloated site, the parameter/canonical cleanup is the unblock.
2. **Zero-click strong positions.** Rewriting a title for a page already at position 5-10 with near-zero CTR is the cheapest click recovery on the board.
3. **Striking distance.** Small pushes on page-2 queries that already have impressions.
4. **Hygiene that stops the bleed.** 404 fixes, param exclusions, junk-URL noindex. Low effort, prevents further dilution.

De-prioritize: chasing off-audience query clusters (vanity impressions), building new content on a site that cannot get its existing content indexed, and anything whose expected effect you cannot articulate.

## Who executes what (label every action)

- **Code change**, needs a dev edit (canonical logic, robots rules, schema, redirects). If the project has a technical-SEO or codebase-audit skill, point there.
- **GSC action**, manual Search Console step (Request Indexing, submit sitemap, validate fix). Give the exact click-path and the prioritized URL list to paste.
- **Content edit**, rewrite title/meta, add a section, consolidate posts, refresh. Route through the project's content workflow (never raw DB edits on content fields).
- **Off-site**, earn a link, get a mention, cross-post. Usually the real lever for authority-gated sites; connect to whatever promotion workflow exists.

## Tone

Write like a senior analyst briefing the owner, not a report generator. Lead with the decision, support with the number, name the uncertainty. No filler, no "it is worth noting," no em dashes. If the honest read is "your content is fine, the problem is nobody links to you," say exactly that.

# Analysis playbook

Work every dimension the data supports. Most sites show only a subset; report those, name what is missing, and never invent a finding the data does not carry. The parser (`scripts/parse_gsc.py`) computes the arithmetic for most dimensions below; you supply judgment and the ones it cannot.

Order below is roughly "most decisive first." On any given site, one or two dimensions dominate. Find the dominant one and lead with it.

---

## 0. Baseline and shape

Read the totals first (from the Dates/Chart file, which is authoritative; query/page files are a top-N sample).

- **Clicks, impressions, CTR, average position.** Position is impression-weighted, so a few high-impression page-2 queries drag it up.
- **Branded vs non-branded** (needs `--brand`). Branded clicks are people who already know the site. If branded is most of the traffic, the site is not winning new audiences. If the site cannot even rank #1 for its own brand, that is a distinct, fixable problem (entity signals, homepage authority).
- **Concentration.** What share of impressions sits on the single top page? High concentration (one page ≫ everything) means the site's search presence is one lucky page, not a system. Ask whether that page is even on-topic for the site.

Sanity flag: if impressions are large but clicks are near zero, the whole site is probably ranking on page 2, or the snippets are wrong, or the impressions are for queries the content does not actually answer.

## 1. Indexing / coverage pathology

Often the real story, and people never look. Diagnose which pattern:

- **Index-starvation:** more pages excluded than indexed, good content sitting in "Crawled - currently not indexed" or "Discovered - currently not indexed." Cause is almost always **authority + internal linking + crawl priority**, not content quality. Fixes: earn links, strengthen internal links from indexed high-authority pages, get hub/index pages indexed, request indexing for priority URLs, submit a clean sitemap. Do NOT tell a starved site to publish more; it just grows the parked pile.
- **Index-bloat:** thousands of thin/duplicate/parameter URLs indexed or clogging crawl. Fixes: canonicalize, noindex thin templates, `Disallow` parameter and faceted URLs, consolidate.
- **Canonical confusion:** large "Duplicate without user-selected canonical" or "Alternate page with proper canonical tag." Check that canonicals are self-consistent and point where you intend.

Coverage reason cheat-sheet (GSC label → what it means → move):

| Reason | Meaning | Move |
|--------|---------|------|
| Crawled - currently not indexed | Google fetched it, chose not to index | Quality/authority/linking. Improve links + request indexing. If genuinely thin, improve or noindex. |
| Discovered - currently not indexed | Known but not even crawled yet | Crawl budget / low priority. Internal links + sitemap + authority. |
| Duplicate without user-selected canonical | Google picked a different canonical | Set explicit canonical; consolidate duplicates. |
| Duplicate, Google chose different canonical than user | Your canonical was overridden | The "duplicate" is stronger; merge or differentiate. |
| Alternate page with proper canonical tag | Correctly canonicalized dupe | Usually fine. Expected for paginated/param/AMP. |
| Page with redirect | 301/302 | Fine if intended (restructure fallout). Audit for redirect chains. |
| Not found (404) | Broken | Fix internal links or 301 to the right page. |
| Soft 404 | Thin/empty page returning 200 | Add content or return real 404. |
| Blocked by robots.txt | Disallowed | Verify intentional; accidental blocks are common after CMS/CDN changes. |
| Blocked due to unauthorized request (401) | Auth-gated | Fine if intended. |
| Server error (5xx) | Broke during crawl | Reliability issue; investigate logs. |

Then **categorize the not-indexed URL list** (the parser buckets by generic path pattern: content, taxonomy-tag, taxonomy-category, product, pagination, parameter-url, feed, redirect-endpoint). Interpret per site: which buckets are junk that should never be index candidates (feeds, redirect endpoints, param/sort URLs, thin tag pages) vs which are real content being wrongly excluded. Junk → exclude from crawl. Real content excluded → the starvation fight above.

## 2. Striking distance

Queries at **positions 11-20** with real impressions are the highest-ROI lever on most sites: they already rank, just below the fold of page 1. Small on-page improvements (title/meta relevance, adding the exact sub-topic, one internal link, a bit more depth) can push them onto page 1 where clicks live. Sort by impressions; the top of that list is the week's to-do. Positions 5-10 with low CTR belong to dimension 3, not here.

## 3. CTR vs position

For queries in the top 10, compare actual CTR to the expected-for-position curve (parser flags actual below 40% of expected). Under-performers are a **snippet or intent problem**, not a ranking problem:
- Title/meta does not match the query intent (page ranks for "X system requirements" but the title says "X overview").
- A SERP feature (featured snippet owned by someone else, People-Also-Ask, a pack, an ad block) is eating the clicks.
- The URL/brand looks untrustworthy in the SERP.

Fixes: rewrite the title to lead with the searcher's outcome and include the query's head term; tighten the meta description; add the structured data that could win the feature (defer schema specifics to `a-geo-optimizer`, or to your project's code-level SEO skill). Zero-click at a strong position is the loudest, cheapest win on the board.

## 3b. How much of the traffic do query rows actually cover?

Before trusting anything computed from query rows, total them and compare against the page rows. Google withholds queries too few people ran, so the query table is a sample, and on a low-volume site it is a small one. Measured on one low-traffic blog over 28 days: page rows carried roughly 50 clicks and 12,000 impressions while query rows carried 4 and 1,900. Query rows saw 16% of impressions and 8% of clicks. Expect the gap to shrink as a site gets busier, and check it rather than assuming.

That does not invalidate striking-distance or CTR work, but it changes what those findings are: statements about the disclosed minority, not about the site. State the coverage ratio in the report. Never present a query-row total as the site's traffic, and never subtract query clicks from page clicks and call the remainder anything.

Related, and worth not chasing: `dates.csv` and `pages.csv` disagree slightly, around 1% on the window above. Both are Google's own aggregates over the same data. Use page rows for totals.

## 4. Cannibalization and duplicate URLs (partial)

- **Parameter/duplicate URLs** (parser detects): the same clean URL indexed under `?ref=`, `?sort=`, `?utm=`, session IDs, tracking tags, pagination. Splits signals and wastes crawl. Fix: self-referential-clean canonical, `Disallow` the params, or 301 the tracked variants.
- **True query cannibalization** (multiple distinct pages competing for one query): the parser detects this whenever a `pairs.csv` is present, which `fetch_gsc.py` writes and no manual export contains. Google's UI export lists queries and pages in separate tables, so from exports alone this stays unprovable and the section is omitted rather than reported as clean. A flagged query has two or more of your pages each holding at least the impression floor, with the runner-up on 20% or more of the query's impressions. Read the **position/impression mismatch** flag first: it means Google shows one page most often while a different one ranks better, which is the clearest case for consolidating. Fix: pick one canonical page, merge the others into it, 301 the losers. (Classic symptom: several near-identical titles, none ranking well, positions bouncing.)
- **Over-splitting a topic:** a cluster of thin posts on near-identical sub-topics that would rank better as one strong page. Consolidation beats proliferation on low-authority sites.

## 5. Content gaps (partial)

Queries the site gets impressions for but has no dedicated page answering. Signals: a broad page ranking at position 9-15 for many specific long-tail variants it does not directly address (e.g., one overview post catching dozens of "X requirements / X pricing / X vs Y" queries). Two responses:
1. If on-topic and worth it: build the dedicated page (or a strong section) that directly answers the cluster.
2. If off-topic for the site's purpose: ignore it. Chasing off-audience queries grows vanity impressions. Judge against who the site is for.

GSC only shows queries you already appear for. For demand you are missing entirely, you need a keyword tool or the competitor lens.

## 6. Decay and trend (needs two date ranges)

`fetch_gsc.py` writes `current/` and `previous/` windows of equal length by default, so this dimension runs without anyone having remembered to tick Compare in the UI. Parse each directory and diff. With hand exports it still depends on the Compare box, and if only one range is present, say the dimension was skipped rather than implying stability.

With a comparison export, diff period-over-period:
- **Declining queries/pages:** falling clicks or impressions, or rising position number. Often a competitor overtook, content went stale, or a redesign/restructure lost signals. Freshness updates and re-earning links help.
- **Position improved but CTR dropped:** you moved up but the SERP got more competitive (features, ads) or the snippet stopped matching. Snippet work.
- **Seasonality:** distinguish a real drop from an annual dip. Year-over-year comparison separates them.
- **Restructure fallout:** a category/URL migration typically shows a wave of new redirects, duplicate-canonical entries, and a not-indexed bump as Google re-evaluates. Time findings against known deploy dates before blaming quality.

Without two ranges, say so and recommend pulling the comparison export next time.

## 7. Device and geo

- **Device CTR gap:** if mobile CTR ≪ desktop at the same position (or vice versa), the mobile SERP layout or the page's mobile snippet/experience is the issue.
- **Geo/language mismatch:** impressions concentrated in countries the site does not serve or in a language it does not publish (common: a homelab/tech post pulling global traffic a personal/consulting site cannot use). Or the reverse: target market underrepresented, implying weak local relevance or hreflang gaps. Match the geo distribution against the site's actual market.

## 8. Rich results / search appearance

The Search-appearance breakdown (and Enhancements report) shows which rich-result types the site is eligible for and which are erroring. Missing eligibility for obvious types (Article, FAQ, Product, Breadcrumb, HowTo) is a structured-data opportunity. Hand the specifics to `a-geo-optimizer` (AI-citation schema) or to whatever skill owns code changes in the project; this skill just flags the gap from the data.

## 9. Authority and internal linking (needs Links export)

- **Referring domains:** few linking sites = low authority = the index-starvation gate. The clearest tell: only backlinked pages are indexed and ranking. One earned link often pulls in a whole cluster.
- **Internal-link distribution:** pages with zero internal links are orphans Google deprioritizes. Ensure every important page is linked from indexed, higher-authority pages (home, hubs, related content), not just from each other.
- **Anchor concentration:** if all internal anchors are generic ("read more"), you are wasting relevance signals.

---

## Site-type adaptations

The dominant failure mode shifts with what the site is.

- **Blog / content site:** cannibalization from overlapping posts, thin-tag index bloat, striking-distance long-tail, freshness decay. Growth = consolidate + strengthen the best, not endless new posts (on low authority).
- **Ecommerce:** faceted-navigation parameter explosion (the #1 cause of index bloat), thin product pages, category-vs-product cannibalization, out-of-stock/discontinued 404s, seasonality. Canonicals and parameter handling dominate.
- **SaaS / marketing site:** small page count, so every page matters; branded vs non-branded split is diagnostic; feature/comparison/alternative pages are the striking-distance goldmine; docs subdomain interplay.
- **Docs / knowledge base:** huge page count, internal-search-style queries, version duplication (v1/v2 pages competing), "answer" intent where featured snippets matter most.
- **Local business:** geo distribution is everything; Maps/local-pack impressions; branded + "near me" queries; NAP consistency and local schema.
- **Directory / aggregator / marketplace:** thin auto-generated pages Google routinely refuses to index (expect large "crawled not indexed"); the fight is proving unique value per page or accepting partial indexing and concentrating authority on the pages that matter.

## Thresholds used by the parser (tune per site)

- Striking distance: position 11-20, impressions ≥ `--min-impressions` (default 5).
- CTR underperformer: position ≤ 10 and actual CTR < 40% of the expected-for-position value.
- Zero-click: clicks == 0 and impressions ≥ max(min-impressions, 10).
- Expected CTR curve is a blended industry average (pos 1 ≈ 27%, pos 10 ≈ 2%, page 2 ≈ 1%). It is a reference for spotting outliers, not ground truth; real CTR varies wildly by query type and SERP layout.

# Competitor analysis (dependency-free)

GSC shows what a site ranks for. It cannot show who is beating it, or what demand it is missing entirely. This lens fills that gap using only `WebSearch` and `WebFetch` (no paid API, no new dependency). If the user has an Ahrefs/Semrush/Moz export, fold it in for real volume and backlink numbers, but the lens works without one.

Run it when: content gaps or striking-distance dominate the findings, the user asks about competitors, or a target query cluster is worth a serious push.

## Method

### 1. Pick the battleground queries
From the analysis, take the striking-distance and target-content-gap queries that actually matter for the site's audience (not the vanity clusters). Cap it: 5-15 queries is enough to see the pattern.

### 2. See who ranks
For each query, `WebSearch` it and record the top 5-10 organic results (skip ads and the site's own pages). Tally domains across queries. The domains that recur are the real competitors for this space, which is more accurate than whoever the user assumes.

Note SERP features too: who owns the featured snippet, what People-Also-Ask questions appear (these are content-gap gold), whether a pack/carousel/forum result (Reddit, Stack Overflow) dominates. If forums own the SERP, the winning move may be a genuinely better resource or participating there, not another blog post.

### 3. Read the winners
For the top 2-4 competitor pages per key query, `WebFetch` the page and compare against the site's competing (or missing) page on:
- **Depth and coverage**, sub-topics they answer that the site does not.
- **Structure**, headings, tables, direct answers up top, scannability.
- **Freshness**, visible last-updated date; stale winners are beatable.
- **Format match**, does the query want a listicle, a how-to, a definition, a comparison, a tool? Ranking pages reveal the intent.
- **Schema / rich results**, what they render in the SERP (rating stars, FAQ, breadcrumbs).
- **Authority proxy**, domain seniority, how referenced they are. Without a backlink tool this is a judgment call; say so.

### 4. Build the gap matrix
A simple table: rows = sub-topics/angles, columns = the site + top competitors, cells = covered / thin / missing. The site's "missing" rows that competitors all cover are the content plan. The rows everyone is thin on are the differentiation opportunity.

### 5. Output competitive moves
Fold into the action plan as its own tier:
- **Beat on depth:** specific sub-topics to add to an existing striking-distance page to overtake a shallow winner.
- **Beat on freshness:** pages where the winner is stale and a refresh + re-earn-links wins.
- **Beat on format:** where the site's format mismatches intent (a prose post where the SERP wants a comparison table or tool).
- **New pages:** only for gaps that are on-audience and winnable given the site's authority. Do not recommend targeting a query cluster owned by DR-90 sites on a new site; name that wall when it exists.

## Honest limits

- `WebSearch` results are personalized/regional and a moment in time, not a ranking tool's aggregate. Treat them as a strong sample, not a rank tracker.
- No backlink or traffic numbers without a paid export. State authority comparisons as estimates.
- "Winnable" depends on the site's own authority (from the Links export and the indexing diagnosis). A gap you cannot rank for yet is not an opportunity, it is a someday. Sequence competitive content behind the authority work if the site is index-starved.

## Optional: paid-tool CSV inputs

If the user provides them, these upgrade the lens from qualitative to quantitative:
- **Keyword export** (volume, difficulty) → prioritize gaps by real demand, not just GSC impressions.
- **Backlink export** (referring domains, competitor link profiles) → quantify the authority gap and surface link targets (who links to competitors but not the site).
- **Site audit export** → competitor technical posture.

Read them as extra CSVs alongside the GSC findings; the volume/difficulty columns slot straight into prioritization.

# GSC export guide

Exact steps to pull the data this skill needs. Google Search Console UI wording drifts; match on intent if a label moved.

## Prefer the API for Performance

`scripts/fetch_gsc.py` pulls the whole Performance half over the API, so export 1 below is only needed when the API is not set up for a property. It is not just less clicking, it is strictly more data: the UI export caps its query and page tables near 1000 rows while the API returns 25,000 per request, it writes a matching previous window so trend analysis always runs, and it returns **page+query pairs**, a shape no export has ever contained and the only way to prove true cannibalization.

```bash
python3 scripts/fetch_gsc.py --property sc-domain:example.com --days 90
```

Output is a directory of CSVs in Google's own export shape, so `parse_gsc.py` reads it exactly like a hand-exported zip and everything below stays a working fallback.

**One-time setup per Google account:**
1. `console.cloud.google.com` → pick or create a project.
2. APIs & Services → Library → **Google Search Console API** → Enable. Skipping this returns a 403 from the query endpoint whose wording sounds like a permissions problem rather than a disabled API.
3. APIs & Services → Credentials → Create credentials → **Service account**. No project IAM roles are needed; property access is granted in Search Console, not in IAM.
4. Open it → **Keys** → Add key → Create new key → **JSON**. Save it outside any webroot at `~/.config/gsc/<property-slug>.json`, mode 600. The slug is the property with `:`, `/` and `.` turned into `-`, so `sc-domain:example.com` becomes `example-com`.

**Per property:** Search Console → Settings → Users and permissions → Add user → the service account's `...iam.gserviceaccount.com` address. One service account can serve every property you own.

**What the API cannot give you, ever:** the Index Coverage report (exports 2 and 3 below) and the Links report. There is no endpoint for either. The URL Inspection API is not a substitute for coverage: it only answers for URLs you hand it, so it can confirm a sitemap but can never reveal index bloat from URLs Google holds that you never submitted, which is half of what that report is for.

## The three core exports

Open [search.google.com/search-console](https://search.google.com/search-console) and select the property. With the API set up, only 2 and 3 still need doing by hand.

### 1. Performance (Search results)
1. Left nav → **Performance** → **Search results**.
2. Top of page: click the **Date** filter. Choose **Last 3 months** (or longer; 12 months is better if the site is older). To get trend data in one shot, use the **Compare** tab → "Compare last 3 months to previous period".
3. Make sure the four metric toggles are all on: **Total clicks, Total impressions, Average CTR, Average position**.
4. Click **Export** (top right) → **Download CSV** (a zip).

This zip contains: `Queries.csv`, `Pages.csv`, `Countries.csv`, `Devices.csv`, `Dates.csv` (or `Chart.csv`), `Filters.csv`, `Search appearance.csv`. Note: the Queries and Pages files are a top-N sample (~1000 rows), not every query, so query-level totals are a sample, not the site total. The Dates/Chart file carries the true totals.

### 2. Indexing summary (Pages)
1. Left nav → **Indexing** → **Pages**.
2. Click **Export** on the summary view.

Gives the indexed vs not-indexed counts over time (`Chart.csv`) and the reason breakdown (`Critical issues.csv` / `Non-critical issues.csv`, columns `Reason,Source,Validation,Pages`).

### 3. Indexing drilldown (the one people skip)
1. Still in **Indexing → Pages**, scroll to "Why pages aren't indexed".
2. Click the largest reason (usually **Crawled - currently not indexed**, or **Discovered - currently not indexed**, or **Duplicate without user-selected canonical**).
3. On the detail page, click **Export**. This gives `Table.csv` (`URL,Last crawled`), the actual list of affected URLs.
4. Repeat for any other reason with a meaningful count.

The drilldown is what turns "44 pages aren't indexed" into "these specific 44 URLs, and here's the pattern." Without it the analysis is guesswork.

## Getting trend data (two date ranges)

With `fetch_gsc.py` this is automatic: it writes `current/` and `previous/` windows of equal length unless you pass `--no-compare`. The rest of this section is the manual route.

A single export is a snapshot: it cannot show "position improved but CTR dropped" or "this query is decaying." To diff:
- **Easiest:** in Performance, use the **Compare** tab (previous period or year-over-year), then Export. The CSVs then carry both periods.
- **Manual:** export "Last 3 months" today, and separately export the prior 3 months (custom range). Hand both to the skill; it will diff them.

## Optional exports that sharpen the analysis

| Report | Where | Adds |
|--------|-------|------|
| **Links** | Left nav → Links → Export | Top linking sites, top linked pages, internal links. Reveals authority and orphan pages. |
| **Sitemaps** | Indexing → Sitemaps | Submitted vs discovered counts; sitemap errors. |
| **Core Web Vitals** | Experience → Core Web Vitals | Slow-URL groups (LCP/CLS/INP). CWV code fixes belong to a performance or technical-SEO skill, not here. |
| **Enhancements / Structured data** | Experience/Enhancements | Which rich-result types are eligible and erroring. |
| **Manual actions / Security** | Security & Manual Actions | Penalties or hacks. Always check; a manual action explains everything at once. |

## Alternative and supplementary data sources

- **Bing Webmaster Tools** exports in a similar shape; the parser reads generic Clicks/Impressions/CTR/Position CSVs, so Bing "Search Performance" exports work too.
- **Ahrefs / Semrush / Moz** CSV exports (keyword volume, difficulty, backlinks) are not GSC but plug into the competitor lens as extra inputs for real search-volume and referring-domain numbers. GSC has no volume or difficulty data, so if the user has these, ask for them.
- **GA4** organic-landing-page export pairs conversions with the GSC clicks, answering "traffic but no conversions." GSC alone cannot see conversions.

## What GSC data cannot tell you (state these up front)

- **Search volume or keyword difficulty**, GSC shows your impressions, not total market demand. Use a keyword tool for that.
- **Why visitors don't convert**, clicks in, but not what happened after. Needs analytics.
- **Query × page pairing at scale**, the standard export gives queries and pages separately, not which query drove which page (unless you export a page-filtered Performance view). True one-query-many-pages cannibalization needs that filtered export or the API.
- **Causation**, correlation only. A change and a metric move in the same week is a hypothesis, not proof.

#!/usr/bin/env python3
"""Pull Search Console Performance data straight from the API, as CSVs.

Replaces the click-through-and-download-three-zips intake for the Performance
half of a-seo-gsc. Writes the same CSV shapes Google's own export produces, so
parse_gsc.py reads a fetched directory and a hand-exported zip identically and
the manual route stays a working fallback.

What this CANNOT fetch, because no API serves it:
  - the Index Coverage report (indexed vs not, reason breakdown, the
    "Crawled - currently not indexed" URL table). The URL Inspection API only
    answers for URLs you already hand it, so it can confirm a sitemap but never
    reveal bloat from URLs Google holds that you never submitted. Still a
    manual export.
  - the Links report. No API at all.

What it fetches that no export can give you:
  - page+query PAIRS, which is the only way to prove true cannibalization.
    Google's UI export lists queries and pages in separate tables, so the
    question "which of my pages compete for this query" has never been
    answerable from an export.
  - a second, immediately preceding window of equal length, so decay analysis
    works without anyone remembering to tick Compare in the UI.

Usage:
    python3 fetch_gsc.py --property sc-domain:example.com
    python3 fetch_gsc.py --property https://example.com/ --days 180 --out ./gsc
    python3 fetch_gsc.py --property sc-domain:example.com --no-compare

Auth is a Google service account with the Search Console API enabled, added as
a user on the property. Key file resolution, in order: --key, then
$GSC_SERVICE_ACCOUNT_JSON, then ~/.config/gsc/<property-slug>.json.

Standard library only, like parse_gsc.py. RS256 is signed by shelling out to
openssl rather than pulling in google-auth or cryptography, so this runs on any
machine without a pip install. The private key is passed to openssl through a
pipe file descriptor and never written to disk.
"""

from __future__ import annotations

import argparse
import csv
import datetime as dt
import io
import json
import os
import subprocess
import sys
import urllib.error
import urllib.parse
import urllib.request

API_BASE = "https://www.googleapis.com/webmasters/v3/sites/"
TOKEN_URI = "https://oauth2.googleapis.com/token"
SCOPE = "https://www.googleapis.com/auth/webmasters.readonly"

# Google's per-request cap. Paginated past with startRow.
ROW_LIMIT = 25000
# Runaway guard, not a real limit: 40 pages is a million rows.
MAX_PAGES = 40

# Search Console finishes processing a day about two days late, and a range
# ending today reads as a slump that is really just missing data.
LAG_DAYS = 3

# Each output CSV, as (filename, dimensions, first column header).
#
# Everything except dates.csv is aggregated over the whole range, one row per
# dimension value, because that is the shape Google's export has and the shape
# parse_gsc.py analyses. Asking for ['date', 'query'] instead would return one
# row per query PER DAY, and every query would then appear ninety times in the
# striking-distance table.
DATASETS = [
    ("dates.csv", ["date"], "Date"),
    ("queries.csv", ["query"], "Query"),
    ("pages.csv", ["page"], "Page"),
    ("countries.csv", ["country"], "Country"),
    ("devices.csv", ["device"], "Device"),
    ("pairs.csv", ["page", "query"], "Page"),
]


def eprint(*a: object) -> None:
    print(*a, file=sys.stderr)


# ---------------------------------------------------------------- auth


def load_key(path: str) -> dict:
    with open(path, encoding="utf-8") as f:
        key = json.load(f)
    for field in ("client_email", "private_key"):
        if not key.get(field):
            raise SystemExit(
                f"{path} has no {field}. Point --key at the JSON Google issued, unedited."
            )
    return key


def b64url(raw: bytes) -> bytes:
    import base64

    return base64.urlsafe_b64encode(raw).rstrip(b"=")


def sign_rs256(payload: bytes, private_key_pem: str) -> bytes:
    """Sign with openssl, handing it the key on a pipe rather than a temp file.

    A service-account private key written to /tmp, even at mode 600, outlives
    the process if anything crashes between write and unlink. The pipe dies
    with the process. 64KB is the default pipe buffer and a PEM is under 2KB,
    so writing the whole key before the child reads cannot deadlock.
    """
    read_fd, write_fd = os.pipe()
    try:
        os.write(write_fd, private_key_pem.encode())
    finally:
        os.close(write_fd)
    try:
        proc = subprocess.run(
            ["openssl", "dgst", "-sha256", "-sign", f"/dev/fd/{read_fd}"],
            input=payload,
            capture_output=True,
            pass_fds=(read_fd,),
        )
    except FileNotFoundError:
        raise SystemExit("openssl is not on PATH; it is needed to sign the assertion.")
    finally:
        os.close(read_fd)
    if proc.returncode != 0:
        raise SystemExit(
            "openssl rejected the private key while signing: "
            + proc.stderr.decode("utf-8", "replace")[:300]
        )
    return proc.stdout


def access_token(key: dict) -> str:
    now = int(dt.datetime.now(dt.timezone.utc).timestamp())
    header = b64url(json.dumps({"alg": "RS256", "typ": "JWT"}).encode())
    claims = b64url(
        json.dumps(
            {
                "iss": key["client_email"],
                "scope": SCOPE,
                "aud": key.get("token_uri", TOKEN_URI),
                "iat": now,
                "exp": now + 3600,
            }
        ).encode()
    )
    signing_input = header + b"." + claims
    assertion = signing_input + b"." + b64url(sign_rs256(signing_input, key["private_key"]))

    body = urllib.parse.urlencode(
        {
            "grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
            "assertion": assertion.decode(),
        }
    ).encode()
    req = urllib.request.Request(
        key.get("token_uri", TOKEN_URI),
        data=body,
        headers={"Content-Type": "application/x-www-form-urlencoded"},
    )
    try:
        with urllib.request.urlopen(req, timeout=30) as resp:
            token = json.load(resp).get("access_token")
    except urllib.error.HTTPError as e:
        # The failure body can echo part of the assertion back, so truncate it.
        raise SystemExit(
            f"Token exchange failed (HTTP {e.code}): "
            + e.read().decode("utf-8", "replace")[:500]
        )
    if not token:
        raise SystemExit("Google returned no access_token for the service account assertion.")
    return token


# ---------------------------------------------------------------- fetch


def query_api(prop: str, token: str, dimensions: list[str], start: str, end: str,
              search_type: str) -> list[dict]:
    """One dataset, paginated to exhaustion."""
    url = API_BASE + urllib.parse.quote(prop, safe="") + "/searchAnalytics/query"
    rows: list[dict] = []
    for page in range(MAX_PAGES):
        payload = {
            "startDate": start,
            "endDate": end,
            "dimensions": dimensions,
            "rowLimit": ROW_LIMIT,
            "startRow": page * ROW_LIMIT,
            "type": search_type,
        }
        req = urllib.request.Request(
            url,
            data=json.dumps(payload).encode(),
            headers={
                "Authorization": f"Bearer {token}",
                "Content-Type": "application/json",
            },
        )
        try:
            with urllib.request.urlopen(req, timeout=90) as resp:
                batch = json.load(resp).get("rows", [])
        except urllib.error.HTTPError as e:
            detail = e.read().decode("utf-8", "replace")[:600]
            raise SystemExit(
                f"Search Console returned HTTP {e.code} for {'+'.join(dimensions)} "
                f"{start}..{end}:\n{detail}"
            )
        rows.extend(batch)
        if len(batch) < ROW_LIMIT:
            break
    return rows


def write_csv(path: str, dimensions: list[str], first_header: str, rows: list[dict]) -> int:
    """Write one dataset in Google's own export shape.

    CTR is written as a PERCENTAGE NUMBER, not the 0-1 fraction the API
    returns, because a real export writes '1.92%' and parse_gsc.py divides the
    CTR column by 100. Getting this wrong understates every CTR by 100x and the
    CTR-underperformer analysis then flags the entire site.
    """
    headers = [first_header]
    if len(dimensions) > 1:
        headers.append("Query" if dimensions[1] == "query" else dimensions[1].title())
    headers += ["Clicks", "Impressions", "CTR", "Position"]

    with open(path, "w", encoding="utf-8", newline="") as f:
        w = csv.writer(f, lineterminator="\n")
        w.writerow(headers)
        for r in rows:
            keys = r.get("keys", [])
            w.writerow(
                keys
                + [
                    int(r.get("clicks", 0)),
                    int(r.get("impressions", 0)),
                    f"{r.get('ctr', 0.0) * 100:.2f}%",
                    f"{r.get('position', 0.0):.2f}",
                ]
            )
    return len(rows)


def write_filters(path: str, search_type: str, start: str, end: str) -> None:
    """The sibling Filters.csv a real export carries.

    parse_gsc.py reads it to tell a Web export from an Image one before merging
    anything. Without it a fetched directory behaves differently from an
    exported one the moment the two are analysed together.
    """
    with open(path, "w", encoding="utf-8", newline="") as f:
        w = csv.writer(f, lineterminator="\n")
        w.writerow(["Filter", "Value"])
        w.writerow(["Search type", search_type.title()])
        w.writerow(["Date range", f"{start} to {end}"])


def fetch_window(prop: str, token: str, start: str, end: str, outdir: str,
                 search_type: str) -> dict[str, int]:
    os.makedirs(outdir, exist_ok=True)
    counts: dict[str, int] = {}
    for filename, dimensions, first_header in DATASETS:
        rows = query_api(prop, token, dimensions, start, end, search_type)
        counts[filename] = write_csv(
            os.path.join(outdir, filename), dimensions, first_header, rows
        )
    write_filters(os.path.join(outdir, "Filters.csv"), search_type, start, end)
    return counts


# ---------------------------------------------------------------- cli


def property_slug(prop: str) -> str:
    return (
        prop.replace("sc-domain:", "")
        .replace("https://", "")
        .replace("http://", "")
        .strip("/")
        .replace("/", "-")
        .replace(":", "-")
        .replace(".", "-")
    )


def resolve_key_path(explicit: str | None, prop: str) -> str:
    for candidate in (
        explicit,
        os.environ.get("GSC_SERVICE_ACCOUNT_JSON"),
        os.path.expanduser(f"~/.config/gsc/{property_slug(prop)}.json"),
    ):
        if candidate and os.path.isfile(candidate):
            return candidate
    raise SystemExit(
        "No service-account key found. Pass --key PATH, set "
        "GSC_SERVICE_ACCOUNT_JSON, or put the JSON at "
        f"~/.config/gsc/{property_slug(prop)}.json"
    )


def main() -> int:
    ap = argparse.ArgumentParser(description="Fetch Search Console Performance data as CSVs.")
    ap.add_argument("--property", required=True,
                    help="Property exactly as Search Console names it, e.g. sc-domain:example.com")
    ap.add_argument("--days", type=int, default=90, help="Window length in days (default 90)")
    ap.add_argument("--end", default="",
                    help=f"Last day, YYYY-MM-DD (default: today minus {LAG_DAYS} days)")
    ap.add_argument("--key", default=None, help="Service-account JSON path")
    ap.add_argument("--out", default="", help="Output directory (default: ./gsc-<slug>-<end>)")
    ap.add_argument("--search-type", default="web",
                    choices=["web", "image", "video", "news", "discover", "googleNews"])
    ap.add_argument("--no-compare", action="store_true",
                    help="Skip the preceding window; decay analysis then cannot run")
    args = ap.parse_args()

    if args.days < 1:
        raise SystemExit("--days must be at least 1")

    end = (
        dt.date.fromisoformat(args.end)
        if args.end
        else dt.date.today() - dt.timedelta(days=LAG_DAYS)
    )
    start = end - dt.timedelta(days=args.days - 1)
    prev_end = start - dt.timedelta(days=1)
    prev_start = prev_end - dt.timedelta(days=args.days - 1)

    outroot = args.out or f"./gsc-{property_slug(args.property)}-{end.isoformat()}"
    key = load_key(resolve_key_path(args.key, args.property))
    token = access_token(key)

    eprint(f"Property: {args.property}  ({key['client_email']})")

    windows = [("current", start, end)]
    if not args.no_compare:
        windows.append(("previous", prev_start, prev_end))

    for label, w_start, w_end in windows:
        outdir = os.path.join(outroot, label)
        counts = fetch_window(
            args.property, token, w_start.isoformat(), w_end.isoformat(),
            outdir, args.search_type,
        )
        summary = ", ".join(f"{n} {f.replace('.csv', '')}" for f, n in counts.items() if n)
        eprint(f"{label:8} {w_start} .. {w_end}  ->  {outdir}")
        eprint(f"         {summary or 'no rows returned'}")

    if all(
        os.path.getsize(os.path.join(outroot, w[0], "dates.csv")) < 40 for w in windows
    ):
        eprint("\nEvery dataset came back empty. Check the property name and that the "
               "service account is a user on it.")
        return 1

    print(outroot)
    eprint(f"\nNext: python3 parse_gsc.py {outroot}/current --brand \"...\"")
    if not args.no_compare:
        eprint(f"      and {outroot}/previous for the decay diff.")
    eprint("Coverage is not in here. Export Indexing -> Pages by hand; no API serves it.")
    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
parse_gsc.py: normalize Google Search Console exports into a findings report.

Deterministic pre-processor for the a-seo-gsc skill. Point it at one or more
GSC export zips (or a directory / loose CSVs) and it auto-detects each report
type, normalizes the numbers, and emits a structured set of findings: striking-
distance queries, zero-click high-impression queries, CTR outliers vs an
expected curve, parameter-duplicate cannibalization, coverage buckets, and a
categorized list of not-indexed URLs.

The model reads this output and layers strategy on top. Keeping the arithmetic
here makes the analysis fast, cheap, and repeatable across sites.

Pure standard library. No third-party deps. Cross-platform.

Usage:
    python3 parse_gsc.py <path...> [options]

    <path>              One or more GSC export .zip files, a directory
                        containing them (or extracted CSVs), or loose .csv files.

Options:
    --brand "a,b,c"     Comma-separated brand terms to split branded vs
                        non-branded queries (case-insensitive substring match).
    --min-impressions N Minimum impressions for a query to surface in findings
                        (default: 5).
    --json              Emit JSON instead of the markdown report.
    --out FILE          Write output to FILE instead of stdout.

Exit codes: 0 on success, 2 on "no recognizable GSC data found".
"""

from __future__ import annotations

import argparse
import csv
import io
import json
import os
import sys
import tempfile
import zipfile

# Rough organic CTR-by-position reference (desktop+mobile blended, 2023-2024
# industry averages). Used only to flag snippet underperformance, not as truth.
EXPECTED_CTR = {
    1: 0.275, 2: 0.150, 3: 0.100, 4: 0.070, 5: 0.055,
    6: 0.043, 7: 0.034, 8: 0.028, 9: 0.024, 10: 0.021,
}
PAGE2_CTR = 0.010  # positions 11-20, generic


def _num(s: str) -> float:
    """Parse a GSC numeric cell: '1.92%', '11.85', '1,234', '40%', '' -> float."""
    if s is None:
        return 0.0
    s = s.strip().replace("%", "")
    if not s:
        return 0.0
    # Thousands separators vs decimal comma: GSC uses '.' decimals, but be lenient.
    if s.count(",") == 1 and "." not in s:
        s = s.replace(",", ".")
    else:
        s = s.replace(",", "")
    try:
        return float(s)
    except ValueError:
        return 0.0


def _read_csv(text: str) -> tuple[list[str], list[list[str]]]:
    """Return (header, rows). Handles quoted multi-line cells (GSC query cells)."""
    reader = csv.reader(io.StringIO(text))
    rows = [r for r in reader if r]
    if not rows:
        return [], []
    return rows[0], rows[1:]


def _find_col(header: list[str], *names: str) -> int:
    low = [h.strip().lower() for h in header]
    for name in names:
        n = name.lower()
        for i, h in enumerate(low):
            if h == n or h.startswith(n) or n in h:
                return i
    return -1


def _collect_csvs(paths: list[str], tmpdir: str) -> tuple[dict[str, str], dict[str, str]]:
    """Return ({logical_name: csv_text}, {logical_name: source_export}) from zips/dirs/loose csvs.

    The second map records which export each CSV came from (the zip, or the
    directory holding it). Performance exports carry their settings in a
    sibling Filters.csv, so grouping by source is what lets us tell a Web
    export from an Image one before anything gets merged.
    """
    out: dict[str, str] = {}
    src: dict[str, str] = {}

    def add(name: str, text: str, source: str) -> None:
        base = os.path.basename(name)
        key = base
        n = 1
        while key in out:  # keep duplicates from multiple exports distinct
            n += 1
            key = f"{base}#{n}"
        out[key] = text
        src[key] = source

    def handle_file(path: str) -> None:
        if not os.path.exists(path):
            print(f"warning: path not found, skipping: {path}", file=sys.stderr)
            return
        low = path.lower()
        if low.endswith(".zip"):
            try:
                with zipfile.ZipFile(path) as z:
                    for info in z.namelist():
                        if info.lower().endswith(".csv"):
                            with z.open(info) as f:
                                add(info, f.read().decode("utf-8-sig", "replace"), path)
            except zipfile.BadZipFile:
                print(f"warning: not a valid zip, skipping: {path}", file=sys.stderr)
        elif low.endswith(".csv"):
            with open(path, encoding="utf-8-sig", errors="replace") as f:
                add(path, f.read(), os.path.dirname(os.path.abspath(path)))

    for p in paths:
        if os.path.isdir(p):
            for root, _dirs, files in os.walk(p):
                for fn in files:
                    handle_file(os.path.join(root, fn))
        else:
            handle_file(p)
    return out, src


# Performance-report members. Coverage and drilldown CSVs are not in here, so
# they always survive the search-type filter below.
_PERF_FILES = {
    "queries.csv", "pages.csv", "countries.csv", "devices.csv",
    "dates.csv", "chart.csv", "filters.csv", "search appearance.csv",
    "pairs.csv",
}


def _basename_of(key: str) -> str:
    return key.split("#", 1)[0].lower()


def _search_types(csvs: dict[str, str], src: dict[str, str]) -> dict[str, str]:
    """Map source export -> declared search type, read from its Filters.csv."""
    types: dict[str, str] = {}
    for key, text in csvs.items():
        if _basename_of(key) != "filters.csv":
            continue
        _header, rows = _read_csv(text)
        for row in rows:
            if len(row) >= 2 and row[0].strip().lower() == "search type":
                types[src.get(key, "")] = row[1].strip().lower()
    return types


def _split_by_search_type(
    csvs: dict[str, str], src: dict[str, str], prefer: str
) -> tuple[dict[str, str], str, dict[str, int]]:
    """Keep one search type's performance data; drop the rest.

    Web and Image exports describe the same URLs from different indexes.
    Merging them double-counts every page and surfaces the duplicates as
    parameter cannibalization, which is an artifact rather than a finding.
    """
    types = _search_types(csvs, src)
    present = {t for t in types.values() if t}
    if len(present) <= 1:
        return csvs, (next(iter(present)) if present else ""), {}

    if prefer and prefer.lower() in present:
        keep = prefer.lower()
    elif "web" in present:
        keep = "web"
    else:
        keep = sorted(present)[0]

    kept: dict[str, str] = {}
    dropped: dict[str, int] = {}
    for key, text in csvs.items():
        t = types.get(src.get(key, ""))
        if t and t != keep and _basename_of(key) in _PERF_FILES:
            dropped[t] = dropped.get(t, 0) + 1
            continue
        kept[key] = text
    return kept, keep, dropped


def _classify_url(url: str) -> str:
    """Generic path-pattern bucket for a not-indexed URL (site-agnostic)."""
    u = url.lower()
    path = u.split("://", 1)[-1]
    path = "/" + path.split("/", 1)[1] if "/" in path else "/"
    if "?" in u:
        return "parameter-url"
    if any(seg in path for seg in ("/feed", "/rss", "/atom")) or path.endswith(".xml"):
        return "feed"
    if any(seg in path for seg in ("/visit", "/go/", "/out/", "/redirect", "/goto")):
        return "redirect-endpoint"
    if "/page/" in path or "/p/" in path.rstrip("/").rsplit("/", 1)[0] + "/":
        return "pagination"
    if any(seg in path for seg in ("/tag/", "/tags/", "/label/", "/topic/", "/topics/")):
        return "taxonomy-tag"
    if any(seg in path for seg in ("/category/", "/categories/", "/cat/", "/c/")):
        return "taxonomy-category"
    if any(seg in path for seg in ("/product/", "/products/", "/shop/", "/item/")):
        return "product"
    return "content-page"


class Report:
    def __init__(self) -> None:
        self.queries: list[dict] = []
        self.pages: list[dict] = []
        self.countries: list[dict] = []
        self.devices: list[dict] = []
        self.dates: list[dict] = []          # perf time series
        self.coverage_issues: list[dict] = []
        self.coverage_series: list[dict] = []  # indexed vs not-indexed over time
        self.not_indexed: list[dict] = []     # drilldown table
        self.pairs: list[dict] = []           # page+query rows (API only)
        self.drill_issue: str = ""


def _ingest(csvs: dict[str, str]) -> Report:
    r = Report()
    for name, text in csvs.items():
        header, rows = _read_csv(text)
        if not header:
            continue
        h = [c.strip().lower() for c in header]

        # --- Coverage drilldown table: URL,Last crawled ---
        if "url" in h and any("crawl" in c for c in h):
            ci = _find_col(header, "url")
            cc = _find_col(header, "last crawled", "crawled")
            for row in rows:
                if len(row) > ci and row[ci].strip():
                    r.not_indexed.append({
                        "url": row[ci].strip(),
                        "last_crawled": row[cc].strip() if cc >= 0 and len(row) > cc else "",
                    })
            continue

        # --- Coverage issue tables: Reason,Source,Validation,Pages ---
        if "reason" in h and "pages" in h:
            cr = _find_col(header, "reason")
            cp = _find_col(header, "pages")
            for row in rows:
                if len(row) > cr and row[cr].strip():
                    r.coverage_issues.append({
                        "reason": row[cr].strip(),
                        "pages": int(_num(row[cp])) if cp >= 0 and len(row) > cp else 0,
                    })
            continue

        # --- Coverage chart: Date,Not indexed,Indexed,Impressions ---
        if "date" in h and any("not indexed" in c for c in h) and any(c == "indexed" or c.endswith("indexed") for c in h):
            cn = _find_col(header, "not indexed")
            cidx = -1
            for i, c in enumerate(h):
                if c == "indexed" or (c.endswith("indexed") and "not" not in c):
                    cidx = i
                    break
            for row in rows:
                if len(row) <= cn or row[cn].strip() == "":
                    continue
                r.coverage_series.append({
                    "date": row[0].strip(),
                    "not_indexed": int(_num(row[cn])),
                    "indexed": int(_num(row[cidx])) if cidx >= 0 and len(row) > cidx else 0,
                })
            continue

        # --- Page+query pairs (API only; no GSC export has this shape) ---
        #
        # Checked BEFORE the generic performance branch on purpose. That branch
        # buckets on column 0, and a pairs file leads with Page, so it would be
        # merged into r.pages and double every page's clicks and impressions.
        if ("clicks" in h and "impressions" in h
                and any("quer" in c for c in h)
                and any(c.startswith("page") or c.startswith("url") for c in h)):
            cq = _find_col(header, "query")
            cu = _find_col(header, "page", "url")
            cc = _find_col(header, "clicks")
            cimp = _find_col(header, "impressions")
            cpos = _find_col(header, "position")
            for row in rows:
                if len(row) <= max(cq, cu) or not row[cq].strip() or not row[cu].strip():
                    continue
                r.pairs.append({
                    "query": row[cq].strip(),
                    "url": row[cu].strip(),
                    "clicks": int(_num(row[cc])) if len(row) > cc else 0,
                    "impressions": int(_num(row[cimp])) if len(row) > cimp else 0,
                    "position": _num(row[cpos]) if cpos >= 0 and len(row) > cpos else 0.0,
                })
            continue

        # --- Performance breakdowns: have Clicks + Impressions + Position ---
        if "clicks" in h and "impressions" in h and "position" in h:
            cc = _find_col(header, "clicks")
            cimp = _find_col(header, "impressions")
            cctr = _find_col(header, "ctr")
            cpos = _find_col(header, "position")
            dim = header[0].strip().lower()
            bucket = None
            label = "name"
            if "quer" in dim:
                bucket, label = r.queries, "query"
            elif "page" in dim or "url" in dim:
                bucket, label = r.pages, "url"
            elif "countr" in dim:
                bucket, label = r.countries, "country"
            elif "device" in dim:
                bucket, label = r.devices, "device"
            elif "date" in dim:
                bucket, label = r.dates, "date"
            else:
                continue
            for row in rows:
                if not row or not row[0].strip():
                    continue
                rec = {
                    label: row[0].strip(),
                    "clicks": int(_num(row[cc])) if len(row) > cc else 0,
                    "impressions": int(_num(row[cimp])) if len(row) > cimp else 0,
                    "ctr": _num(row[cctr]) / 100.0 if cctr >= 0 and len(row) > cctr else 0.0,
                    "position": _num(row[cpos]) if cpos >= 0 and len(row) > cpos else 0.0,
                }
                bucket.append(rec)
            continue
    return r


def _clean_url(url: str) -> str:
    return url.split("?", 1)[0].split("#", 1)[0]


def analyze(r: Report, brand: list[str], min_imp: int) -> dict:
    f: dict = {}

    # Baseline totals: prefer the date series (authoritative), else devices.
    if r.dates:
        tot_clicks = sum(d["clicks"] for d in r.dates)
        tot_imp = sum(d["impressions"] for d in r.dates)
    elif r.devices:
        tot_clicks = sum(d["clicks"] for d in r.devices)
        tot_imp = sum(d["impressions"] for d in r.devices)
    else:
        tot_clicks = sum(q["clicks"] for q in r.queries)
        tot_imp = sum(q["impressions"] for q in r.queries)
    wpos = (sum(q["position"] * q["impressions"] for q in r.queries) /
            sum(q["impressions"] for q in r.queries)) if r.queries else 0.0
    f["baseline"] = {
        "clicks": tot_clicks,
        "impressions": tot_imp,
        "ctr": (tot_clicks / tot_imp) if tot_imp else 0.0,
        "impression_weighted_position": round(wpos, 2),
        "queries_tracked": len(r.queries),
        "pages_tracked": len(r.pages),
    }

    # Branded vs non-branded
    if brand:
        bl = [b.lower() for b in brand if b.strip()]
        def is_brand(q: str) -> bool:
            ql = q.lower()
            return any(b in ql for b in bl)
        bq = [q for q in r.queries if is_brand(q["query"])]
        nbq = [q for q in r.queries if not is_brand(q["query"])]
        f["branded_split"] = {
            "branded": {"clicks": sum(q["clicks"] for q in bq), "impressions": sum(q["impressions"] for q in bq)},
            "non_branded": {"clicks": sum(q["clicks"] for q in nbq), "impressions": sum(q["impressions"] for q in nbq)},
        }

    # Striking distance: page-2 queries (pos 11-20) with real impressions.
    striking = sorted(
        [q for q in r.queries if 10.5 < q["position"] <= 20.5 and q["impressions"] >= min_imp],
        key=lambda q: -q["impressions"],
    )
    f["striking_distance"] = striking[:30]

    # Page-1 low CTR: pos <=10, impressions >= min, actual CTR far below expected.
    ctr_outliers = []
    for q in r.queries:
        if q["position"] <= 10.5 and q["impressions"] >= min_imp:
            exp = EXPECTED_CTR.get(round(q["position"]), PAGE2_CTR)
            if q["ctr"] < exp * 0.4:  # under 40% of expected = snippet/intent problem
                ctr_outliers.append({**q, "expected_ctr": round(exp, 3)})
    f["ctr_underperformers"] = sorted(ctr_outliers, key=lambda q: -q["impressions"])[:30]

    # Zero-click high-impression
    f["zero_click"] = sorted(
        [q for q in r.queries if q["clicks"] == 0 and q["impressions"] >= max(min_imp, 10)],
        key=lambda q: -q["impressions"],
    )[:30]

    # Parameter-duplicate cannibalization (same clean URL indexed under variants)
    groups: dict[str, list[dict]] = {}
    for p in r.pages:
        groups.setdefault(_clean_url(p["url"]), []).append(p)
    dup = []
    for clean, variants in groups.items():
        if len(variants) > 1:
            dup.append({
                "clean_url": clean,
                "variants": sorted(variants, key=lambda p: -p["impressions"]),
                "total_impressions": sum(v["impressions"] for v in variants),
            })
    f["param_cannibalization"] = sorted(dup, key=lambda d: -d["total_impressions"])

    # True query cannibalization: one query, several of your pages competing.
    #
    # Only possible from page+query pairs, which no GSC export contains. The
    # UI's own export lists queries and pages in separate tables, so this
    # question was unanswerable until the API path existed. Absent pairs, the
    # key is left out entirely rather than reported as "none found", because
    # nothing was looked at and saying otherwise would read as a clean bill.
    if r.pairs:
        by_query: dict[str, list[dict]] = {}
        for row in r.pairs:
            by_query.setdefault(row["query"], []).append(row)
        cannibal = []
        for query, hits in by_query.items():
            total = sum(x["impressions"] for x in hits)
            # Rank by impressions, but report position too: the actionable
            # signal is a mismatch, where the page Google shows most often is
            # not the page that ranks best for the query.
            ranked = sorted(hits, key=lambda x: -x["impressions"])
            rivals = [x for x in ranked if x["impressions"] >= min_imp]
            if len(rivals) < 2 or total <= 0:
                continue
            # A runner-up holding a token share is noise, not competition.
            if rivals[1]["impressions"] / total < 0.20:
                continue
            # Over rivals, NOT over hits. Picking the best position from every
            # page for the query lets a page below the impression floor win,
            # and the report then names a URL that does not appear in its own
            # evidence list while the listed numbers contradict the claim.
            best = min(rivals, key=lambda x: x["position"] if x["position"] else 999)
            cannibal.append({
                "query": query,
                "total_impressions": total,
                "rival_impressions": sum(x["impressions"] for x in rivals),
                "pages_seen": len(hits),
                "total_clicks": sum(x["clicks"] for x in hits),
                "pages": rivals,
                "most_shown": ranked[0]["url"],
                "best_ranked": best["url"],
                "split_mismatch": best["url"] != ranked[0]["url"],
            })
        f["query_cannibalization"] = sorted(
            cannibal, key=lambda d: -d["total_impressions"]
        )[:30]
        f["pairs_seen"] = len(r.pairs)

    # Concentration: what share of impressions does the top page hold?
    if r.pages:
        top = max(r.pages, key=lambda p: p["impressions"])
        f["concentration"] = {
            "top_page": top["url"],
            "top_page_impressions": top["impressions"],
            "share_of_total": round(top["impressions"] / tot_imp, 3) if tot_imp else 0.0,
        }

    # Coverage
    if r.coverage_issues:
        buckets = {}
        for c in r.coverage_issues:
            buckets[c["reason"]] = buckets.get(c["reason"], 0) + c["pages"]
        f["coverage_issues"] = buckets
    if r.coverage_series:
        first = r.coverage_series[0]
        last = r.coverage_series[-1]
        f["coverage_trend"] = {
            "first": first, "last": last,
            "indexed_delta": last["indexed"] - first["indexed"],
            "not_indexed_delta": last["not_indexed"] - first["not_indexed"],
            "diagnosis": (
                "index-starvation (more pages excluded than indexed)"
                if last["not_indexed"] > last["indexed"]
                else "index-bloat risk" if last["not_indexed"] > last["indexed"] * 0.5
                else "healthy"
            ),
        }

    # Not-indexed URL categorization
    if r.not_indexed:
        cats: dict[str, list[str]] = {}
        for u in r.not_indexed:
            cats.setdefault(_classify_url(u["url"]), []).append(u["url"])
        f["not_indexed"] = {
            "issue": r.drill_issue or "not indexed",
            "total": len(r.not_indexed),
            "by_category": {k: {"count": len(v), "sample": v[:12]} for k, v in
                            sorted(cats.items(), key=lambda kv: -len(kv[1]))},
        }
    return f


def _pct(x: float) -> str:
    return f"{x * 100:.2f}%"


def to_markdown(f: dict) -> str:
    o: list[str] = ["# GSC parsed findings", ""]
    b = f["baseline"]
    o += [
        "## Baseline",
        f"- Clicks: **{b['clicks']}**  |  Impressions: **{b['impressions']}**  |  CTR: **{_pct(b['ctr'])}**",
        f"- Impression-weighted avg position: **{b['impression_weighted_position']}**",
        f"- Queries tracked: {b['queries_tracked']}  |  Pages tracked: {b['pages_tracked']}",
        "",
    ]
    if "branded_split" in f:
        bs = f["branded_split"]
        o += [
            "## Branded vs non-branded",
            f"- Branded: {bs['branded']['clicks']} clicks / {bs['branded']['impressions']} impr",
            f"- Non-branded: {bs['non_branded']['clicks']} clicks / {bs['non_branded']['impressions']} impr",
            "",
        ]
    if f.get("concentration"):
        c = f["concentration"]
        o += ["## Concentration",
              f"- Top page holds **{_pct(c['share_of_total'])}** of all impressions: `{c['top_page']}` ({c['top_page_impressions']} impr)",
              ""]

    def qtable(title: str, rows: list[dict], extra: str = "") -> None:
        nonlocal o
        o.append(f"## {title}")
        if not rows:
            o.extend(["_none_", ""])
            return
        o.append(f"| query | clicks | impr | CTR | pos |{' expected CTR |' if extra=='exp' else ''}")
        o.append(f"|---|--:|--:|--:|--:|{'--:|' if extra=='exp' else ''}")
        for q in rows:
            line = f"| {q['query']} | {q['clicks']} | {q['impressions']} | {_pct(q['ctr'])} | {q['position']:.1f} |"
            if extra == "exp":
                line += f" {_pct(q.get('expected_ctr', 0))} |"
            o.append(line)
        o.append("")

    qtable("Striking distance (pos 11-20, quick wins)", f.get("striking_distance", []))
    qtable("Page-1 CTR underperformers (snippet/intent problem)", f.get("ctr_underperformers", []), "exp")
    qtable("Zero-click high-impression queries", f.get("zero_click", []))

    o.append("## Parameter-duplicate cannibalization")
    if f.get("param_cannibalization"):
        for d in f["param_cannibalization"]:
            o.append(f"- `{d['clean_url']}`: {len(d['variants'])} indexed variants, {d['total_impressions']} impr total")
            for v in d["variants"]:
                o.append(f"    - {v['impressions']} impr, pos {v['position']:.1f}: `{v['url']}`")
    else:
        o.append("_none detected_")
    o.append("")

    if "query_cannibalization" in f:
        o.append("## Query cannibalization (from page+query pairs)")
        o.append(f"_{f.get('pairs_seen', 0)} page+query rows analysed._")
        if f["query_cannibalization"]:
            for d in f["query_cannibalization"]:
                flag = "  **position/impression mismatch**" if d["split_mismatch"] else ""
                tail = ""
                if d.get("pages_seen", 0) > len(d["pages"]):
                    tail = (f", {d['pages_seen'] - len(d['pages'])} more below the "
                            f"impression floor")
                o.append(
                    f"- `{d['query']}`: {len(d['pages'])} pages competing on "
                    f"{d.get('rival_impressions', d['total_impressions'])} of "
                    f"{d['total_impressions']} impr, {d['total_clicks']} clicks{tail}{flag}"
                )
                for pg in d["pages"]:
                    o.append(
                        f"    - {pg['impressions']} impr, {pg['clicks']} clicks, "
                        f"pos {pg['position']:.1f}: `{pg['url']}`"
                    )
                if d["split_mismatch"]:
                    o.append(
                        f"    - Google shows `{d['most_shown']}` most, but "
                        f"`{d['best_ranked']}` ranks better."
                    )
        else:
            o.append("_none detected_")
        o.append("")

    if "coverage_issues" in f:
        o.append("## Coverage issues")
        for reason, n in sorted(f["coverage_issues"].items(), key=lambda kv: -kv[1]):
            o.append(f"- **{n}**: {reason}")
        o.append("")
    if "coverage_trend" in f:
        t = f["coverage_trend"]
        o += ["## Coverage trend",
              f"- {t['first']['date']}: {t['first']['indexed']} indexed / {t['first']['not_indexed']} not indexed",
              f"- {t['last']['date']}: {t['last']['indexed']} indexed / {t['last']['not_indexed']} not indexed",
              f"- Diagnosis: **{t['diagnosis']}**", ""]
    if "not_indexed" in f:
        ni = f["not_indexed"]
        o.append(f"## Not-indexed URLs ({ni['total']} total)")
        for cat, d in ni["by_category"].items():
            o.append(f"### {cat} ({d['count']})")
            for u in d["sample"]:
                o.append(f"- {u}")
            if d["count"] > len(d["sample"]):
                o.append(f"- … +{d['count'] - len(d['sample'])} more")
            o.append("")
    return "\n".join(o)


def main() -> int:
    ap = argparse.ArgumentParser(description="Normalize GSC exports into findings.")
    ap.add_argument("paths", nargs="+")
    ap.add_argument("--brand", default="")
    ap.add_argument("--min-impressions", type=int, default=5)
    ap.add_argument("--json", action="store_true")
    ap.add_argument("--out", default="")
    ap.add_argument(
        "--search-type",
        default="web",
        help="which search type to parse when the paths hold more than one "
             "(web, image, video, news). Default web.",
    )
    args = ap.parse_args()

    with tempfile.TemporaryDirectory() as tmp:
        csvs, sources = _collect_csvs(args.paths, tmp)
    if not csvs:
        print("No CSV files found in the given paths.", file=sys.stderr)
        return 2

    csvs, kept_type, dropped = _split_by_search_type(csvs, sources, args.search_type)
    if dropped:
        others = ", ".join(f"{t} ({n} files)" for t, n in sorted(dropped.items()))
        print(
            f"note: the given paths hold more than one search type. Parsed '{kept_type}' "
            f"and ignored {others}. Merging them would double-count every page. "
            f"Re-run with --search-type to read another one.",
            file=sys.stderr,
        )
    elif kept_type:
        print(f"note: search type '{kept_type}'.", file=sys.stderr)

    report = _ingest(csvs)
    if not (report.queries or report.pages or report.coverage_issues or report.not_indexed):
        print("No recognizable GSC performance or coverage data found.", file=sys.stderr)
        return 2

    brand = [b for b in args.brand.split(",") if b.strip()]
    findings = analyze(report, brand, args.min_impressions)
    out = json.dumps(findings, indent=2, ensure_ascii=False) if args.json else to_markdown(findings)

    if args.out:
        with open(args.out, "w", encoding="utf-8") as fh:
            fh.write(out + "\n")
        print(f"Wrote {args.out}")
    else:
        print(out)
    return 0


if __name__ == "__main__":
    sys.exit(main())