Freshness Signals vs Crawl Reality
First findings from 28 days of verified crawler requests, revealing sharp format differences and an average requested content age of 120 days.
Crawler Observations
Descriptive findings, case studies and technical analysis from verified request data
Crawler research
AI search
The Ultimate Technical AI Search Optimisation Guide
A technical framework for AI search visibility, covering crawler access, rendering, canonicalisation, chunking, structured data, freshness, measurement, and governance.
Technical guidance
AI Ingestion & Visibility Technical Guidelines
A practical framework for crawler access, machine-readable architecture, structured data, bot logging and retrieval visibility.
Machine web
Markdown for Agents
What Cloudflare’s Markdown for Agents proposal reveals about low-noise formats, crawler access and machine-readable publishing.
Google's Guidelines
Google’s 2 MB Crawl Limit
What Google’s 2 MB crawl limit means for indexing, rendering, resource size and machine-readable publishing.
AI etiquette
When AI Hesitates to Crawl
An observed interaction in which ChatGPT initially declined to read Scrubnet feeds, illustrating how agent guardrails can affect user-directed fetching.
About the observations
This hub collects descriptive observations from Scrubnet’s public feeds and verified crawler logs, alongside broader technical SEO and GEO analysis. We document scope and limitations so crawler access is not confused with indexing, ranking, model use or AI citation.
Reports identify the observation period, included crawler requests and relevant exclusions. Where available, they also describe the resources that were eligible to be requested. Differences are reported as associations within Scrubnet’s environment, not as proof that a feed characteristic caused a crawler response.