From Crawl to Contribution
A practical evidence model separating crawler access from retrieval, citation, referral and the value a source contributes to an AI answer.
Articles & Research
Practical insight for technical SEO specialists and web developers
Latest articles and research
AEO measurement
Crawler research
Freshness Signals vs Crawl Reality
First findings from 28 days of verified crawler requests, revealing sharp format differences and an average requested content age of 120 days.
AI search
The Ultimate Technical AI Search Optimisation Guide
A technical framework for AI search visibility, covering crawler access, rendering, canonicalisation, chunking, structured data, freshness, measurement, and governance.
Technical guidance
AI Ingestion & Visibility Technical Guidelines
A practical framework for crawler access, machine-readable architecture, structured data, bot logging and retrieval visibility.
Machine web
Markdown for Agents
What Cloudflare’s Markdown for Agents proposal reveals about low-noise formats, crawler access and machine-readable publishing.
Google's Guidelines
Google’s 2 MB Crawl Limit
What Google’s 2 MB crawl limit means for indexing, rendering, resource size and machine-readable publishing.
AI etiquette
When AI Hesitates to Crawl
An observed interaction in which ChatGPT initially declined to read Scrubnet feeds, illustrating how agent guardrails can affect user-directed fetching.
About the articles and research
This hub brings together practical articles for technical SEO specialists and web developers. Topics include technical SEO, web development, better working methods, AI and search crawling, and evidence-led observations from the Scrubnet Crawler Observatory.
For research based on the observatory, we document scope and limitations so crawler access is not confused with indexing, ranking, model use or AI citation.
Reports identify the observation period, included crawler requests and relevant exclusions. Where available, they also describe the resources that were eligible to be requested. Differences are reported as associations within Scrubnet’s environment, not as proof that a feed characteristic caused a crawler response.