The Ultimate Technical AI Search Optimisation Guide
A technical framework for AI search visibility, covering crawler access, rendering, canonicalisation, chunking, structured data, freshness, measurement, and governance.
Discoveries
Experiments, notes and insights from building the machine web
Latest articles
AI search
Machine web
Markdown for Agents
Cloudflare proposes a cleaner publishing format for AI systems. Why low noise, structured content reinforces Scrubnet’s vision of a dedicated web layer for crawlers and LLMs.
Google's Guidelines
Google’s 2 MB Crawl Limit
Google now limits crawling to 2 MB per resource. Learn what this means for indexing, rendering, and how Scrubnet publishes content designed for modern crawlers and AI agents.
AI etiquette
When AI Hesitates to Crawl
A live experiment where ChatGPT initially declined to read Scrubnet feed pages, then proceeded after explicit instruction. What this reveals about the machine web.
What is this page
This hub collects small discoveries that happen while building Scrubnet. It focuses on practical tests, crawl behaviour, feed formats and anything that helps brands publish clean data for AI.