# Articles & Research Practical insight for technical SEO specialists and web developers ## Latest articles and research ![Crawler request and response outcome charts in the Scrubnet Observatory](https://scrubnet.org/crawler-log-dashboard.webp) AEO measurement ### From Crawl to Contribution A practical evidence model separating crawler access from retrieval, citation, referral and the value a source contributes to an AI answer. 17 August 2026 [Read From Crawl to Contribution](https://scrubnet.org/discoveries/from-crawl-to-contribution-aeo-measurement.html) ![ScrubberDuck navigating crawler freshness data](https://scrubnet.org/discoveries/amazon-bot-crawl-data.webp) Crawler research ### Freshness Signals vs Crawl Reality First findings from 28 days of verified crawler requests, revealing sharp format differences and an average requested content age of 120 days. 20 July 2026 [Read Freshness Signals vs Crawl Reality](https://scrubnet.org/discoveries/freshness-signals-vs-crawl-reality.html) ![The Ultimate Technical AI Search Optimisation Guide](https://scrubnet.org/discoveries/ultimate-tech-ai-search-optimisation-guide.webp) AI search ### The Ultimate Technical AI Search Optimisation Guide A technical framework for AI search visibility, covering crawler access, rendering, canonicalisation, chunking, structured data, freshness, measurement, and governance. 7 June 2026 [Read The Ultimate Technical AI Search Optimisation Guide](https://scrubnet.org/discoveries/ultimate-technical-ai-search-optimisation-guide.html) ![Technical SEO and crawler visibility framework](https://scrubnet.org/discoveries/beyond-traditional-technical-seo.webp) Technical guidance ### AI Ingestion & Visibility Technical Guidelines A practical framework for crawler access, machine-readable architecture, structured data, bot logging and retrieval visibility. 19 May 2026 [Read AI Ingestion and Visibility Technical Guidelines](https://scrubnet.org/discoveries/ai-ingestion-visibility-technical-guidelines.html) ![Scrubberduck organising clean markdown content for AI agents](https://scrubnet.org/discoveries/cloudflare-markdown-scrubnet.webp) Machine web ### Markdown for Agents What Cloudflare’s Markdown for Agents proposal reveals about low-noise formats, crawler access and machine-readable publishing. 22 February 2026 [Read Markdown for Agents](https://scrubnet.org/discoveries/markdown-for-agents-machine-web.html) ![Scrubberduck in the digital world avoiding google limitations.](https://scrubnet.org/discoveries/2m-google-limit-scrubberduck.webp) Google's Guidelines ### Google’s 2 MB Crawl Limit What Google’s 2 MB crawl limit means for indexing, rendering, resource size and machine-readable publishing. 6 February 2026 [Google’s 2 MB Crawl Limit](https://scrubnet.org/discoveries/google-2mb-crawl-limit-machine-web.html) ![Screenshot illustrating an observed AI interaction with a Scrubnet feed](https://scrubnet.org/discoveries/ai-hesitate-conversation-screenshot.webp) AI etiquette ### When AI Hesitates to Crawl An observed interaction in which ChatGPT initially declined to read Scrubnet feeds, illustrating how agent guardrails can affect user-directed fetching. 3 November 2025 [Read When AI Hesitates to Crawl](https://scrubnet.org/discoveries/ai-hesitates-to-crawl.html) ## About the articles and research This hub brings together practical articles for technical SEO specialists and web developers. Topics include technical SEO, web development, better working methods, AI and search crawling, and evidence-led observations from the Scrubnet Crawler Observatory. For research based on the observatory, we document scope and limitations so crawler access is not confused with indexing, ranking, model use or AI citation. Reports identify the observation period, included crawler requests and relevant exclusions. Where available, they also describe the resources that were eligible to be requested. Differences are reported as associations within Scrubnet’s environment, not as proof that a feed characteristic caused a crawler response.