Crawler Observatory & Live Data

A research environment within Scrubnet

The crawler observatory is Scrubnet's open environment for examining how verified search and AI crawlers discover, fetch and revisit public machine-readable feeds. It complements our tools and articles with first-party request data that technical SEO specialists, developers and researchers can inspect directly.

Observatory findings inform practical guidance and new questions for investigation. They remain one part of Scrubnet's wider work to create better ways of inspecting, understanding and working with the technical web.

Explore the observatory

Live data

Live Crawler Logs

Filter verified requests by crawler, path, format, response and date.

Updated throughout the day

Interactive analysis

Freshness Observatory

Explore request timing, content age, formats and crawler distributions.

Interactive crawler data

Open infrastructure

Public Crawler Feeds

Inspect the consistent machine-readable surface used by the observatory.

HTML, JSON, TXT and Markdown

Participate

Add a Site

Contribute authorised public content and broaden the observable dataset.

Free for eligible sites

From observation to practical insight

Live requests are examined alongside feed formats, timestamps, status codes and content changes. Useful patterns become documented observations, technical articles and questions that can improve audits, publishing systems and crawler-aware development workflows.

Read Articles & Research Collaborate With Scrubnet

Meet ScrubberDuck

ScrubberDuck is the lightweight, robots-aware feed compiler behind the observatory. It collects authorised public content and creates consistent feeds for observing discovery, formats, freshness signals and recrawl behaviour while minimising unnecessary requests.

Illustration of ScrubberDuck, the feed compiler used by Scrubnet

User-Agent: ScrubberDuck/1.0 (+https://scrubnet.org)

Interpreting the data

A request confirms that a resource was fetched from Scrubnet. It does not by itself prove indexing, ranking, model training, retrieval, citation or a crawler's reason for visiting. Scrubnet reports descriptive findings, documents relevant limitations and keeps observable evidence separate from assumptions.