Crawler Observatory
===================

Site-wide request data and public feeds for observing search and AI crawler behaviour

Explore Live Logs (https://scrubnet.org/dashboard.php)

A research environment within Scrubnet
--------------------------------------

The crawler observatory is Scrubnet's open environment for examining how verified search and AI crawlers discover, fetch and revisit resources across the Scrubnet domain, including public machine-readable feeds. It complements our tools and articles with first-party request data that technical SEO specialists, developers and researchers can inspect directly.

Observatory findings inform practical guidance and new questions for investigation. They remain one part of Scrubnet's wider work to create better ways of inspecting, understanding and working with the technical web.

Explore the observatory
-----------------------

Image: Live Crawler Logs dashboard showing crawler request charts (https://scrubnet.org/crawler-log-dashboard.webp)

Live data

Live Crawler Logs
-----------------

Filter verified requests by crawler, path, format, response and date.

Updated throughout the day

Open the live crawler logs: https://scrubnet.org/dashboard.php

Image: Freshness Observatory dashboard showing crawler request data by content age (https://scrubnet.org/freshness-observatory.webp)

Interactive analysis

Freshness Observatory
---------------------

Explore request timing, content age, formats and crawler distributions.

Interactive crawler data

Open the Freshness Observatory: https://scrubnet.org/freshness-lab.php

Open infrastructure

Public Crawler Feeds
--------------------

Inspect the consistent machine-readable surface used by the observatory.

HTML, JSON, TXT and Markdown

Explore Scrubnet public crawler feeds: https://scrubnet.org/llms.html

Participate

Add a Site
----------

Contribute authorised public content and broaden the observable dataset.

Free for eligible sites

Add a site to the Scrubnet observatory: https://scrubnet.org/brands.html

From observation to practical insight
-------------------------------------

Requests across Scrubnet are examined alongside resource formats, timestamps, status codes and content changes. Useful patterns become documented observations, technical articles and questions that can improve audits, publishing systems and crawler-aware development workflows.

Read Articles & Research (https://scrubnet.org/discoveries.html) Collaborate With Scrubnet (https://scrubnet.org/partnerships.html)

Meet ScrubberDuck
-----------------

ScrubberDuck is the lightweight, robots-aware feed compiler behind the observatory. It collects authorised public content and creates consistent feeds for observing discovery, formats, freshness signals and recrawl behaviour while minimising unnecessary requests.

Image: Illustration of ScrubberDuck, the feed compiler used by Scrubnet (https://scrubnet.org/scrubberduck-200.webp)

User-Agent:ScrubberDuck/1.0 (+https://scrubnet.org)

Interpreting the data
---------------------

A request confirms that a resource was fetched from Scrubnet. It does not by itself prove indexing, ranking, model training, retrieval, citation or a crawler's reason for visiting. Scrubnet reports descriptive findings, documents relevant limitations and keeps observable evidence separate from assumptions.