Scrubnet data lab

Freshness Observatory

This interactive view is part of Scrubnet's crawler observatory, the evidence layer within our wider toolkit and knowledge hub for technical SEO and web development.

Read time from left to right and formats from front to back. Each column combines the selected crawlers; height shows the selected metric and colour shows average content age.

Interpretation: this is descriptive request data from Scrubnet. Differences between formats or crawlers do not by themselves establish preference or causation.

Dataset version loading…

Interactive crawler freshness data

Range:
Verified requests — Selected date and path range
Median content age — At request time
Known-age coverage — Share included in age analysis
Unique resources — Selected range
Verified crawlers — Distinct verified labels
Median recrawl interval — Same crawler and resource
Latest observation — UTC
Hourly detail is available for date ranges up to 31 days. Crawler focus
All crawlers
Formats shown
All formats
Loading live data…
Time →
Height: requests
Drag to orbit · pinch or scroll to zoom · tap or click a column to inspect
Fresh ≤1 day Older ≥30 days
Selected column

Choose a column

Click any column to inspect its format, time period, combined crawler activity, freshness, response quality and transfer volume.

Freshness mix

Requests by content age

Accessible 2D comparison

Content age by format and time

Each cell shows the request-weighted mean content age for the currently focused crawlers. Use the request counts in each cell to judge how much activity supports it.

Observation denominator

Requests and eligible resources by format

Times are UTC. HTML, TXT, JSON and MD lanes combine the selected verified crawlers. Only requests with a known content age are shown.

Methodology and limitations

Scrubnet includes every verified request to HTML, TXT, JSON and Markdown feeds in the selected date and path range. Content age is the difference between the recorded source modification date and request time. Unknown and future-dated values are excluded from age statistics, never converted into positive ages.

The 3D and 2D views are descriptive. Counts do not prove preference, indexing or downstream use. Eligible-resource denominators come from the latest daily feed inventory snapshot; they are unavailable when filtering to a path subset. Top crawlers are shown individually and remaining verified crawlers are grouped as Other.