Scrubnet data lab

Freshness Observatory

Read time from left to right and formats from front to back. Each column combines the selected crawlers; height shows the selected metric and colour shows average content age.

Interactive crawler freshness data

Requests — Four content formats shown
Average content age — At request time
Successful responses — HTTP 2xx–3xx
Hourly detail is available for date ranges up to 31 days. Crawler focus
All crawlers
Formats shown
All formats
Loading live data…
Time →
Height: requests
Drag to orbit · scroll to zoom · click a column to inspect
Fresh ≤1 day Older ≥30 days
Selected column

Choose a column

Click any column to inspect its format, time period, combined crawler activity, freshness, response quality and transfer volume.

Freshness mix

Requests by content age

Times are UTC. HTML, TXT, JSON and MD lanes combine the selected verified crawlers. Only requests with a known content age are shown.