0
requests
0
unique resources
Coverage unavailable
Live Crawler Behaviour Logs
This live view is part of Scrubnet's crawler observatory, the evidence layer within our wider toolkit and knowledge hub for technical SEO and web development.
Explore requests across Scrubnet by path, crawler, response, format and date. The database and this view refresh every 10 minutes.
Interpretation: a request confirms access to Scrubnet only; it does not prove indexing, ranking, model use or citation.
Dataset version loading…
| Time | Last modified | Content age | Method | Status | Bot | Verification | Format | Path | File size | Bytes sent | User Agent |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Loading data... | |||||||||||
Page 1 of 1
Requests by crawler over time
Top 10 crawlers are shown individually; any remaining crawlers are grouped as Other. Click a name to hide its line.
Loading chart...
Response outcomes over time
Content responses, revalidation, redirects and errors are shown separately.
Loading chart...
Requests by crawler, response, status, verification and method
| Crawler | Requests |
|---|---|
| Loading... | |
| Status | Requests |
|---|---|
| Loading... | |
| Verification | Requests |
|---|---|
| Loading... | |
| Method | Requests |
|---|---|
| Loading... | |
| Response outcome | Requests |
|---|---|
| Loading... | |
Requests by format
0
requests
0
unique resources
Coverage unavailable
0
requests
0
unique resources
Coverage unavailable
0
requests
0
unique resources
Coverage unavailable
0requests
0unique resources
Coverage unavailable
Formats over time
Current filters apply. The chart includes HTML, TXT, JSON and Markdown resource requests, plus XML sitemaps when enabled. Click a format name to show or hide its line.
Loading chart...
Rolling 24 hours (UTC)
Path, crawler, method, format, response, verification and discovery-request filters apply; the selected date range does not.
Loading chart...
How to interpret this dataset
The default view includes verified crawler requests across Scrubnet. Verification uses published provider IP ranges or forward-confirmed reverse DNS where available. “Content age” is the time between a resource’s recorded modification date and the request; unknown or future dates are excluded from age statistics and reported through coverage counters.
Request counts show access to Scrubnet, not indexing, ranking, training, retrieval or citation. Format totals are descriptive. Coverage denominators use the latest daily public-resource inventory snapshot and are unavailable when a path filter is active.