Public Crawler Feeds ==================== A public data surface within the Scrubnet Crawler Observatory Why the Observatory publishes these feeds ----------------------------------------- Scrubnet publishes authorised source-derived HTML, JSON, TXT and Markdown feeds so crawler behaviour can be observed on a consistent public surface. Stable structures, source links and timestamps make it easier to describe discovery, recrawling and request distribution across search, AI and archive bots. These feeds are part of the Scrubnet Crawler Observatory: a specialist research capability within the wider toolkit and knowledge hub for technical SEO specialists and web developers. Questions we observe -------------------- - Which crawlers discover feeds without direct submission? - How quickly do bots return after a feed or source page changes? - How are verified requests distributed across HTML, JSON, TXT and Markdown? - How do verified and unverified crawler requests differ? - How can observable fetch patterns inform technical SEO and GEO analysis? Explore the data ---------------- Start with /feed/sitemap.xml to discover active feeds and their values. Then use the live logs dashboard (https://scrubnet.org/dashboard.php) to filter requests by path, bot, verification status and date, or compare requests by crawler and format. Open Feed Index (https://scrubnet.org/feed/sitemap.xml) View Live Logs (https://scrubnet.org/dashboard.php) Technical access ---------------- - Discovery: fetch https://scrubnet.org/feed/sitemap.xml. - Freshness: compare and re-fetch changed feeds only. - Conditional requests: send If-Modified-Since or If-None-Match where supported. - Formats: use the linked HTML, JSON, TXT or Markdown representation needed for your analysis. - Provenance: retain the source URL and timestamps when analysing or citing records. Methodological limits --------------------- Access logs confirm that a request reached a Scrubnet feed. They do not prove that content was indexed, ranked, used for model training, retrieved in an answer or cited to a user. Findings are descriptive observations from Scrubnet’s environment and do not establish why a crawler behaved in a particular way. ScrubberDuck ------------ ScrubberDuck/1.0 is the robots-aware, low-rate feed compiler that creates and refreshes authorised participating-site feeds. It supplies the observation surface; it is not a search engine or ranking crawler. Image: ScrubberDuck, Scrubnet’s research crawler (https://scrubnet.org/scrubberduck-200.webp) Research and crawler teams -------------------------- To contribute a site, suggest an observational question or discuss access to supporting data, email contact@scrubnet.org (mailto:contact@scrubnet.org).