Add Your Site to Scrubnet

Contribute to crawler research

Scrubnet uses custom feeds as an experimental surface for studying how search crawlers and AI bots discover, fetch and revisit content. Adding more sites gives the feeds a broader content base and helps us compare behaviour across different subjects, structures and update patterns.

Participation is not a promise to optimise your content for bots. It is a way to contribute to an open, evidence-led study of crawler behaviour that may inform technical SEO and GEO work.

What Scrubnet does

• Collects public pages: ScrubberDuck follows your accessible sitemap and respects robots.txt.
• Creates custom feeds: selected content is published in lightweight HTML, JSON and text formats.
• Tracks change signals: source URLs and last-modified timestamps are preserved where available.
• Observes requests: recognised bot visits to Scrubnet feeds appear in the public log dashboard.
• Publishes learnings: useful patterns become experiments, case studies and practical guidance.

What you can observe

The live crawler logs show which bots request Scrubnet feeds, the paths and formats they fetch, their verification status, response codes, bytes transferred and timing relative to the feed’s last modification.

A feed request shows access, not indexing, ranking, citation or use inside an AI product. Scrubnet does not guarantee visibility outcomes.

How it works

1. Subscribe for free: use the Stripe checkout to tell us which public website you want to contribute.
2. Allow ScrubberDuck: keep your sitemap accessible and permit ScrubberDuck/1.0 in robots.txt.
3. We build and refresh the feeds: content is fetched politely and published on Scrubnet.
4. We observe and learn: crawler requests contribute to the live dataset and future research.

Participation terms

• Free for one website with up to 50,000 public URLs.
• Only publicly accessible content is collected.
• You can request corrections, an export or removal at any time.
• Source attribution and timestamps are retained.
• No rankings, indexing, AI citations or traffic outcomes are promised.

Add a site

Use the free subscription below to add a website to Scrubnet’s crawler research dataset.