Add Your Site to Scrubnet

Contribute to the crawler observatory

Scrubnet publishes authorised source-derived feeds and records how verified search crawlers and AI bots discover, fetch and revisit them. Adding more sites broadens the observable content base and helps document request patterns across different subjects, structures and update cycles.

Participation is not a promise to optimise your content for bots. It contributes to an open observational dataset that may inform technical SEO and GEO analysis without claiming that the feeds cause a visibility outcome.

What Scrubnet does

• Collects authorised public pages: ScrubberDuck follows your accessible sitemap and respects robots.txt.
• Creates machine-readable feeds: selected content is published in lightweight HTML, JSON, TXT and Markdown formats.
• Tracks change signals: source URLs and last-modified timestamps are preserved where available.
• Observes requests: recognised bot visits to Scrubnet feeds appear in the public log dashboard.
• Publishes observations: useful request patterns become descriptive reports, case studies and practical guidance.

What you can observe

The live crawler logs show which bots request Scrubnet feeds, the paths and formats they fetch, their verification status, response codes, bytes transferred and timing relative to the feed’s last modification.

A feed request shows access, not indexing, ranking, citation or use inside an AI product. Scrubnet does not guarantee visibility outcomes.

How it works

1. Subscribe for free: use the Stripe checkout to tell us which website you are authorised to contribute.
2. Allow ScrubberDuck: keep your sitemap accessible and permit ScrubberDuck/1.0 in robots.txt.
3. We build and refresh the feeds: content is fetched politely and published on Scrubnet.
4. We observe and learn: crawler requests contribute to the live dataset and future research.

Participation terms

• Free for one website with up to 50,000 public URLs.
• You confirm that you own the submitted site or are authorised to contribute its content.
• Only publicly accessible content covered by that authorisation is collected.
• You can request corrections, an export or removal at any time.
• Source attribution and timestamps are retained.
• No rankings, indexing, AI citations or traffic outcomes are promised.

Add a site

Use the free subscription below to add an authorised website to Scrubnet’s observational crawler dataset.