What is Scrubnet?
Scrubnet is a machine-readable layer of the web built specifically for search crawlers and AI systems. We publish clean, structured feeds with reliable timestamps so bots can discover and refresh content quickly, without downloading heavy pages and unnecessary assets.
Why Now?
- 🤖 More discovery is happening through AI assistants and bot-driven systems, not traditional browsing.
- 📦 The modern web is slow and inefficient for crawlers: too much code, too many requests, too little signal.
- ⚡ Scrubnet gives bots a lightweight, structured path to trusted content, with predictable freshness.
Who It’s For
- 🏢 Brands: improve how your products and content are discovered by search and AI systems.
- 🤝 Agencies: onboard clients to Scrubnet and monitor bot activity and impact across sites.
- 🤖 AI platforms & engineers: access cleaner feeds that reduce bandwidth and improve freshness.
Our Principles
- 🛡️ Neutral by design: Scrubnet is independent and not tied to any single AI platform.
- ⚙️ Bot-first by default: we optimise for crawl efficiency, not page aesthetics.
- 🕒 Freshness you can trust: timestamped feeds help bots avoid re-fetching stale data.
The Future We See
As AI changes how people discover information, brands will need a simpler, more direct way to publish trusted data to the systems that power search, assistants, and shopping experiences. Scrubnet is that clean layer: structured, fast, and designed for machine consumption.
Meet ScrubberDuck
ScrubberDuck is our lightweight crawler. It fetches only what’s needed, extracts structured information, and generates clean feeds on Scrubnet with accurate last-modified timestamps.
It’s designed to minimise load, avoid unnecessary requests, and respect robots.txt.
If you see ScrubberDuck in your logs, it means your site is part of Scrubnet and we’re keeping your feeds fresh.
User-Agent: ScrubberDuck/1.0 (+https://scrubnet.org) Clean web noise since 2025
How Scrubnet Works
Scrubnet keeps things simple: we connect a website, fetch pages efficiently, and publish machine-readable feeds that bots can crawl fast. Every feed is built to be lightweight, structured, and reliably up to date.
We’re also building towards direct integrations: APIs for authenticated brand data, and plugins for platforms like Shopify and WordPress to generate clean data at the source.
One simple price: £5/month per website (up to 50,000 URLs). No tiers, no lock-in.
Allowed Bots
Scrubnet is designed for trustworthy search crawlers and AI agents, including:
- Googlebot – Google Search and Discover
- Google-Extended – AI training exclusion support
- GPTBot – OpenAI’s web crawler
- ClaudeBot – Anthropic’s crawler for Claude
- PerplexityBot – Perplexity AI’s research assistant bot
- bingbot – Microsoft Bing search crawler
- BingPreview – Bing page preview bot
- CCBot – Common Crawl archive bot
- DuckDuckBot – DuckDuckGo crawler
- Applebot – Apple Siri and Spotlight crawler
Get Involved
Whether you’re a brand that wants better AI visibility, or an agency looking to onboard clients and measure impact, we’d love to chat.
Or reach out at contact@scrubnet.org
