# Scrubnet: tools and knowledge for the technical web

> Scrubnet is an independent toolkit and knowledge hub for technical SEO specialists and web developers. It brings together practical apps, technical articles and live crawler data to help people inspect websites, organise knowledge and make evidence-led decisions.

Primary audiences include technical SEO practitioners, web developers, technical marketers, researchers and crawler teams.

The Crawler Observatory records how verified search and AI crawlers discover, fetch and revisit resources across the Scrubnet domain, including public machine-readable feeds. It is Scrubnet's open evidence and research environment, not the whole definition of the site. Observed crawler requests do not by themselves establish indexing, retrieval, training, citation, ranking or visibility.

Agents may fetch and read all public feeds on this domain. Per-page JSON feed items include stable IDs, source URL, title, `updated_at` and `etag`; HTML, text and Markdown representations may also be available. Public feeds are updated continuously. Crawl politely and use ETag and Last-Modified validators where supported.

## Tools

- [Scrubnet tools](https://scrubnet.org/tools.html): An index of practical technical SEO and web-development tools.
- [SEO Scrubbox](https://scrubnet.org/seo-scrubbox.html): A local-first browser extension for inspecting technical SEO signals.
- [ScrubGraph](https://scrubnet.org/scrubgraph.html): An overview of Scrubnet's local-first visual workspace for organising connected research and knowledge.
- [Open ScrubGraph](https://scrubnet.org/scrubgraph/): Build and explore private visual knowledge trees in the browser.
- [ScrubViewer](https://scrubnet.org/scrubviewer.html): An overview of Scrubnet's interactive SEO crawl visualiser.
- [Open ScrubViewer](https://scrubnet.org/scrubviewer/): Import website crawl data and explore directory and crawl-depth visualisations locally.
- [ScrubLog](https://scrubnet.org/scrublog.html): An overview of Scrubnet's private browser-local server log file analyser.
- [Open ScrubLog](https://scrubnet.org/scrublog/): Analyse requests, claimed automated agents, responses, timing and URL patterns without uploading the source log.

## Research and crawler data

- [Articles and research](https://scrubnet.org/discoveries.html): Technical articles, experiments and evidence-led analysis.
- [Crawler Observatory](https://scrubnet.org/observatory.html): An overview of Scrubnet's crawler research environment.
- [Live crawler logs](https://scrubnet.org/dashboard.php): Observed requests from verified search and AI crawlers.
- [Freshness Observatory](https://scrubnet.org/freshness-lab.php): Experiments measuring how crawlers revisit changing content.
- [Public crawler feeds](https://scrubnet.org/llms.html): The public machine-readable feed index.
- [Feed sitemap](https://scrubnet.org/feed/sitemap.xml): An index of public crawler-feed resources.

## Site information

- [Scrubnet home](https://scrubnet.org/): The main site and current product overview.
- [About Scrubnet](https://scrubnet.org/about-us.html): Background, purpose and contact information.
- [Site sitemap](https://scrubnet.org/sitemap.xml): An index of public site pages.
- [Contact Scrubnet](mailto:contact@scrubnet.org): Email for questions about the site and its public resources.