Scrubnet ======== Practical tools and technical insight for people who build and optimise the web Explore Tools (https://scrubnet.org/tools.html) Browse Articles (https://scrubnet.org/discoveries.html) Better ways to do technical work -------------------------------- Scrubnet is an independent toolkit and knowledge hub for technical SEO specialists and web developers. It brings together focused apps, practical articles and first-hand crawler data to make complex work easier to inspect, organise and understand. - Inspect: investigate technical signals and diagnose website issues. - Organise: turn complex information into useful, portable working knowledge. - Learn: explore technical guidance, experiments and evidence from the live web. Featured Scrubnet tools ----------------------- SEO Scrubbox and ScrubGraph support complementary parts of technical SEO and knowledge work, with a consistent local-first approach. Image: SEO Scrubbox browser extension interface (https://scrubnet.org/seo-scrubbox.png) Chrome extension SEO Scrubbox ------------ Compare source and rendered SEO signals, test indexability, inspect bot access and run technical diagnostics directly in Chrome. Free browser extension Explore SEO Scrubbox: https://scrubnet.org/seo-scrubbox.html Image: ScrubGraph visual knowledge tree interface (https://scrubnet.org/scrubgraph.png) Knowledge app ScrubGraph ---------- Build and navigate private visual knowledge trees with Markdown notes, flexible editing, search and portable branch or full-graph exports. Private, local-first browser app Explore ScrubGraph: https://scrubnet.org/scrubgraph.html See the full tool collection → (https://scrubnet.org/tools.html) Articles for technical practitioners ------------------------------------ Scrubnet publishes practical articles about technical SEO, web development, AI search and crawler behaviour. Some explain better ways of working; others turn experiments and observed data into careful, useful analysis. Browse all articles and research → (https://scrubnet.org/discoveries.html) The crawler observatory ----------------------- The observatory is Scrubnet’s live research layer. It records how recognised search and AI crawlers request authorised machine-readable feeds, providing public data for descriptive analysis of discovery, recrawling, freshness and format choices. Requests show access behaviour, not indexing, ranking or citation. Image: Live Crawler Logs dashboard showing crawler request charts (https://scrubnet.org/crawler-log-dashboard.webp) Live data Crawler request logs -------------------- Filter recent requests by crawler, path, format, response and date. Crawler request logs: https://scrubnet.org/dashboard.php Image: Freshness Observatory dashboard showing crawler request data by content age (https://scrubnet.org/freshness-observatory.webp) Interactive analysis Freshness Observatory --------------------- Explore request timing, content age and crawler distribution visually. Freshness Observatory: https://scrubnet.org/freshness-lab.php Start here Observatory overview -------------------- Understand the research environment, its methods and how to interpret the data. Observatory overview: https://scrubnet.org/observatory.html Open infrastructure Public crawler feeds -------------------- Inspect the machine-readable feeds behind the observation environment. Public crawler feeds: https://scrubnet.org/llms.html Practitioner-built and evidence-led ----------------------------------- Scrubnet was founded by Charles Vayssié, a technical SEO specialist and web developer who leads strategy and delivery across complex e-commerce and international websites. His work spans migrations, rendering, site architecture, performance, structured data and international SEO, with technical findings translated into practical engineering roadmaps. - Independent: Scrubnet is not tied to a search engine, AI platform or tool vendor. - Practical: tools and guidance are created around real technical workflows. - Evidence-led: observed behaviour is kept separate from hypotheses and marketing claims. - Open where possible: crawler feeds, live logs and findings remain public wherever access and privacy allow. Learn more about Scrubnet → (https://scrubnet.org/about-us.html) Image: Member of Manchester Digital Badge (https://scrubnet.org/Member-MD-B&W.webp) (https://www.manchesterdigital.com/member/scrubnet-ltd) Take part --------- Add an authorised site to broaden the crawler dataset, suggest a technical topic, or collaborate with Scrubnet on an analysis, case study or useful way of working. Observatory participation is free: add a website with up to 50,000 public URLs. There are no visibility or ranking guarantees. Collaborate (https://scrubnet.org/partnerships.html) Want to contribute to the crawler research? Add an authorised site (https://scrubnet.org/brands.html). Or reach out at contact@scrubnet.org (mailto:contact@scrubnet.org)