About Scrubnet | Public Crawler Observatory

Hi, I’m Charles

I’m a technical SEO specialist and web engineer with a background spanning site builds, IT systems, infrastructure and digital strategy. My work covers audits, migrations, Core Web Vitals, crawl management, schema, international SEO and data-led reporting.

Previously, as a Head of IT, I managed infrastructure, cybersecurity and hardware. That server-to-SERP perspective shapes how I investigate crawlers: start with what the systems actually request and record, then build conclusions from evidence.

Why I built Scrubnet

Scrubnet began as an attempt to create a cleaner publishing layer for search and AI systems. Running that infrastructure surfaced a more useful question: what do crawlers actually request when they encounter consistent, machine-readable feeds?

Scrubnet now focuses on that question as a public crawler observatory. It brings together authorised source-derived feeds, verified access logs, descriptive analysis and practical tools to improve our understanding of search and AI crawler behaviour. The aim is not to claim a shortcut to visibility or explain causation, but to make observable request behaviour easier to inspect and interpret.

Independent of any search engine, AI platform or tool vendor.
Observable through public feeds, logs and documented methods.
Careful about the difference between crawling, indexing, ranking and citation.
Open to collaborators who can add sites, questions, analysis or crawler expertise.

Get in touch

To contribute a site, suggest an observational question or discuss access to supporting data, email contact@scrubnet.org.