SEO Scrubbox

Interactive visual crawler

ScrubberCrawl maps site architecture in real time as pages are discovered, giving you a live crawl graph directly inside the browser.

• Builds a dynamic node graph as the crawler discovers internal URLs
• Switches between directory routing and crawl depth visualisations
• Maps node size and colour to metrics such as status code, indexability, inlinks, outlinks, word count, and file size
• Filters the visible graph by indexability or HTTP status
• Highlights specific URLs or metrics with live search
• Saves crawl history so previous snapshots can be loaded, reviewed, or deleted
• Imports and exports crawl data as CSV for reporting and graph review
• Crawls up to 500 URLs per run from your browser

Fast checks with rendered awareness

• Detects current URL and full redirect chains in correct order
• Shows real HTTP status codes with colour-coded interpretation
• Compares canonical tags before and after JavaScript rendering
• Flags canonical changes introduced by JS
• Detects meta robots and Googlebot directives
• Highlights noindex, nofollow, none, X-Robots-Tag, and conflicting signals
• Changes the extension icon colour based on live indexability signals
• Toggles visual overlays for rel="nofollow" links and data-nosnippet elements
• Identifies JavaScript changes to titles and meta descriptions while ignoring harmless entity differences
• Checks H1 structure and whether the H1 is genuinely the first heading on the page
• Includes a per-site JavaScript toggle for controlled render testing

Deep technical audit

When you need more than a quick read, the deep audit runs controlled HTTP-based checks to validate crawl paths, staging environments, sitemap structures, internal linking health, and hreflang integrity.

• Passes active session cookies and authorisation headers through to the crawler in staging mode
• Audits pre-production and password-protected staging environments
• Controls concurrency, request delay, internal URL limits, and sitemap check limits
• Recursively discovers and evaluates sitemaps declared in robots.txt
• Detects sitemap index vs URL set structure
• Shows lastmod values when available
• Applies intelligent safety limits for very large sitemap environments
• Validates hreflang status codes
• Checks hreflang return link reciprocity
• Audits internal links with status codes, redirect chains, and rel="nofollow" flags
• Groups internal link issues by HTTP status code
• Counts internal link frequency with one-click spreadsheet-ready copying
• Flags links injected exclusively by client-side JavaScript
• Uses session caching for fast re-runs and includes an abort control for cancelling deep audits

Server, security, and caching diagnostics

SEO Scrubbox checks the HTTP layer around the page so you can spot crawl, security, and caching signals that are invisible in the rendered DOM.

• Audits security headers including Content-Security-Policy, Strict-Transport-Security, and X-Content-Type-Options
• Evaluates Cache-Control policies, ETags, and Vary headers
• Detects modern compression formats such as Brotli, Gzip, and Zstd
• Differentiates actual HTTP redirects from client-side canonical rewriting
• Shows final destination status codes for redirected internal links

Rendered vs non-rendered comparison

Modern sites often rewrite key signals after JavaScript execution. SEO Scrubbox makes those shifts explicit so you can diagnose issues like canonical drift, missing internal links, or content that only appears after rendering.

• Compares internal link count before and after rendering with visual summaries
• Compares word count before and after rendering
• Helps identify JavaScript SEO issues and hidden content
• Surfaces DOM rewriting issues, JS-injected links, and dynamically loaded text

Structured data inspection

SEO Scrubbox detects JSON-LD on the page, validates it per script, and organises schema into readable accordions. This helps you verify both correctness and consistency for indexing and AI entity understanding.

• Detects all JSON-LD scripts on the page
• Validates JSON format per script
• Flags each script with a clear valid or not valid status
• Extracts @type and @graph details so each entity is labelled clearly
• Formats and indents raw JSON-LD for review
• Copies raw schema and grouped link errors in spreadsheet-ready formats
• Supports complex schema types including Organization, Product, FAQPage, CollectionPage, BreadcrumbList, ItemList and more
• Extracts structured data in a way that aligns with AI search ingestion and entity understanding

Indexability clarity

Indexability checks combine rendered DOM signals, HTTP headers, and robots.txt rules so quick checks and deep audits tell the same story.

• Parses robots.txt with Google-style precedence logic
• Checks AI crawler access for agents including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended
• Flags nosnippet and max-snippet:0 controls across meta tags and X-Robots-Tag headers
• Evaluates X-Robots-Tag headers alongside DOM meta robots tags
• Maintains consistency between instant quick checks and full deep audits

Search and AI crawler diagnostics

SEO Scrubbox includes checks for the signals exposed to traditional search engines, answer engines and AI crawlers, without treating crawler access as a visibility guarantee.

• Google Search Generative Experiences and AI Overviews
• Bing AI and emerging answer engines
• LLM-based crawlers and AI agents
• Entity clarity, structured data consistency, and canonical stability
• Clean, machine-readable crawl signals and rendering consistency

By comparing view-source HTML and rendered DOM signals, it helps you test whether crawler-facing directives and content remain consistent across traditional search and AI access paths.

Core Web Vitals (CrUX integration)

SEO Scrubbox includes an optional Chrome UX Report integration so you can view real user metrics directly in your audit workflow.

• Uses your own Google Chrome UX Report API key
• Displays real-world field data for LCP, INP, and CLS
• Classifies metrics as Good, Needs Improvement, or Poor

User-agent control

SEO Scrubbox can emulate crawler requests so you can test how servers respond to different agents and headers.

• Emulates Googlebot Smartphone and Googlebot Desktop
• Supports custom user agents
• Dynamically modifies request headers to test server responses