ScrubberDuck navigating crawler freshness data

Freshness Signals vs Crawl Reality

First findings from Scrubnet’s Freshness Observatory

Scrubnet’s new 3D Freshness Observatory examines how verified search and AI crawlers interact with HTML, TXT, JSON and Markdown feeds and how fresh the requested content actually is.

Freshness is frequently discussed as an important part of technical SEO and AI search optimisation.

Sitemaps can tell crawlers when a URL was last modified. HTTP response headers can communicate whether a resource has changed. Publishing systems can expose accurate dates, while protocols such as IndexNow can notify participating search engines when content is added, updated or removed.

The theory is straightforward: provide accurate freshness signals and crawlers should be better equipped to find updated content efficiently.

But what does crawler activity actually look like when these signals are carefully controlled?

Scrubnet’s new Freshness Observatory was created to investigate that question.

It provides a visual representation of verified crawler requests across four content formats, showing not only when bots access the feeds, but also the age of the content they choose to request.

The first 28 days of data have already revealed several unexpected patterns.

Most notably, despite Scrubnet placing a strong emphasis on accurate freshness information, the average age of requested content was approximately 120 days.

Only 4% of requests were for content no more than one day old, while another 8% targeted content between one and seven days old.

That means approximately 88% of the observed requests were for content more than seven days old.

This does not prove that freshness signals are ineffective. It does, however, challenge the simple assumption that accurate freshness information will cause crawler activity to concentrate primarily on recently updated content.

AmazonBot crawl requests overtime by format and freshness signals
AmazonBot crawl requests overtime by format and freshness signals.

Why Scrubnet focuses on freshness

Scrubnet is an independent crawler research and experimentation hub.

It hosts structured, bot-friendly feeds that allow crawler behaviour to be observed in a more controlled environment than would usually be possible on a conventional website.

The feeds are organised into four formats:

A central objective of the system is to make its freshness signals as accurate as possible.

When content changes, Scrubnet attempts to ensure that the modification information exposed through the feed, sitemap and HTTP response is aligned with the real update. When content has not changed, timestamps should not be artificially refreshed simply because a page has been regenerated or requested.

This distinction matters.

Google states that it uses sitemap <lastmod> values when they are consistently and verifiably accurate. Its crawling infrastructure also supports HTTP cache validators such as Last-Modified, If-Modified-Since, ETag and If-None-Match.

Scrubnet is therefore designed around signal accuracy, not timestamp inflation.

The purpose is not to claim that these feeds will receive more crawling, indexing or AI visibility. It is to create an environment where crawler behaviour can be studied while reducing uncertainty about the underlying freshness data.

Building the Freshness Observatory

Traditional log dashboards are useful for totals and timelines, but it becomes difficult to understand the relationship between several dimensions at once:

The Freshness Observatory was built as an interactive 3D HTML5 visualisation to make these relationships easier to explore.

The graph is divided into four lanes representing HTML, TXT, JSON and Markdown. Each column represents crawler activity for a particular time bucket and format.

The height of a column is linearly proportional to request volume, while its colour represents the average age of the requested content. The freshness colour scale adapts to the visible dataset, helping differences remain readable when filters or date ranges change.

Users can orbit around the visualisation, zoom into specific periods, select individual columns and inspect detailed information about the underlying requests.

The Observatory also includes filters for:

Hourly analysis is limited to ranges of 31 days or fewer to keep the visualisation and aggregation service responsive.

The summary counters above the graph update with the active filters and show:

All averages and response percentages are calculated from the data currently visible in the graph rather than from the unfiltered database.

Keeping the dataset focused

The Observatory is backed by an updated MySQL aggregation service.

Only verified crawler requests are included. Records without a usable content age are excluded, including sitemap-style requests where the age of an individual content resource cannot be determined reliably.

The dataset is also restricted to HTML, TXT, JSON and Markdown so unrelated assets do not distort the comparison.

This produces a cleaner research dataset, but it is important to understand what it represents.

The Observatory measures requests, not indexing, retrieval, model training, rankings or citations. A crawler requesting a resource only confirms that the crawler accessed it.

The findings below describe behaviour observed on Scrubnet’s feeds during the selected period. They should not automatically be applied to other domains, larger websites or different content environments.

Amazonbot appears to strongly favour Markdown

One of the clearest format differences came from Amazonbot.

During the analysed period:

Amazonbot also appeared to follow a roughly 24-hour activity cycle. Its crawl rate increased more strongly overnight, based on the timezone used by the Scrubnet dashboard.

The combination of higher Markdown activity and older requested content is particularly interesting.

It could indicate a format-specific queue, a recurring historical recrawl process or a different purpose for the Markdown requests. However, the data does not yet reveal why the behaviour occurs.

It would also be premature to interpret this as evidence that Markdown is universally preferred by Amazon systems. The result currently applies only to the eligible Scrubnet resources and the observed time period.

Amazon’s documentation describes Amazonbot as a crawler used to improve Amazon products and services, with collected material potentially contributing to AI model training.

Longer observation periods may help determine whether the Markdown preference remains stable or was influenced by a temporary crawl campaign.

Meta’s Markdown crawling happens in concentrated bursts

Meta’s external crawler showed a different pattern.

Its HTML, TXT and JSON requests were comparatively distributed over time. Markdown requests, however, appeared in large, concentrated spikes.

Rather than crawling Markdown at a steady background rate, Meta appeared to request groups of Markdown resources during narrower periods of activity.

The age of requested content also varied by format. Meta requested older HTML content than TXT or JSON content during the analysed period.

This suggests that request scheduling may not be shared equally across formats. Different formats may be processed through separate queues, campaigns or systems, although the logs alone cannot confirm the underlying architecture.

The main observation is that aggregate request totals can conceal important behaviour. Two formats may receive a similar number of requests across a month while following completely different temporal patterns.

ClaudeBot maintains a low baseline with occasional large spikes

ClaudeBot produced one of the more recognisable crawl profiles.

For much of the period, it maintained a relatively low but consistent request rate. On occasional days, that baseline was interrupted by unexpectedly large crawl spikes.

No obvious repeating schedule has yet been identified for these bursts.

ClaudeBot accessed all four formats, with HTML receiving the highest overall request volume. Compared with Amazonbot and Bingbot, its format distribution appeared more balanced.

Anthropic distinguishes ClaudeBot, which can collect public web content for model development, from its user-directed and search-specific agents. Anthropic also states that its bots respect robots.txt controls.

The unexplained spikes may represent wider collection runs, recrawl campaigns or changes in demand, but that remains a hypothesis.

A longer dataset will be needed to determine whether the spikes are genuinely irregular or part of a slower cycle that is not visible within 28 days.

Googlebot and GoogleOther may be less operationally separate than expected

Googlebot and GoogleOther both accessed all four Scrubnet formats relatively consistently.

The surprising part was not their overall format coverage, but the timing of specific activity.

Both crawlers produced a large spike in TXT requests at approximately the same time.

A single synchronised event is not enough to demonstrate coordination. It does, however, suggest that the two crawler identities should not necessarily be treated as completely disconnected systems.

Google describes GoogleOther as a generic crawler that may be used by different product teams to fetch publicly accessible content. Google has also explained that multiple crawler clients can operate through the same underlying crawling infrastructure while using different crawler identities and configurations.

Shared infrastructure does not mean Googlebot and GoogleOther have the same purpose. It does mean that similarities in timing could potentially result from shared scheduling, discovery sources, infrastructure or crawl-capacity decisions.

More synchronised events will need to be recorded before drawing a stronger conclusion.

Bingbot appears to largely ignore Markdown

Bingbot showed almost the reverse of Amazonbot’s format behaviour.

Markdown activity was effectively absent from the selected dataset, while HTML received most of Bingbot’s requests.

This is an important reminder that there may not be one universally “crawler-friendly” format.

Markdown is frequently presented as an ideal format for AI systems because it is lightweight, structured and easy to process. On Scrubnet, that appears relevant to Amazonbot’s behaviour but not to Bingbot’s observed activity.

HTML remains the most consistently requested format across the wider collection of crawlers.

That does not necessarily make HTML intrinsically superior. It may simply have broader support, stronger discovery paths or a more established role in crawler processing systems.

The biggest finding: crawlers mostly requested older content

The crawler-specific format differences are useful, but the most surprising result appears when all verified crawler requests are examined together.

Across the selected 28-day period:

Scrubnet’s feeds are specifically designed to expose accurate freshness information. Nevertheless, crawler requests were overwhelmingly directed towards older resources.

At first glance, this may appear to conflict with the emphasis that search platforms place on sitemaps, modification dates, HTTP caching and update notifications.

The more careful interpretation is that freshness signals and crawl allocation are not the same thing.

Freshness signals can help a crawler understand that something has changed. They may help prioritise a URL, avoid unnecessary data transfer or determine whether a cached resource needs to be refreshed.

They do not guarantee that recently changed URLs will account for most requests.

Crawler activity may also be influenced by:

It is also possible that accurate freshness signals make the processing of older resources more efficient without visibly reducing the number of requests. Conditional requests and 304 Not Modified responses, for example, can allow a crawler to validate an existing resource without downloading its complete body.

The current Observatory primarily shows which resources were requested and their recorded age. Future analysis of conditional request headers and response behaviour may reveal another part of the freshness story.

What the findings do not prove

These results should be treated as observations, not universal crawler rules.

They do not prove that:

Scrubnet represents one host, one feed architecture and one controlled collection of resources.

The purpose of publishing these findings is not to produce another definitive optimisation checklist. It is to make crawler behaviour visible enough that better questions can be asked.

There is no single best format for every crawler

The first findings from the Freshness Observatory point towards a fragmented crawler ecosystem.

Amazonbot heavily favoured Markdown within this dataset, while Bingbot largely ignored it. ClaudeBot accessed every format with a stronger emphasis on HTML. Meta’s Markdown activity was concentrated into large spikes. Googlebot and GoogleOther showed unexpectedly similar timing during one major TXT crawl event.

The format that looks optimal for one crawler may be almost irrelevant to another.

The most important result, however, is not about format.

It is that most requests targeted relatively old content, even in an environment where freshness information is carefully maintained.

That does not make freshness signals unimportant. It suggests their influence is more subtle than simply causing bots to spend most of their time on the newest resources.

Accurate freshness information remains part of building an efficient, trustworthy and crawler-friendly web. But understanding how crawlers actually respond requires direct observation, controlled experiments and longer datasets.

That is what the Scrubnet Freshness Observatory was built to provide.