Tech SEO North · 20 minute talk

01 / The web is full of bots

Access logs & freshness signals.

What bots are really doing.

We optimise websites for bots.
But do we actually know what they’re doing?

Charles Vayssié Journey Further & Scrubnet

24 sections · Access logs, crawler behaviour and freshness signals

This talk has two parts: practical access-log analysis for technical SEO, followed by freshness signals and observations from Scrubnet that open up further questions. Read the supporting context below, or use Present to explore the talk as a deck.

01 / A quick introduction

Hi, I’m Charles.

Technical SEO specialist, web developer and founder of Scrubnet.

Charles Vayssié, founder of Scrubnet
Charles Vayssié
Journey Further

Technical SEO grounded in evidence.

Working across complex e-commerce and international websites, from technical investigation to practical changes.

Scrubnet

A place to build and investigate.

Practical SEO tools and crawler research, including SEO Scrubbox and the Freshness Observatory.

Part 01

Access logs

What logs record. Where to find them. How we use them.

Technical SEO · Client work · Practical insights

01 / The web is full of bots

The web has a very large
non-human audience.

Cloudflare reported that non-human traffic crossed 50% in 2026.

>50%

The threshold described in Cloudflare’s
report.

Search is only part of the picture.

Training crawlers, assistants, monitoring systems and other automation also make requests.

This is a wider traffic observation. It does not tell us the bot mix on any particular website.

Cloudflare · July 2026 report

01 / The web is full of bots

“A bot visited” is not the whole story.

Separate the documented purpose of a crawler from what one request can prove.

Purpose

Find and refresh search content

Googlebot and Bingbot are familiar examples. AI search has its own crawlers too.

Example

Googlebot

Google’s crawler for Google Search. Its user-agent strings contain Googlebot/2.1. A visit is not a guarantee of indexing.

Purpose

Collect material for model development

Some crawlers have a collection purpose rather than an immediate user query.

Example

GPTBot

OpenAI says content it crawls may be used for model training. A log entry does not show that training happened.

Purpose

Fetch a page for a user action

An assistant may request a resource while carrying out a particular task.

Example

ChatGPT-User

OpenAI distinguishes this agent from automatic web crawling. Keep it separate in your analysis.

Google · Googlebot user agents OpenAI · training and user-triggered agents

02 / Understanding access logs

Start with a single request.

A log entry records an interaction. Understanding it means knowing what each field tells us.

access.log · combined-style entry Illustrative
192.0.2.24 - -
[01/Nov/2026:10:32:04 +0000]
"GET /discoveries/freshness-signals-vs-crawl-reality.html HTTP/2.0"
200 12418 "-"
"Googlebot/2.1 (+http://www.google.com/bot.html)"
The request

A method and a resource.

GET requests the HTML resource at this path. It tells us which representation was requested, rather than just which topic attracted attention.

The timestamp

Put the request on a timeline.

The timestamp includes a timezone offset. Align timezones before comparing crawl activity with releases, content updates or incidents.

The response

A status and a transfer size.

200 is the recorded response status. In this example’s log format, 12418 is the response body size in bytes. A successful response does not establish indexing.

The claimed identity

A user agent is a claim.

This entry claims to be Googlebot. Verification needs a separate check of the source IP against the crawler provider’s documented verification process.

This is an invented entry using a documentation IP address. It is not evidence of a verified Googlebot visit.

Where the log came from matters. An origin server log can miss requests served entirely by a CDN. CDN and server logs may describe different points in the same request journey. Available fields, response sizes and timing measures depend on the logging configuration.

For Google crawlers, use the documented IP-range or DNS verification methods. Keep the verified identity separate from the raw user-agent string.

Apache access logs NGINX log format

02 / Understanding access logs

Which part of the journey
did your log actually see?

A CDN and an origin server observe different parts of a request.

  1. 01

    Crawler

    Requests the URL.

  2. 02

    CDN / edge

    May serve a cached response or apply a rule.

  3. 03

    Origin

    Receives the request if it is forwarded.

Server access logs

Apache / NGINX / hosting platform

Start with the configured access-log destination or your host’s request-log export.

CDN request logs

Cloudflare / Akamai / other edges

Check the provider’s export, fields, retention and coverage. A CDN cache hit may be absent from the origin log.

Apache logging Cloudflare Logpush

02 / Understanding access logs

A user agent is a claim.
Verification is a separate step.

For Google requests, follow Google’s published verification process.

  1. 01

    Preserve the claim

    Keep the original user-agent string.

  2. 02

    Check the source

    Use the provider’s published IP ranges, or its documented reverse-and-forward DNS method.

  3. 03

    Record the result

    Keep verified, unverified and unknown identities distinct.

Behind a proxy, first establish which recorded IP is trustworthy. An unmatched request is not automatically a malicious one.

Google · verifying crawler requests

03 / Turning logs into SEO insights

Three sources.
Three different questions.

Once we know who requested the URL, add its technical state and search performance.

Access logs

What was requested?

Recorded crawler, resource, time, response and, where logged, transfer size or timing.

Screaming Frog

What signals does the URL expose?

A crawl snapshot of status, canonicals, indexability signals, internal links and other technical details.

Google Search Console

What does Google report?

Search performance, indexing reports and URL-level inspection to add another perspective.

03 / Turning logs into SEO insights

Putting it together:
Bot Control.

Journey Further’s dashboard approach: move from separate datasets to an investigation.

ILLUSTRATIVE JOINED VIEW
Synthetic URLs and values · Example seven-day window

Invented example combining crawl, access-log and Google Search Console data over seven days.
Example pageIndexable¹Verified requests²5xx²Search clicks³
/guides/log-analysis/Yes20140
/guides/crawl-signals/Yes64048310
/search/?sort=latestNo1,28000
NEXT QUESTION →

An indexable page has search traffic but few recorded bot requests. Is that expected for its update frequency and crawl history?

¹ Screaming Frog crawl ² Access logs ³ Google Search Console

A reason to investigate, rather than an automatic diagnosis. Low request volume does not establish a crawl problem. High request volume to a non-indexable URL does not automatically mean wasted crawling. Check the purpose of the resource, the observation window, verification coverage and whether logs capture all relevant requests.

This is a schematic for the talk, not a screenshot of Bot Control or a client result. Its invented URLs and values illustrate the method without representing client data or measured client results.

03 / Turning logs into SEO insights

Find the requests other reports
cannot show in detail.

See the affected URL, crawler and timestamp for failures and interrupted requests.

Find

Missing or blocked resources

Pinpoint 404 and 410 requests, access blocks such as 403, and throttled requests returning 429.

Connect

Which bots hit which URLs?

Group requests by crawler, URL pattern and time to locate the affected resources and recurring paths.

Find

Failures bots actually encountered

Locate 500, 502 and 503 responses in recorded crawler requests, including short-lived incidents.

Connect

The exact request and time

Match failures to important pages, origin or edge activity, and logged response times. See whether later requests succeeded.

Find · NGINX logs

Requests that ended early

499 records a client disconnect before NGINX sends response headers. Logs reveal the URL, time and recorded client.

Extra visibility

A detail Search Console cannot label as 499

This is a local logging code, not an HTTP response sent to Googlebot. Search Console may record a fetch problem, but does not expose your NGINX 499 log entry.

Google · HTTP status handling NGINX · 499 definition Search Console · Crawl Stats

Part 02

Freshness signals

What we communicate. What we observe. What remains unclear.

Scrubnet · Crawler observations · Open questions

04 / Freshness signals: are bots listening?

What happens after
we update content?

Access logs help us find technical issues. On Scrubnet, I also use them to observe what follows a content update.

Authorised content, published in different representations for readers and automated systems.

HTML Web page
MD Markdown
TXT Plain text
JSON Structured data

From finding issues to understanding behaviour.

We update the content and communicate the change. Which crawlers return, when, and for which format?

Scrubnet · research context

04 / Freshness signals: are bots listening?

Freshness signals do
different jobs.

Publication, editorial updates, page updates and HTTP versions describe different events.

<lastmod>2026-11-01</lastmod>
Significant page update

When did this URL meaningfully change?

Sitemap lastmod can reflect significant changes to content, links or structured data. It is not simply the time the sitemap was generated.

"dateModified":
"2026-11-01T10:00:00Z"
Editorial content

When was this content updated?

Structured data can describe the article or another entity on the page. Compare dates for the same item; sitemap lastmod describes the URL as a whole.

Last-Modified:
Sun, 08 Nov 2026 10:00:00 GMT
HTTP representation update

When did the served resource change?

A regenerated or redeployed HTML file may have a newer HTTP date without an editorial update. Newer than lastmod or dateModified is not automatically a conflict.

ETag: "feed-html-v7"
HTTP representation validator

Does this version still match?

An ETag identifies a representation version, not a publication date. A client can send If-None-Match to validate its cached copy.

"datePublished": "2026-07-20"

Published: 20 July 2026
Updated: 1 November 2026
Publication and reader context

Created, then meaningfully updated.

Publication and editorial update are separate events. Visible dates should make that distinction clear and agree with the content they describe.

Google · sitemap lastmod HTTP · Last-Modified Google · page dates

04 / Freshness signals: are bots listening?

A request can save a download.

A conditional GET can still appear in the access log even when no content body is returned.

Client → server
GET /resource.html
Server → client
200 OK
ETag: "v1"

[content body]

The client stores the representation and its validator.

Client → server
GET /resource.html
If-None-Match: "v1"
Server → client
304 Not Modified
ETag: "v1"

[no content body]

There was still a request. The existing body can be reused.

Client → server
GET /resource.html
If-None-Match: "v1"
Server → client
200 OK
ETag: "v2"

[updated content body]

The validator no longer matches, so the updated representation is sent.

HTTP conditional requests Google crawler caching support

05 / What I’ve started discovering

Observe the request.
Keep the context.

The Freshness Observatory connects crawler, format and request time with the recorded age of the resource.

Historical Amazonbot Observatory screenshot, showing different request volumes and content ages across format lanes
Published Amazonbot screenshot · its own filtered view.
Illustrative age calculation · all times UTC
  1. Recorded update
  2. Crawler request
  3. Age at request 3 days old

Date source: feed JSON dateModified, with a file-timestamp fallback.

Request time − stored resource modification time.

The published study covers verified requests with a usable age, across HTML, Markdown, TXT and JSON. It does not measure indexing or training.

Method and published observations Open the public Observatory

05 / What I’ve started discovering

Older content still received
most of the requests.

A historical observation from the first 28 days reported by Scrubnet’s Freshness Observatory.

88%

of observed requests targeted content more than seven days old.

  • No more than one day old 4%
  • More than one, up to seven days old 8%
  • More than seven days old 88%

What might explain this?

Possible explanations
  • Revisit schedules: bots may be returning to familiar resources on established cycles.
  • The available content: older URLs may make up most of the resources they can request.
  • Other priorities: discovery paths, links and crawler-specific goals may matter alongside age.

Several factors could contribute to the same request mix.

Historical snapshot, not a live dashboard. These approximate percentages come from Freshness Signals vs Crawl Reality, published on . The article describes an initial 28-day observation window; its publication date is not an exact start or end date for that window.

Interpreting the request mix requires the age distribution of eligible resources, the recorded definition of content age and other influences on crawling. The observation alone cannot establish indexing, ranking, AI training use or the causal effect of a particular signal. The figures refer to that historical observation window, rather than the current state of the live tool.

05 / What I’ve started discovering

The crawlers did not
follow one shared pattern.

Historical observations from the published 28-day Scrubnet study.

Recorded formats

All four formats

Googlebot and GoogleOther both requested HTML, Markdown, TXT and JSON.

Request timing

A coinciding TXT spike

Both crawlers produced a large spike in TXT requests at approximately the same time.

Recorded formats

Mostly HTML

HTML received most of Bingbot’s requests in the selected dataset.

Markdown activity

Very few requests

Very little Markdown activity was recorded during the observed period.

Markdown activity

Approximately twice the requests

Amazonbot requested Markdown roughly twice as often as either HTML or TXT.

JSON activity

No requests observed

No Amazonbot requests to JSON resources were recorded in this dataset.

Markdown activity

Concentrated bursts

Meta’s external crawler requested groups of Markdown resources in large spikes.

Other formats

More spread over time

Its HTML, TXT and JSON requests were comparatively distributed across the observed period.

Request timing

A low baseline, then large spikes

ClaudeBot maintained relatively low activity for much of the period, interrupted by occasional large crawl spikes.

Recorded formats

All four formats

HTML, Markdown, TXT and JSON were requested, with HTML receiving the highest overall request volume.

Scrubnet · crawler-by-crawler findings

05 / What I’ve started discovering

An observatory,
not a controlled experiment.

Scrubnet’s current setup is built for observation. It does not yet follow a controlled experimentation protocol.

Observed setting

One site, one setup

Scrubnet’s feeds, resource mix and selected 28-day window. Another website may show different behaviour.

Study design

No controlled comparison

Formats and resources were not randomised treatments. Multiple factors can vary together, so this setup does not isolate causes.

How to use it

Questions to explore

Use these patterns as food for thought. They are not yet general recommendations for crawling, formats or freshness on other sites.

Keep each observation tied to its website, resources and time window.

Scrubnet · study scope and method

Conclusion

Takeaways

What to use today. What to keep observing.

Access logs · Joined data · Freshness signals

06 / There is still so much to discover

Three ways to get closer
to bot activity.

Use access logs to connect crawler requests with the pages that matter.

  1. 01

    Look at your logs

    CDN or server: see which bots request your URLs, when, and what happens.

  2. 02

    Add the page context

    Join Search Console performance with regular crawls, indexability signals and internal links.

  3. 03

    Investigate sooner

    As logs arrive, spot request patterns and failures before they appear in delayed page reports.

A more detailed, timely view of bot interactions — with search performance alongside it.

Search Console · Crawl Stats coverage

06 / There is still so much to discover

Accurate signals.
A wider crawling picture.

Keep freshness information meaningful. Its influence on revisit timing remains difficult to isolate.

What we can do

Describe changes accurately

Keep publication, editorial, sitemap and HTTP signals consistent with their roles. The dates do not all need to match.

What remains unclear

How much each signal changes crawling

Discovery, linking, server behaviour and crawler priorities form a wider picture. These observations do not isolate the effect of a freshness signal.

I’m documenting what happens on Scrubnet and sharing those observations as the dataset develops.

06 / There is still so much to discover

There is still so much
to discover.

“The more we observe how bots behave, the more we realise how much there is still to learn.”
Questions or observations?

Charles Vayssié · Journey Further & Scrubnet

scrubnet.org/talks/access-logs-and-freshness-signals.html ↗

06 / Keep in touch

Thank you.

Questions or observations? Let’s keep the conversation going.

QR code for https://scrubnet.org/
Scan to explore Scrubnet Tools, articles and crawler research.

Your browser is the deck.

Present switches to a focused layout. All 24 slides remain on the page, so ordinary scrolling and the six chapter links still work. Supporting explanations return in reading mode and print.

← / → · PgUp / PgDn
Previous or next interactive step, then section, in presentation mode.
Space / Shift + Space
Next or previous step in presentation mode. Focused buttons keep their usual Space action.
Home / End
First or last section in presentation mode.
P · Esc
Toggle presentation mode · return to reading.
F · ?
Toggle fullscreen · open this help.

Next explores each slide’s interactive states before continuing: log fields, crawler purposes, dashboard investigations, response codes, freshness signals, HTTP validation, observed crawler patterns. Previous reverses the sequence. The on-screen navigation buttons follow the same order in either mode.

The fullscreen button needs browser support. The layout also works in a normal browser window. Print creates a readable handout from the same content.