We optimise websites for bots. But do we actually know what they’re doing?
CV
Charles Vayssié Journey Further & Scrubnet
24 sections · Access logs, crawler behaviour and freshness signals
This talk has two parts: practical access-log analysis for technical SEO, followed by freshness signals and observations from Scrubnet that open up further questions. Read the supporting context below, or use Present to explore the talk as a deck.
01 / A quick introduction
Hi, I’m Charles.
Technical SEO specialist, web developer and founder of Scrubnet.
Charles Vayssié
Journey Further
Technical SEO grounded in evidence.
Working across complex e-commerce and international websites, from technical investigation to practical changes.
Scrubnet
A place to build and investigate.
Practical SEO tools and crawler research, including SEO Scrubbox and the Freshness Observatory.
GET requests the HTML resource at this path. It tells us which representation was requested, rather than just which topic attracted attention.
The timestamp
Put the request on a timeline.
The timestamp includes a timezone offset. Align timezones before comparing crawl activity with releases, content updates or incidents.
The response
A status and a transfer size.
200 is the recorded response status. In this example’s log format, 12418 is the response body size in bytes. A successful response does not establish indexing.
The claimed identity
A user agent is a claim.
This entry claims to be Googlebot. Verification needs a separate check of the source IP against the crawler provider’s documented verification process.
This is an invented entry using a documentation IP address. It is not evidence of a verified Googlebot visit.
Where the log came from matters. An origin server log can miss requests served entirely by a CDN. CDN and server logs may describe different points in the same request journey. Available fields, response sizes and timing measures depend on the logging configuration.
Once we know who requested the URL, add its technical state and search performance.
Access logs
What was requested?
Recorded crawler, resource, time, response and, where logged, transfer size or timing.
Screaming Frog
What signals does the URL expose?
A crawl snapshot of status, canonicals, indexability signals, internal links and other technical details.
Google Search Console
What does Google report?
Search performance, indexing reports and URL-level inspection to add another perspective.
03 / Turning logs into SEO insights
Putting it together: Bot Control.
Journey Further’s dashboard approach: move from separate datasets to an investigation.
ILLUSTRATIVE JOINED VIEW Synthetic URLs and values · Example seven-day window
Invented example combining crawl, access-log and Google Search Console data over seven days.
Example page
Indexable¹
Verified requests²
5xx²
Search clicks³
/guides/log-analysis/
Yes
2
0
140
/guides/crawl-signals/
Yes
640
48
310
/search/?sort=latest
No
1,280
0
0
NEXT QUESTION →
An indexable page has search traffic but few recorded bot requests. Is that expected for its update frequency and crawl history?
¹ Screaming Frog crawl² Access logs³ Google Search Console
A reason to investigate, rather than an automatic diagnosis. Low request volume does not establish a crawl problem. High request volume to a non-indexable URL does not automatically mean wasted crawling. Check the purpose of the resource, the observation window, verification coverage and whether logs capture all relevant requests.
This is a schematic for the talk, not a screenshot of Bot Control or a client result. Its invented URLs and values illustrate the method without representing client data or measured client results.
03 / Turning logs into SEO insights
Find the requests other reports cannot show in detail.
See the affected URL, crawler and timestamp for failures and interrupted requests.
Find
Missing or blocked resources
Pinpoint 404 and 410 requests, access blocks such as 403, and throttled requests returning 429.
Connect
Which bots hit which URLs?
Group requests by crawler, URL pattern and time to locate the affected resources and recurring paths.
Find
Failures bots actually encountered
Locate 500, 502 and 503 responses in recorded crawler requests, including short-lived incidents.
Connect
The exact request and time
Match failures to important pages, origin or edge activity, and logged response times. See whether later requests succeeded.
Find · NGINX logs
Requests that ended early
499 records a client disconnect before NGINX sends response headers. Logs reveal the URL, time and recorded client.
Extra visibility
A detail Search Console cannot label as 499
This is a local logging code, not an HTTP response sent to Googlebot. Search Console may record a fetch problem, but does not expose your NGINX 499 log entry.
Publication, editorial updates, page updates and HTTP versions describe different events.
<lastmod>2026-11-01</lastmod>
Significant page update
When did this URL meaningfully change?
Sitemap lastmod can reflect significant changes to content, links or structured data. It is not simply the time the sitemap was generated.
"dateModified":
"2026-11-01T10:00:00Z"
Editorial content
When was this content updated?
Structured data can describe the article or another entity on the page. Compare dates for the same item; sitemap lastmod describes the URL as a whole.
Last-Modified:
Sun, 08 Nov 2026 10:00:00 GMT
HTTP representation update
When did the served resource change?
A regenerated or redeployed HTML file may have a newer HTTP date without an editorial update. Newer than lastmod or dateModified is not automatically a conflict.
ETag: "feed-html-v7"
HTTP representation validator
Does this version still match?
An ETag identifies a representation version, not a publication date. A client can send If-None-Match to validate its cached copy.
"datePublished": "2026-07-20"
Published: 20 July 2026
Updated: 1 November 2026
Publication and reader context
Created, then meaningfully updated.
Publication and editorial update are separate events. Visible dates should make that distinction clear and agree with the content they describe.
Older content still received most of the requests.
A historical observation from the first 28 days reported by Scrubnet’s Freshness Observatory.
88%
of observed requests targeted content more than seven days old.
No more than one day old4%
More than one, up to seven days old8%
More than seven days old88%
What might explain this?
Possible explanations
Revisit schedules: bots may be returning to familiar resources on established cycles.
The available content: older URLs may make up most of the resources they can request.
Other priorities: discovery paths, links and crawler-specific goals may matter alongside age.
Several factors could contribute to the same request mix.
Historical snapshot, not a live dashboard. These approximate percentages come from Freshness Signals vs Crawl Reality, published on . The article describes an initial 28-day observation window; its publication date is not an exact start or end date for that window.
Interpreting the request mix requires the age distribution of eligible resources, the recorded definition of content age and other influences on crawling. The observation alone cannot establish indexing, ranking, AI training use or the causal effect of a particular signal. The figures refer to that historical observation window, rather than the current state of the live tool.
05 / What I’ve started discovering
The crawlers did not follow one shared pattern.
Historical observations from the published 28-day Scrubnet study.
Recorded formats
All four formats
Googlebot and GoogleOther both requested HTML, Markdown, TXT and JSON.
Request timing
A coinciding TXT spike
Both crawlers produced a large spike in TXT requests at approximately the same time.
Recorded formats
Mostly HTML
HTML received most of Bingbot’s requests in the selected dataset.
Markdown activity
Very few requests
Very little Markdown activity was recorded during the observed period.
Markdown activity
Approximately twice the requests
Amazonbot requested Markdown roughly twice as often as either HTML or TXT.
JSON activity
No requests observed
No Amazonbot requests to JSON resources were recorded in this dataset.
Markdown activity
Concentrated bursts
Meta’s external crawler requested groups of Markdown resources in large spikes.
Other formats
More spread over time
Its HTML, TXT and JSON requests were comparatively distributed across the observed period.
Request timing
A low baseline, then large spikes
ClaudeBot maintained relatively low activity for much of the period, interrupted by occasional large crawl spikes.
Recorded formats
All four formats
HTML, Markdown, TXT and JSON were requested, with HTML receiving the highest overall request volume.
External technical sources checked on ; sitemap, HTTP freshness and request-reporting references reviewed again on . Scrubnet figures are historical, from the article published on 20 July 2026. Worked Bot Control examples are illustrative and contain no client data.
06 / Keep in touch
Thank you.
Questions or observations? Let’s keep the conversation going.