{
  "format": "scrubnet-page",
  "version": 1,
  "language": "en-GB",
  "title": "Access Logs & Freshness Signals: What Bots Are Really Doing | Scrubnet",
  "description": "Tech SEO North presentation by Charles Vayssié: using access logs, crawl data and Search Console to investigate bot activity and freshness signals.",
  "source": "https://scrubnet.org/talks/access-logs-and-freshness-signals.html",
  "representations": {
    "html": "https://scrubnet.org/talks/access-logs-and-freshness-signals.html",
    "markdown": "https://scrubnet.org/talks/access-logs-and-freshness-signals.md",
    "text": "https://scrubnet.org/talks/access-logs-and-freshness-signals.txt",
    "json": "https://scrubnet.org/talks/access-logs-and-freshness-signals.json"
  },
  "languageAlternates": {
    "en-GB": "https://scrubnet.org/talks/access-logs-and-freshness-signals.html",
    "fr-FR": "https://scrubnet.org/fr/talks/logs-acces-et-signaux-de-fraicheur.html"
  },
  "content": [
    {
      "type": "section",
      "id": "the-question",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "link",
              "url": "https://scrubnet.org/",
              "content": [
                {
                  "type": "text",
                  "text": "Home"
                }
              ]
            },
            {
              "type": "text",
              "text": " "
            },
            {
              "type": "link",
              "url": "https://scrubnet.org/talks.html",
              "content": [
                {
                  "type": "text",
                  "text": "Talks"
                }
              ]
            },
            {
              "type": "text",
              "text": " Tech SEO North"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "01 / The web is full of bots"
            }
          ]
        },
        {
          "type": "heading",
          "level": 1,
          "id": "question-heading",
          "content": [
            {
              "type": "text",
              "text": "Access logs & freshness signals."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "What bots are really doing."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "We optimise websites for bots."
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "text",
              "text": "But do we actually know what they’re doing?"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "CV"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "strong",
              "content": [
                {
                  "type": "text",
                  "text": "Charles Vayssié"
                }
              ]
            },
            {
              "type": "text",
              "text": " Journey Further & Scrubnet"
            }
          ]
        },
        {
          "type": "aside",
          "label": "Two parts of the talk",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "THIS TALK, IN TWO PARTS"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "01"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Practical client work "
                },
                {
                  "type": "strong",
                  "content": [
                    {
                      "type": "text",
                      "text": "Access logs"
                    }
                  ]
                },
                {
                  "type": "text",
                  "text": " What they record, how to get them and how we use them for technical SEO."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "02"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Observations & open questions "
                },
                {
                  "type": "strong",
                  "content": [
                    {
                      "type": "text",
                      "text": "Freshness signals"
                    }
                  ]
                },
                {
                  "type": "text",
                  "text": " What they mean, how Scrubnet is set up and the questions raised by crawler behaviour."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Tools we use today, then questions we’re still exploring."
                }
              ]
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "24 sections · Access logs, crawler behaviour and freshness signals"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "This talk has two parts: practical access-log analysis for technical SEO, followed by freshness signals and observations from Scrubnet that open up further questions. Read the supporting context below, or use "
            },
            {
              "type": "strong",
              "content": [
                {
                  "type": "text",
                  "text": "Present"
                }
              ]
            },
            {
              "type": "text",
              "text": " to explore the talk as a deck."
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "the-question-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Practical log analysis and open freshness questions"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The first part explains what access logs record, where to find them and how they support technical SEO work already being carried out with clients. Combining request evidence with crawl data and Search Console helps connect crawler activity with the technical state and importance of the pages involved."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The second part introduces freshness signals, Scrubnet’s publishing setup and observations from its Freshness Observatory. It offers patterns to consider and questions to explore about how crawlers respond to content changes. The two parts connect practical tools used today with a subject where there is still more to understand."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "about-charles",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "01 / A quick introduction"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "about-charles-heading",
          "content": [
            {
              "type": "text",
              "text": "Hi, I’m Charles."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Technical SEO specialist, web developer and founder of Scrubnet."
            }
          ]
        },
        {
          "type": "figure",
          "image": {
            "url": "https://scrubnet.org/charles-vayssie-about.webp",
            "alt": "Charles Vayssié, founder of Scrubnet"
          },
          "caption": [
            {
              "type": "text",
              "text": "Charles Vayssié"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Journey Further"
            }
          ]
        },
        {
          "type": "heading",
          "level": 3,
          "content": [
            {
              "type": "text",
              "text": "Technical SEO grounded in evidence."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Working across complex e-commerce and international websites, from technical investigation to practical changes."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Scrubnet"
            }
          ]
        },
        {
          "type": "heading",
          "level": 3,
          "content": [
            {
              "type": "text",
              "text": "A place to build and investigate."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Practical SEO tools and crawler research, including SEO Scrubbox and the Freshness Observatory."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "link",
              "url": "https://www.linkedin.com/in/charles-vayssie/",
              "content": [
                {
                  "type": "text",
                  "text": "linkedin.com/in/charles-vayssie ↗"
                }
              ]
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "about-charles-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Connecting technical SEO with development"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Technical SEO often sits between identifying a problem and making a change that resolves it. My background spans web development, technical SEO and IT infrastructure, which helps me connect what appears in a crawl or search report with the systems delivering the website."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "At Journey Further, that means turning technical evidence into practical decisions across complex e-commerce and international websites. Scrubnet provides a space to build useful tools and investigate questions about crawler behaviour. This talk brings those two perspectives together: applying the evidence to real work and testing the assumptions behind it."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "access-logs-part",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Part 01"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "access-logs-part-heading",
          "content": [
            {
              "type": "text",
              "text": "Access logs"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "What logs record. Where to find them. How we use them."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Technical SEO · Client work · Practical insights"
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "access-logs-part-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Access logs in practical technical SEO"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Access logs record requests observed by a CDN or server. This first part explains what that record contains, where to obtain it and how to establish which crawlers made the requests."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The examples then connect request evidence with crawl data and Search Console to support technical SEO investigations in client work. This provides the practical foundation for the freshness observations in the second part."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "a-web-full-of-bots",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "01 / The web is full of bots"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "a-web-full-of-bots-heading",
          "content": [
            {
              "type": "text",
              "text": "The web has a very large"
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "text",
              "text": "non-human audience."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Cloudflare reported that non-human traffic crossed 50% in 2026."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": ">50%"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "The threshold described in Cloudflare’s"
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "time",
              "datetime": "2026-07-01",
              "content": [
                {
                  "type": "text",
                  "text": "1 July 2026"
                }
              ]
            },
            {
              "type": "text",
              "text": " report."
            }
          ]
        },
        {
          "type": "heading",
          "level": 3,
          "content": [
            {
              "type": "text",
              "text": "Search is only part of the picture."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Training crawlers, assistants, monitoring systems and other automation also make requests."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "This is a wider traffic observation. It does not tell us the bot mix on any particular website."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "link",
              "url": "https://blog.cloudflare.com/agentic-internet-bot-report/",
              "content": [
                {
                  "type": "text",
                  "text": "Cloudflare · July 2026 report"
                }
              ]
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "a-web-full-of-bots-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Putting the traffic statistic in context"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The Cloudflare figure describes the scale of non-human activity across a much wider environment than one website. It is not a benchmark for the proportion of requests a particular site should receive. A site’s audience, content, infrastructure and observation period all affect its recorded traffic mix."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The useful question for technical SEO is therefore more specific than “how much traffic is automated?” Different bots can request the same resource for different purposes. Separating those identities and purposes makes it possible to investigate search crawling, data collection and user-triggered access on their own terms."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "different-bots-different-jobs",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "01 / The web is full of bots"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "different-bots-different-jobs-heading",
          "content": [
            {
              "type": "text",
              "text": "“A bot visited” is not the whole story."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Separate the documented purpose of a crawler from what one request can prove."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Search Training User-triggered"
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Purpose"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "Find and refresh search content"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Googlebot and Bingbot are familiar examples. AI search has its own crawlers too."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Example"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "code",
                  "text": "Googlebot"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Google’s crawler for Google Search. Its user-agent strings contain "
                },
                {
                  "type": "code",
                  "text": "Googlebot/2.1"
                },
                {
                  "type": "text",
                  "text": ". A visit is not a guarantee of indexing."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Purpose"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "Collect material for model development"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Some crawlers have a collection purpose rather than an immediate user query."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Example"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "code",
                  "text": "GPTBot"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "OpenAI says content it crawls may be used for model training. A log entry does not show that training happened."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Purpose"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "Fetch a page for a user action"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "An assistant may request a resource while carrying out a particular task."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Example"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "code",
                  "text": "ChatGPT-User"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "OpenAI distinguishes this agent from automatic web crawling. Keep it separate in your analysis."
                }
              ]
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "link",
              "url": "https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#googlebot",
              "content": [
                {
                  "type": "text",
                  "text": "Google · Googlebot user agents"
                }
              ]
            },
            {
              "type": "text",
              "text": " · "
            },
            {
              "type": "link",
              "url": "https://developers.openai.com/api/docs/bots",
              "content": [
                {
                  "type": "text",
                  "text": "OpenAI · training and user-triggered agents"
                }
              ]
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "different-bots-different-jobs-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Why the crawler’s purpose matters"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Search crawling, training collection and user-triggered fetching describe different activities. A provider can operate several agents, and the presence of one agent does not show that every system belonging to that provider has used the content. The documented purpose of an agent also does not prove what happened after an individual request."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Keeping the raw user-agent string, verified identity and purpose classification as separate information preserves that distinction. It also makes the analysis easier to revise if a provider changes its documentation. The appropriate outcome measure depends on the question: a search revisit, a collection request and a user-directed fetch do not automatically represent the same kind of success."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "the-request",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "02 / Understanding access logs"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "request-heading",
          "content": [
            {
              "type": "text",
              "text": "Start with a single request."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "A log entry records an interaction. Understanding it means knowing what each field tells us."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "access.log · combined-style entry Illustrative"
            }
          ]
        },
        {
          "type": "codeBlock",
          "language": "",
          "text": "192.0.2.24 - -\n[01/Nov/2026:10:32:04 +0000]\n\"GET /discoveries/freshness-signals-vs-crawl-reality.html HTTP/2.0\"\n200 12418 \"-\"\n\"Googlebot/2.1 (+http://www.google.com/bot.html)\""
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Request Time Response Bot identity"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "The request"
            }
          ]
        },
        {
          "type": "heading",
          "level": 3,
          "content": [
            {
              "type": "text",
              "text": "A method and a resource."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "code",
              "text": "GET"
            },
            {
              "type": "text",
              "text": " requests the HTML resource at this path. It tells us which representation was requested, rather than just which topic attracted attention."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "The timestamp"
            }
          ]
        },
        {
          "type": "heading",
          "level": 3,
          "content": [
            {
              "type": "text",
              "text": "Put the request on a timeline."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "The timestamp includes a timezone offset. Align timezones before comparing crawl activity with releases, content updates or incidents."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "The response"
            }
          ]
        },
        {
          "type": "heading",
          "level": 3,
          "content": [
            {
              "type": "text",
              "text": "A status and a transfer size."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "code",
              "text": "200"
            },
            {
              "type": "text",
              "text": " is the recorded response status. In this example’s log format, "
            },
            {
              "type": "code",
              "text": "12418"
            },
            {
              "type": "text",
              "text": " is the response body size in bytes. A successful response does not establish indexing."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "The claimed identity"
            }
          ]
        },
        {
          "type": "heading",
          "level": 3,
          "content": [
            {
              "type": "text",
              "text": "A user agent is a claim."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "This entry claims to be Googlebot. Verification needs a separate check of the source IP against the crawler provider’s documented verification process."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "This is an invented entry using a documentation IP address. It is not evidence of a verified Googlebot visit."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "strong",
              "content": [
                {
                  "type": "text",
                  "text": "Where the log came from matters."
                }
              ]
            },
            {
              "type": "text",
              "text": " An origin server log can miss requests served entirely by a CDN. CDN and server logs may describe different points in the same request journey. Available fields, response sizes and timing measures depend on the logging configuration."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "For Google crawlers, use the "
            },
            {
              "type": "link",
              "url": "https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot",
              "content": [
                {
                  "type": "text",
                  "text": "documented IP-range or DNS verification methods"
                }
              ]
            },
            {
              "type": "text",
              "text": ". Keep the verified identity separate from the raw user-agent string."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "link",
              "url": "https://httpd.apache.org/docs/2.4/logs.html",
              "content": [
                {
                  "type": "text",
                  "text": "Apache access logs"
                }
              ]
            },
            {
              "type": "text",
              "text": " · "
            },
            {
              "type": "link",
              "url": "https://nginx.org/en/docs/http/ngx_http_log_module.html",
              "content": [
                {
                  "type": "text",
                  "text": "NGINX log format"
                }
              ]
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "the-request-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Reading a request in its logging context"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The path identifies the requested representation, while the timestamp places that request on a timeline. The recorded response status and transfer size describe the outcome at the logging layer. The sample is deliberately illustrative: its user agent claims to be Googlebot, but the entry is not evidence of a verified Googlebot visit."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Response timing can add useful context, although it is not included in this example. A timing field is only available if the logging configuration records it, and its meaning needs to be understood before systems are compared. Total request time and upstream response time, for example, describe different parts of processing."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "where-the-log-was-written",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "02 / Understanding access logs"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "where-the-log-was-written-heading",
          "content": [
            {
              "type": "text",
              "text": "Which part of the journey"
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "text",
              "text": "did your log actually see?"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "A CDN and an origin server observe different parts of a request."
            }
          ]
        },
        {
          "type": "list",
          "ordered": true,
          "items": [
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "01"
                    }
                  ]
                },
                {
                  "type": "heading",
                  "level": 3,
                  "content": [
                    {
                      "type": "text",
                      "text": "Crawler"
                    }
                  ]
                },
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "Requests the URL."
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "02"
                    }
                  ]
                },
                {
                  "type": "heading",
                  "level": 3,
                  "content": [
                    {
                      "type": "text",
                      "text": "CDN / edge"
                    }
                  ]
                },
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "May serve a cached response or apply a rule."
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "03"
                    }
                  ]
                },
                {
                  "type": "heading",
                  "level": 3,
                  "content": [
                    {
                      "type": "text",
                      "text": "Origin"
                    }
                  ]
                },
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "Receives the request if it is forwarded."
                    }
                  ]
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Server access logs"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "Apache / NGINX / hosting platform"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Start with the configured access-log destination or your host’s request-log export."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "CDN request logs"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "Cloudflare / Akamai / other edges"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Check the provider’s export, fields, retention and coverage. A CDN cache hit may be absent from the origin log."
                }
              ]
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "link",
              "url": "https://httpd.apache.org/docs/2.4/logs.html",
              "content": [
                {
                  "type": "text",
                  "text": "Apache logging"
                }
              ]
            },
            {
              "type": "text",
              "text": " · "
            },
            {
              "type": "link",
              "url": "https://developers.cloudflare.com/logs/logpush/",
              "content": [
                {
                  "type": "text",
                  "text": "Cloudflare Logpush"
                }
              ]
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "where-the-log-was-written-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Understanding gaps in request coverage"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "A CDN can answer a request from its cache without forwarding it to the origin server. An edge rule can also affect a request before the application sees it. Origin logs may therefore omit interactions that are visible in a CDN export, while providing application or upstream context that an edge log does not contain."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "An apparent absence of crawling needs to be interpreted against that coverage. Retention, sampling, logging filters and timezones can all affect the record. When edge and origin data are combined, a trustworthy request identifier can help connect the two views without counting one forwarded interaction twice."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "verify-the-identity",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "02 / Understanding access logs"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "verify-the-identity-heading",
          "content": [
            {
              "type": "text",
              "text": "A user agent is a claim."
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "text",
              "text": "Verification is a separate step."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "For Google requests, follow Google’s published verification process."
            }
          ]
        },
        {
          "type": "list",
          "ordered": true,
          "items": [
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "01"
                    }
                  ]
                },
                {
                  "type": "heading",
                  "level": 3,
                  "content": [
                    {
                      "type": "text",
                      "text": "Preserve the claim"
                    }
                  ]
                },
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "Keep the original user-agent string."
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "02"
                    }
                  ]
                },
                {
                  "type": "heading",
                  "level": 3,
                  "content": [
                    {
                      "type": "text",
                      "text": "Check the source"
                    }
                  ]
                },
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "Use the provider’s published IP ranges, or its documented reverse-and-forward DNS method."
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "03"
                    }
                  ]
                },
                {
                  "type": "heading",
                  "level": 3,
                  "content": [
                    {
                      "type": "text",
                      "text": "Record the result"
                    }
                  ]
                },
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "Keep verified, unverified and unknown identities distinct."
                    }
                  ]
                }
              ]
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Behind a proxy, first establish which recorded IP is trustworthy. An unmatched request is not automatically a malicious one."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "link",
              "url": "https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests",
              "content": [
                {
                  "type": "text",
                  "text": "Google · verifying crawler requests"
                }
              ]
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "verify-the-identity-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Checking the source behind the claimed identity"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "For Google’s documented DNS verification method, the source IP is reverse-resolved, the resulting hostname is checked against the documented domains, and a forward lookup confirms that it resolves back to the original IP. The published IP ranges for the relevant crawler category provide another verification route. A plausible-looking name alone is not sufficient."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Other providers require their own documented checks. Recording the method and when it was applied helps explain the verification coverage of a dataset, particularly when IP ranges change. A verified identity still does not reveal downstream indexing, retrieval or training decisions."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "combine-the-evidence",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "03 / Turning logs into SEO insights"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "combine-the-evidence-heading",
          "content": [
            {
              "type": "text",
              "text": "Three sources."
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "text",
              "text": "Three different questions."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Once we know who requested the URL, add its technical state and search performance."
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Access logs"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "What was requested?"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Recorded crawler, resource, time, response and, where logged, transfer size or timing."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Screaming Frog"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "What signals does the URL expose?"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "A crawl snapshot of status, canonicals, indexability signals, internal links and other technical details."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Google Search Console"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "What does Google report?"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Search performance, indexing reports and URL-level inspection to add another perspective."
                }
              ]
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "combine-the-evidence-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Joining records from different systems"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Each source answers a different question. A site crawl captures technical signals from a particular crawl configuration and moment. Search Console adds Google-specific reports with their own reporting windows. GA4 describes measured interactions, rather than providing a complete request-level record of crawlers."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "A useful join retains the distinctions that matter: query strings, representations, redirects and canonical relationships. The time window also needs to be comparable. Records that share a URL can still describe different versions of a page or different stages in its history, so matching the address is only the beginning of the analysis."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "the-context",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "03 / Turning logs into SEO insights"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "context-heading",
          "content": [
            {
              "type": "text",
              "text": "Putting it together:"
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "text",
              "text": "Bot Control."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Journey Further’s dashboard approach: move from separate datasets to an investigation."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "strong",
              "content": [
                {
                  "type": "text",
                  "text": "ILLUSTRATIVE JOINED VIEW"
                }
              ]
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "text",
              "text": "Synthetic URLs and values · Example seven-day window"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Investigate Low request activity Server errors"
            }
          ]
        },
        {
          "type": "table",
          "rows": [
            {
              "header": true,
              "cells": [
                [
                  {
                    "type": "text",
                    "text": "Example page"
                  }
                ],
                [
                  {
                    "type": "text",
                    "text": "Indexable¹"
                  }
                ],
                [
                  {
                    "type": "text",
                    "text": "Verified requests²"
                  }
                ],
                [
                  {
                    "type": "text",
                    "text": "5xx²"
                  }
                ],
                [
                  {
                    "type": "text",
                    "text": "Search clicks³"
                  }
                ]
              ]
            },
            {
              "header": true,
              "cells": [
                [
                  {
                    "type": "text",
                    "text": "/guides/log-analysis/"
                  }
                ],
                [
                  {
                    "type": "text",
                    "text": "Yes"
                  }
                ],
                [
                  {
                    "type": "strong",
                    "content": [
                      {
                        "type": "text",
                        "text": "2"
                      }
                    ]
                  }
                ],
                [
                  {
                    "type": "text",
                    "text": "0"
                  }
                ],
                [
                  {
                    "type": "text",
                    "text": "140"
                  }
                ]
              ]
            },
            {
              "header": true,
              "cells": [
                [
                  {
                    "type": "text",
                    "text": "/guides/crawl-signals/"
                  }
                ],
                [
                  {
                    "type": "text",
                    "text": "Yes"
                  }
                ],
                [
                  {
                    "type": "text",
                    "text": "640"
                  }
                ],
                [
                  {
                    "type": "text",
                    "text": "48"
                  }
                ],
                [
                  {
                    "type": "text",
                    "text": "310"
                  }
                ]
              ]
            },
            {
              "header": true,
              "cells": [
                [
                  {
                    "type": "text",
                    "text": "/search/?sort=latest"
                  }
                ],
                [
                  {
                    "type": "text",
                    "text": "No"
                  }
                ],
                [
                  {
                    "type": "text",
                    "text": "1,280"
                  }
                ],
                [
                  {
                    "type": "text",
                    "text": "0"
                  }
                ],
                [
                  {
                    "type": "text",
                    "text": "0"
                  }
                ]
              ]
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "NEXT QUESTION →"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "An indexable page has search traffic but few recorded bot requests. Is that expected for its update frequency and crawl history?"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "¹ Screaming Frog crawl ² Access logs ³ Google Search Console"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "strong",
              "content": [
                {
                  "type": "text",
                  "text": "A reason to investigate, rather than an automatic diagnosis."
                }
              ]
            },
            {
              "type": "text",
              "text": " Low request volume does not establish a crawl problem. High request volume to a non-indexable URL does not automatically mean wasted crawling. Check the purpose of the resource, the observation window, verification coverage and whether logs capture all relevant requests."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "This is a schematic for the talk, not a screenshot of Bot Control or a client result. Its invented URLs and values illustrate the method without representing client data or measured client results."
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "the-context-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "From a dashboard to a useful question"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The value of a joined view is the ability to move from a summary to the evidence behind it. A low request count can be checked against the page’s update history, technical signals and search activity. A concentration of server errors can be examined by affected resource, crawler and time."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The schematic demonstrates that approach using invented data. Its two investigations ask different questions: whether a quiet page is being revisited appropriately, and which requests failed on a page with more activity. Neither pattern is a diagnosis by itself. The underlying requests and the dataset’s coverage determine which explanations are plausible."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "response-patterns",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "03 / Turning logs into SEO insights"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "response-patterns-heading",
          "content": [
            {
              "type": "text",
              "text": "Find the requests other reports"
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "text",
              "text": "cannot show in detail."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "See the affected URL, crawler and timestamp for failures and interrupted requests."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "4xx 5xx 499"
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Find"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "Missing or blocked resources"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Pinpoint "
                },
                {
                  "type": "code",
                  "text": "404"
                },
                {
                  "type": "text",
                  "text": " and "
                },
                {
                  "type": "code",
                  "text": "410"
                },
                {
                  "type": "text",
                  "text": " requests, access blocks such as "
                },
                {
                  "type": "code",
                  "text": "403"
                },
                {
                  "type": "text",
                  "text": ", and throttled requests returning "
                },
                {
                  "type": "code",
                  "text": "429"
                },
                {
                  "type": "text",
                  "text": "."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Connect"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "Which bots hit which URLs?"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Group requests by crawler, URL pattern and time to locate the affected resources and recurring paths."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Find"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "Failures bots actually encountered"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Locate "
                },
                {
                  "type": "code",
                  "text": "500"
                },
                {
                  "type": "text",
                  "text": ", "
                },
                {
                  "type": "code",
                  "text": "502"
                },
                {
                  "type": "text",
                  "text": " and "
                },
                {
                  "type": "code",
                  "text": "503"
                },
                {
                  "type": "text",
                  "text": " responses in recorded crawler requests, including short-lived incidents."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Connect"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "The exact request and time"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Match failures to important pages, origin or edge activity, and logged response times. See whether later requests succeeded."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Find · NGINX logs"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "Requests that ended early"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "code",
                  "text": "499"
                },
                {
                  "type": "text",
                  "text": " records a client disconnect before NGINX sends response headers. Logs reveal the URL, time and recorded client."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Extra visibility"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "A detail Search Console cannot label as 499"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "This is a local logging code, not an HTTP response sent to Googlebot. Search Console may record a fetch problem, but does not expose your NGINX 499 log entry."
                }
              ]
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "link",
              "url": "https://developers.google.com/crawling/docs/troubleshooting/http-status-codes",
              "content": [
                {
                  "type": "text",
                  "text": "Google · HTTP status handling"
                }
              ]
            },
            {
              "type": "text",
              "text": " · "
            },
            {
              "type": "link",
              "url": "https://github.com/nginx/nginx/blob/master/src/http/ngx_http_request.h",
              "content": [
                {
                  "type": "text",
                  "text": "NGINX · 499 definition"
                }
              ]
            },
            {
              "type": "text",
              "text": " · "
            },
            {
              "type": "link",
              "url": "https://support.google.com/webmasters/answer/9679690?hl=en",
              "content": [
                {
                  "type": "text",
                  "text": "Search Console · Crawl Stats"
                }
              ]
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "response-patterns-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Request-level detail for technical investigation"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Access logs can identify the particular resource, recorded client and time associated with a failure. This makes short-lived incidents and recurring URL patterns visible across verified search crawlers and other bots, subject to the coverage of the logging layer."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "NGINX introduces 499 to record a client closing the connection before response headers are sent. Google does not receive a 499 response in that situation, so Search Console cannot report that same local event as a received HTTP 499. It may show an associated connection or fetch problem instead. The 499 entry is also not an indexing status: request and upstream timing data can help investigate why the connection ended."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "freshness-part",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Part 02"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "freshness-part-heading",
          "content": [
            {
              "type": "text",
              "text": "Freshness signals"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "What we communicate. What we observe. What remains unclear."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Scrubnet · Crawler observations · Open questions"
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "freshness-part-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "From practical investigations to open questions"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The first part showed how access logs support technical SEO work with clients. The same request evidence can also help us explore a more open question: what happens after content changes and we communicate that update to crawlers?"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "This second part introduces freshness signals and Scrubnet’s publishing setup, then follows the crawler observations. It connects what a website communicates with the requests that can be observed and the questions those patterns leave open."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "scrubnet-feeds",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "04 / Freshness signals: are bots listening?"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "scrubnet-feeds-heading",
          "content": [
            {
              "type": "text",
              "text": "What happens after"
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "text",
              "text": "we update content?"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Access logs help us find technical issues. On Scrubnet, I also use them to observe what follows a content update."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Authorised content, published in different representations for readers and automated systems."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "strong",
              "content": [
                {
                  "type": "text",
                  "text": "HTML"
                }
              ]
            },
            {
              "type": "text",
              "text": " Web page"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "strong",
              "content": [
                {
                  "type": "text",
                  "text": "MD"
                }
              ]
            },
            {
              "type": "text",
              "text": " Markdown"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "strong",
              "content": [
                {
                  "type": "text",
                  "text": "TXT"
                }
              ]
            },
            {
              "type": "text",
              "text": " Plain text"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "strong",
              "content": [
                {
                  "type": "text",
                  "text": "JSON"
                }
              ]
            },
            {
              "type": "text",
              "text": " Structured data"
            }
          ]
        },
        {
          "type": "heading",
          "level": 3,
          "content": [
            {
              "type": "text",
              "text": "From finding issues to understanding behaviour."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "We update the content and communicate the change. Which crawlers return, when, and for which format?"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "link",
              "url": "https://scrubnet.org/discoveries/freshness-signals-vs-crawl-reality.html",
              "content": [
                {
                  "type": "text",
                  "text": "Scrubnet · research context"
                }
              ]
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "scrubnet-feeds-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Following requests after a content update"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The practical investigations above use logs to identify technical issues. Scrubnet applies that same request evidence to another question: what happens after content is updated and the change is communicated? Its public feeds make authorised content available as HTML web pages, Markdown, plain text and structured JSON. These are different ways of representing content for readers and automated systems, rather than four kinds of crawler."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The representations differ in markup, metadata and payload size. Keeping track of what was available and when it changed gives the requests context. I’m documenting which crawlers return, which representations they request and how the patterns develop, then sharing those observations as the dataset grows."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "freshness-signals",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "04 / Freshness signals: are bots listening?"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "freshness-signals-heading",
          "content": [
            {
              "type": "text",
              "text": "Freshness signals do"
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "text",
              "text": "different jobs."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Publication, editorial updates, page updates and HTTP versions describe different events."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Sitemap Editorial update Last-Modified ETag Published / visible dates"
            }
          ]
        },
        {
          "type": "codeBlock",
          "language": "",
          "text": "<lastmod>2026-11-01</lastmod>"
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Significant page update"
            }
          ]
        },
        {
          "type": "heading",
          "level": 3,
          "content": [
            {
              "type": "text",
              "text": "When did this URL meaningfully change?"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Sitemap "
            },
            {
              "type": "code",
              "text": "lastmod"
            },
            {
              "type": "text",
              "text": " can reflect significant changes to content, links or structured data. It is not simply the time the sitemap was generated."
            }
          ]
        },
        {
          "type": "codeBlock",
          "language": "",
          "text": "\"dateModified\":\n\"2026-11-01T10:00:00Z\""
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Editorial content"
            }
          ]
        },
        {
          "type": "heading",
          "level": 3,
          "content": [
            {
              "type": "text",
              "text": "When was this content updated?"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Structured data can describe the article or another entity on the page. Compare dates for the same item; sitemap lastmod describes the URL as a whole."
            }
          ]
        },
        {
          "type": "codeBlock",
          "language": "",
          "text": "Last-Modified:\nSun, 08 Nov 2026 10:00:00 GMT"
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "HTTP representation update"
            }
          ]
        },
        {
          "type": "heading",
          "level": 3,
          "content": [
            {
              "type": "text",
              "text": "When did the served resource change?"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "A regenerated or redeployed HTML file may have a newer HTTP date without an editorial update. Newer than "
            },
            {
              "type": "code",
              "text": "lastmod"
            },
            {
              "type": "text",
              "text": " or "
            },
            {
              "type": "code",
              "text": "dateModified"
            },
            {
              "type": "text",
              "text": " is not automatically a conflict."
            }
          ]
        },
        {
          "type": "codeBlock",
          "language": "",
          "text": "ETag: \"feed-html-v7\""
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "HTTP representation validator"
            }
          ]
        },
        {
          "type": "heading",
          "level": 3,
          "content": [
            {
              "type": "text",
              "text": "Does this version still match?"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "An ETag identifies a representation version, not a publication date. A client can send "
            },
            {
              "type": "code",
              "text": "If-None-Match"
            },
            {
              "type": "text",
              "text": " to validate its cached copy."
            }
          ]
        },
        {
          "type": "codeBlock",
          "language": "",
          "text": "\"datePublished\": \"2026-07-20\"\n\nPublished: 20 July 2026\nUpdated: 1 November 2026"
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Publication and reader context"
            }
          ]
        },
        {
          "type": "heading",
          "level": 3,
          "content": [
            {
              "type": "text",
              "text": "Created, then meaningfully updated."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Publication and editorial update are separate events. Visible dates should make that distinction clear and agree with the content they describe."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "link",
              "url": "https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap",
              "content": [
                {
                  "type": "text",
                  "text": "Google · sitemap lastmod"
                }
              ]
            },
            {
              "type": "text",
              "text": " · "
            },
            {
              "type": "link",
              "url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Last-Modified",
              "content": [
                {
                  "type": "text",
                  "text": "HTTP · Last-Modified"
                }
              ]
            },
            {
              "type": "text",
              "text": " · "
            },
            {
              "type": "link",
              "url": "https://developers.google.com/search/docs/appearance/publication-dates",
              "content": [
                {
                  "type": "text",
                  "text": "Google · page dates"
                }
              ]
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "freshness-signals-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Consistent does not mean identical"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "A publication date describes creation; an editorial modification date describes an update to the relevant content or entity. A sitemap lastmod describes the last significant update to the URL, which can include meaningful changes to links or structured data as well as the main text. An entity within a page and the page as a whole can therefore have different update dates."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "HTTP Last-Modified describes the selected representation according to the serving system. For a static file, a rebuild or redeployment may advance that timestamp even though the article has not been edited. In the examples, the editorial and sitemap dates are 1 November while the HTTP date is 8 November. Those values can be valid together; check how the header is generated rather than requiring all dates to match."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "ETag is a separate version validator and need not contain a date. Useful consistency means each signal describes the event and scope it claims to represent. It does not mean resetting publication or editorial dates whenever a file is deployed. This distinction also informs the Freshness Signals checks in SEO Scrubbox."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "conditional-requests",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "04 / Freshness signals: are bots listening?"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "conditional-requests-heading",
          "content": [
            {
              "type": "text",
              "text": "A request can save a download."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "A conditional GET can still appear in the access log even when no content body is returned."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "1 · First fetch 2 · Unchanged 3 · Changed"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Client → server"
            }
          ]
        },
        {
          "type": "codeBlock",
          "language": "",
          "text": "GET /resource.html"
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Server → client"
            }
          ]
        },
        {
          "type": "codeBlock",
          "language": "",
          "text": "200 OK\nETag: \"v1\"\n\n[content body]"
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "The client stores the representation and its validator."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Client → server"
            }
          ]
        },
        {
          "type": "codeBlock",
          "language": "",
          "text": "GET /resource.html\nIf-None-Match: \"v1\""
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Server → client"
            }
          ]
        },
        {
          "type": "codeBlock",
          "language": "",
          "text": "304 Not Modified\nETag: \"v1\"\n\n[no content body]"
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "There was still a request. The existing body can be reused."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Client → server"
            }
          ]
        },
        {
          "type": "codeBlock",
          "language": "",
          "text": "GET /resource.html\nIf-None-Match: \"v1\""
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Server → client"
            }
          ]
        },
        {
          "type": "codeBlock",
          "language": "",
          "text": "200 OK\nETag: \"v2\"\n\n[updated content body]"
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "The validator no longer matches, so the updated representation is sent."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "link",
              "url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Conditional_requests",
              "content": [
                {
                  "type": "text",
                  "text": "HTTP conditional requests"
                }
              ]
            },
            {
              "type": "text",
              "text": " · "
            },
            {
              "type": "link",
              "url": "https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers",
              "content": [
                {
                  "type": "text",
                  "text": "Google crawler caching support"
                }
              ]
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "conditional-requests-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Why a 304 still counts as a request"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "A client can keep a representation and its validator after an initial download. On a later request, the validator allows the server to determine whether that stored representation still matches. An unchanged resource can produce a 304 response without a content body; a changed resource can produce a new body and validator."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The validation still involves a request. A request-count chart therefore cannot show all the effects of caching. Response statuses, transfer sizes and conditional request headers, where recorded, help distinguish revisits that transfer a new body from those that reuse an existing one. The sequence illustrates HTTP behaviour without assuming that every crawler uses it on every visit."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "freshness-observatory",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "05 / What I’ve started discovering"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "freshness-observatory-heading",
          "content": [
            {
              "type": "text",
              "text": "Observe the request."
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "text",
              "text": "Keep the context."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "The Freshness Observatory connects crawler, format and request time with the recorded age of the resource."
            }
          ]
        },
        {
          "type": "figure",
          "image": {
            "url": "https://scrubnet.org/discoveries/amazon-bot-crawl-data.webp",
            "alt": "Historical Amazonbot Observatory screenshot, showing different request volumes and content ages across format lanes"
          },
          "caption": [
            {
              "type": "text",
              "text": "Published Amazonbot screenshot · its own filtered view."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Illustrative age calculation · all times UTC"
            }
          ]
        },
        {
          "type": "list",
          "ordered": true,
          "items": [
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "Recorded update "
                    },
                    {
                      "type": "strong",
                      "content": [
                        {
                          "type": "time",
                          "datetime": "2026-11-01T10:00:00Z",
                          "content": [
                            {
                              "type": "text",
                              "text": "1 Nov · 10:00"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "Crawler request "
                    },
                    {
                      "type": "strong",
                      "content": [
                        {
                          "type": "time",
                          "datetime": "2026-11-04T10:00:00Z",
                          "content": [
                            {
                              "type": "text",
                              "text": "4 Nov · 10:00"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "Age at request "
                    },
                    {
                      "type": "strong",
                      "content": [
                        {
                          "type": "text",
                          "text": "3 days old"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "strong",
              "content": [
                {
                  "type": "text",
                  "text": "Date source:"
                }
              ]
            },
            {
              "type": "text",
              "text": " feed JSON "
            },
            {
              "type": "code",
              "text": "dateModified"
            },
            {
              "type": "text",
              "text": ", with a file-timestamp fallback."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Request time − stored resource modification time."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "The published study covers verified requests with a usable age, across HTML, Markdown, TXT and JSON. It does not measure indexing or training."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "link",
              "url": "https://scrubnet.org/discoveries/freshness-signals-vs-crawl-reality.html",
              "content": [
                {
                  "type": "text",
                  "text": "Method and published observations"
                }
              ]
            },
            {
              "type": "text",
              "text": " · "
            },
            {
              "type": "link",
              "url": "https://scrubnet.org/freshness-lab.php",
              "content": [
                {
                  "type": "text",
                  "text": "Open the public Observatory"
                }
              ]
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "freshness-observatory-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Defining what the dataset represents"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The screenshot shows the Freshness Observatory filtered to Amazonbot, as published in the historical study. Its request totals and age summary belong to that selection and are separate from the all-crawler 88% figure discussed later."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The example uses invented dates to show the calculation: an update recorded at 10:00 UTC on 1 November and a request at 10:00 UTC on 4 November produce an age of three days. Age is calculated at the time of the request, not when the dashboard is opened. Format and verified crawler identity provide additional ways to split the activity into comparable groups."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The aggregation uses the resource modification timestamp stored with the request, in the "
                },
                {
                  "type": "code",
                  "text": "last_mod"
                },
                {
                  "type": "text",
                  "text": " field. The collector reads "
                },
                {
                  "type": "code",
                  "text": "dateModified"
                },
                {
                  "type": "text",
                  "text": " from the JSON resource, or the matching JSON representation for HTML, Markdown and TXT, and falls back to the file’s modification timestamp when that value is unavailable. This is a recorded resource-age measure; the fallback is not necessarily an editorial update. Historical rows retain their stored date, rather than acquiring a new age from today’s version of the file."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The published analysis focuses on four formats and excludes records without a usable content age. Those eligibility rules are part of the result. Changing the period, crawler selection or aggregation can change the summary, so a finding needs its filters and observation window alongside it."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "the-finding",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "05 / What I’ve started discovering"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "finding-heading",
          "content": [
            {
              "type": "text",
              "text": "Older content still received"
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "text",
              "text": "most of the requests."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "A historical observation from the first 28 days reported by Scrubnet’s Freshness Observatory."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "88%"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "of observed requests targeted content more than seven days old."
            }
          ]
        },
        {
          "type": "list",
          "ordered": false,
          "items": [
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "No more than one day old "
                    },
                    {
                      "type": "strong",
                      "content": [
                        {
                          "type": "text",
                          "text": "4%"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "More than one, up to seven days old "
                    },
                    {
                      "type": "strong",
                      "content": [
                        {
                          "type": "text",
                          "text": "8%"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "More than seven days old "
                    },
                    {
                      "type": "strong",
                      "content": [
                        {
                          "type": "text",
                          "text": "88%"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        },
        {
          "type": "heading",
          "level": 3,
          "content": [
            {
              "type": "text",
              "text": "What might explain this?"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Possible explanations"
            }
          ]
        },
        {
          "type": "list",
          "ordered": false,
          "items": [
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "strong",
                      "content": [
                        {
                          "type": "text",
                          "text": "Revisit schedules:"
                        }
                      ]
                    },
                    {
                      "type": "text",
                      "text": " bots may be returning to familiar resources on established cycles."
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "strong",
                      "content": [
                        {
                          "type": "text",
                          "text": "The available content:"
                        }
                      ]
                    },
                    {
                      "type": "text",
                      "text": " older URLs may make up most of the resources they can request."
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "strong",
                      "content": [
                        {
                          "type": "text",
                          "text": "Other priorities:"
                        }
                      ]
                    },
                    {
                      "type": "text",
                      "text": " discovery paths, links and crawler-specific goals may matter alongside age."
                    }
                  ]
                }
              ]
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Several factors could contribute to the same request mix."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "strong",
              "content": [
                {
                  "type": "text",
                  "text": "Historical snapshot, not a live dashboard."
                }
              ]
            },
            {
              "type": "text",
              "text": " These approximate percentages come from "
            },
            {
              "type": "link",
              "url": "https://scrubnet.org/discoveries/freshness-signals-vs-crawl-reality.html",
              "content": [
                {
                  "type": "text",
                  "text": "Freshness Signals vs Crawl Reality"
                }
              ]
            },
            {
              "type": "text",
              "text": ", published on "
            },
            {
              "type": "time",
              "datetime": "2026-07-20",
              "content": [
                {
                  "type": "text",
                  "text": "20 July 2026"
                }
              ]
            },
            {
              "type": "text",
              "text": ". The article describes an initial 28-day observation window; its publication date is not an exact start or end date for that window."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Interpreting the request mix requires the age distribution of eligible resources, the recorded definition of content age and other influences on crawling. The observation alone cannot establish indexing, ranking, AI training use or the causal effect of a particular signal. The figures refer to that historical observation window, rather than the current state of the live tool."
            }
          ]
        },
        {
          "type": "aside",
          "label": "Sources and further reading",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "strong",
                  "content": [
                    {
                      "type": "text",
                      "text": "Sources & further reading"
                    }
                  ]
                }
              ]
            },
            {
              "type": "list",
              "ordered": false,
              "items": [
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "link",
                          "url": "https://scrubnet.org/discoveries/freshness-signals-vs-crawl-reality.html",
                          "content": [
                            {
                              "type": "text",
                              "text": "Scrubnet: Freshness Signals vs Crawl Reality"
                            }
                          ]
                        },
                        {
                          "type": "text",
                          "text": " — the historical findings and their limitations."
                        }
                      ]
                    }
                  ]
                },
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "link",
                          "url": "https://scrubnet.org/freshness-lab.php",
                          "content": [
                            {
                              "type": "text",
                              "text": "Scrubnet Freshness Observatory"
                            }
                          ]
                        },
                        {
                          "type": "text",
                          "text": " — explore the current public tool separately."
                        }
                      ]
                    }
                  ]
                },
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "link",
                          "url": "https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot",
                          "content": [
                            {
                              "type": "text",
                              "text": "Google Search Central: Verifying Googlebot and other Google crawlers"
                            }
                          ]
                        },
                        {
                          "type": "text",
                          "text": "."
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "the-finding-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "The denominator behind the 88% figure"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The 88% figure describes the age of requested content, rather than the proportion of eligible older pages selected. The published study also reported an average requested content age of approximately 120 days. That average does not reveal the full distribution or the mix of resources available to crawlers."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "If most eligible resources were already old, a large share of requests to older content could be expected. Comparing the request mix with the resource inventory is therefore essential. This observation alone does not establish that freshness signals were ignored or that older content was more valuable to a crawler."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The possible explanations shown above are hypotheses. Established revisit cycles could lead to recurring requests for older resources; an older resource inventory could explain much of the distribution; and discovery, linking or crawler-specific priorities could affect which resources are selected. The percentages alone do not distinguish between these explanations."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "crawler-patterns",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "05 / What I’ve started discovering"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "crawler-patterns-heading",
          "content": [
            {
              "type": "text",
              "text": "The crawlers did not"
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "text",
              "text": "follow one shared pattern."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Historical observations from the published 28-day Scrubnet study."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Google Bing Amazon Meta Claude"
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Recorded formats"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "All four formats"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Googlebot and GoogleOther both requested HTML, Markdown, TXT and JSON."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Request timing"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "A coinciding TXT spike"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Both crawlers produced a large spike in TXT requests at approximately the same time."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Recorded formats"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "Mostly HTML"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "HTML received most of Bingbot’s requests in the selected dataset."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Markdown activity"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "Very few requests"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Very little Markdown activity was recorded during the observed period."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Markdown activity"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "Approximately twice the requests"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Amazonbot requested Markdown roughly twice as often as either HTML or TXT."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "JSON activity"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "No requests observed"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "No Amazonbot requests to JSON resources were recorded in this dataset."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Markdown activity"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "Concentrated bursts"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Meta’s external crawler requested groups of Markdown resources in large spikes."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Other formats"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "More spread over time"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Its HTML, TXT and JSON requests were comparatively distributed across the observed period."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Request timing"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "A low baseline, then large spikes"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "ClaudeBot maintained relatively low activity for much of the period, interrupted by occasional large crawl spikes."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Recorded formats"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "All four formats"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "HTML, Markdown, TXT and JSON were requested, with HTML receiving the highest overall request volume."
                }
              ]
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "link",
              "url": "https://scrubnet.org/discoveries/freshness-signals-vs-crawl-reality.html",
              "content": [
                {
                  "type": "text",
                  "text": "Scrubnet · crawler-by-crawler findings"
                }
              ]
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "crawler-patterns-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Interpreting differences in request patterns"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The patterns summarised here come from Scrubnet’s published first study, rather than a current ranking of crawlers. A concentrated burst, a quiet format or two events occurring at a similar time can suggest a useful question. None exposes the external system’s internal schedule or purpose by itself."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Available resources, discovery paths and the selected window can all shape the distribution. Those factors provide context for interpreting an observation on one host. Longer observation periods and comparable datasets can help assess whether a pattern persists."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The coinciding Google TXT events do not establish coordination, and Bing’s low Markdown activity is not a capability test. Amazonbot’s distribution does not establish a universal format preference. The Meta and Claude patterns describe visible activity without revealing internal scheduling systems or a repeating cycle."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "observation-scope",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "05 / What I’ve started discovering"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "observation-scope-heading",
          "content": [
            {
              "type": "text",
              "text": "An observatory,"
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "text",
              "text": "not a controlled experiment."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Scrubnet’s current setup is built for observation. It does not yet follow a controlled experimentation protocol."
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Observed setting"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "One site, one setup"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Scrubnet’s feeds, resource mix and selected 28-day window. Another website may show different behaviour."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Study design"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "No controlled comparison"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Formats and resources were not randomised treatments. Multiple factors can vary together, so this setup does not isolate causes."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "How to use it"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "Questions to explore"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Use these patterns as food for thought. They are not yet general recommendations for crawling, formats or freshness on other sites."
                }
              ]
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Keep each observation tied to its website, resources and time window."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "link",
              "url": "https://scrubnet.org/discoveries/freshness-signals-vs-crawl-reality.html",
              "content": [
                {
                  "type": "text",
                  "text": "Scrubnet · study scope and method"
                }
              ]
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "observation-scope-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "What the observations can support"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The published study describes crawler requests within one Scrubnet environment and a historical observation window. It is not a controlled experiment with randomised treatments, matched resource groups and the isolation of one freshness signal. Differences in the available resources, discovery paths, linking, formats and crawler mix may contribute to the patterns."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The observations can document what happened and suggest possible explanations. They do not yet establish a causal effect or a change that should be applied to any website. Comparable evidence from other settings would be needed to assess how far a pattern extends beyond this host. Scrubnet’s role here is to keep observing and sharing those patterns, without presenting them as universal rules."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "takeaways-part",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Conclusion"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "takeaways-part-heading",
          "content": [
            {
              "type": "text",
              "text": "Takeaways"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "What to use today. What to keep observing."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Access logs · Joined data · Freshness signals"
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "takeaways-part-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Bringing the two parts together"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The conclusions connect practical log analysis with the open freshness questions. The next sections summarise how to use request evidence alongside other SEO data and what the Scrubnet observations suggest is worth continuing to observe."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "takeaways",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "06 / There is still so much to discover"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "takeaways-heading",
          "content": [
            {
              "type": "text",
              "text": "Three ways to get closer"
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "text",
              "text": "to bot activity."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Use access logs to connect crawler requests with the pages that matter."
            }
          ]
        },
        {
          "type": "list",
          "ordered": true,
          "items": [
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "01"
                    }
                  ]
                },
                {
                  "type": "heading",
                  "level": 3,
                  "content": [
                    {
                      "type": "text",
                      "text": "Look at your logs"
                    }
                  ]
                },
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "CDN or server: see which bots request your URLs, when, and what happens."
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "02"
                    }
                  ]
                },
                {
                  "type": "heading",
                  "level": 3,
                  "content": [
                    {
                      "type": "text",
                      "text": "Add the page context"
                    }
                  ]
                },
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "Join Search Console performance with regular crawls, indexability signals and internal links."
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "03"
                    }
                  ]
                },
                {
                  "type": "heading",
                  "level": 3,
                  "content": [
                    {
                      "type": "text",
                      "text": "Investigate sooner"
                    }
                  ]
                },
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "As logs arrive, spot request patterns and failures before they appear in delayed page reports."
                    }
                  ]
                }
              ]
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "A more detailed, timely view of bot interactions — with search performance alongside it."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "link",
              "url": "https://support.google.com/webmasters/answer/9679690?hl=en",
              "content": [
                {
                  "type": "text",
                  "text": "Search Console · Crawl Stats coverage"
                }
              ]
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "takeaways-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Combine timely requests with useful page context"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "CDN or server access logs expose the requests recorded at that layer, including crawler identity, URL, timestamp and outcome. Joined with a regular technical crawl and Search Console performance, those observations can help prioritise activity affecting important pages."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The request record can become available before a corresponding issue appears in a delayed Search Console page report. How quickly depends on the provider, export and processing pipeline. This is visibility into recorded requests rather than a live index-status feed, and technical crawl snapshots and search reports retain their own timestamps and reporting windows."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "freshness-takeaway",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "06 / There is still so much to discover"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "freshness-takeaway-heading",
          "content": [
            {
              "type": "text",
              "text": "Accurate signals."
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "text",
              "text": "A wider crawling picture."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Keep freshness information meaningful. Its influence on revisit timing remains difficult to isolate."
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "What we can do"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "Describe changes accurately"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Keep publication, editorial, sitemap and HTTP signals consistent with their roles. The dates do not all need to match."
                }
              ]
            }
          ]
        },
        {
          "type": "article",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "What remains unclear"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "content": [
                {
                  "type": "text",
                  "text": "How much each signal changes crawling"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Discovery, linking, server behaviour and crawler priorities form a wider picture. These observations do not isolate the effect of a freshness signal."
                }
              ]
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "I’m documenting what happens on Scrubnet and sharing those observations as the dataset develops."
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "freshness-takeaway-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Freshness information within a wider system"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Accurate freshness information helps describe content and supports useful HTTP validation. It does not guarantee a particular revisit time. Discovery paths, internal linking, the available resource mix, serving conditions and crawler decisions all provide context for the requests that appear in logs."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The Scrubnet results shown here are observations from one site and a historical window. They do not isolate how much an individual freshness signal influenced crawling. Continued observation can reveal patterns and improve the questions being asked."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "keep-discovering",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "06 / There is still so much to discover"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "keep-discovering-heading",
          "content": [
            {
              "type": "text",
              "text": "There is still so much"
            },
            {
              "type": "lineBreak"
            },
            {
              "type": "text",
              "text": "to discover."
            }
          ]
        },
        {
          "type": "quote",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "“The more we observe how bots behave, the more we realise how much there is still to learn.”"
                }
              ]
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "strong",
              "content": [
                {
                  "type": "text",
                  "text": "Questions or observations?"
                }
              ]
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Charles Vayssié · Journey Further & Scrubnet"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "link",
              "url": "https://scrubnet.org/talks/access-logs-and-freshness-signals.html",
              "content": [
                {
                  "type": "text",
                  "text": "scrubnet.org/talks/access-logs-and-freshness-signals.html ↗"
                }
              ]
            }
          ]
        },
        {
          "type": "heading",
          "level": 3,
          "content": [
            {
              "type": "text",
              "text": "Sources & further reading"
            }
          ]
        },
        {
          "type": "list",
          "ordered": false,
          "items": [
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "link",
                      "url": "https://blog.cloudflare.com/agentic-internet-bot-report/",
                      "content": [
                        {
                          "type": "text",
                          "text": "Cloudflare · Content Independence Day, one year on (1 July 2026)"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "link",
                      "url": "https://developers.openai.com/api/docs/bots",
                      "content": [
                        {
                          "type": "text",
                          "text": "OpenAI · crawler and user-agent purposes"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "link",
                      "url": "https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests",
                      "content": [
                        {
                          "type": "text",
                          "text": "Google · verifying crawler requests"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "link",
                      "url": "https://httpd.apache.org/docs/2.4/logs.html",
                      "content": [
                        {
                          "type": "text",
                          "text": "Apache · log files"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "link",
                      "url": "https://nginx.org/en/docs/http/ngx_http_log_module.html",
                      "content": [
                        {
                          "type": "text",
                          "text": "NGINX · access-log configuration"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "link",
                      "url": "https://developers.cloudflare.com/logs/logpush/",
                      "content": [
                        {
                          "type": "text",
                          "text": "Cloudflare · request-log exports"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "link",
                      "url": "https://developers.google.com/crawling/docs/crawl-budget",
                      "content": [
                        {
                          "type": "text",
                          "text": "Google · crawl-budget context"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "link",
                      "url": "https://developers.google.com/crawling/docs/troubleshooting/http-status-codes",
                      "content": [
                        {
                          "type": "text",
                          "text": "Google · HTTP status handling"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "link",
                      "url": "https://github.com/nginx/nginx/blob/master/src/http/ngx_http_request.h",
                      "content": [
                        {
                          "type": "text",
                          "text": "NGINX · non-standard 499 definition"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "link",
                      "url": "https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap",
                      "content": [
                        {
                          "type": "text",
                          "text": "Google · accurate sitemap lastmod"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "link",
                      "url": "https://developers.google.com/search/docs/appearance/publication-dates",
                      "content": [
                        {
                          "type": "text",
                          "text": "Google · page dates"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "link",
                      "url": "https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers",
                      "content": [
                        {
                          "type": "text",
                          "text": "Google · crawler caching support"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "link",
                      "url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Conditional_requests",
                      "content": [
                        {
                          "type": "text",
                          "text": "MDN · conditional requests"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "link",
                      "url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Last-Modified",
                      "content": [
                        {
                          "type": "text",
                          "text": "MDN · HTTP Last-Modified"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "link",
                      "url": "https://support.google.com/webmasters/answer/9679690?hl=en",
                      "content": [
                        {
                          "type": "text",
                          "text": "Search Console · Crawl Stats coverage"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "link",
                      "url": "https://scrubnet.org/discoveries/freshness-signals-vs-crawl-reality.html",
                      "content": [
                        {
                          "type": "text",
                          "text": "Scrubnet · historical 28-day findings and screenshot"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "link",
                      "url": "https://scrubnet.org/freshness-lab.php",
                      "content": [
                        {
                          "type": "text",
                          "text": "Scrubnet · public Freshness Observatory"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "External technical sources checked on "
            },
            {
              "type": "time",
              "datetime": "2026-11-01",
              "content": [
                {
                  "type": "text",
                  "text": "1 November 2026"
                }
              ]
            },
            {
              "type": "text",
              "text": "; sitemap, HTTP freshness and request-reporting references reviewed again on "
            },
            {
              "type": "time",
              "datetime": "2026-11-11",
              "content": [
                {
                  "type": "text",
                  "text": "11 November 2026"
                }
              ]
            },
            {
              "type": "text",
              "text": ". Scrubnet figures are historical, from the article published on 20 July 2026. Worked Bot Control examples are illustrative and contain no client data."
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "keep-discovering-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "The questions that remain"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Crawler behaviour sits at the intersection of documented capabilities, visible requests and decisions that remain outside the website owner’s view. Access logs make some of that behaviour inspectable, while leaving important questions about scheduling and downstream use unresolved."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Comparing observations and improving log coverage can make those questions clearer. Keeping track of the available resources and their ages adds context to the requests. Scrubnet remains a place to observe crawler behaviour and share what becomes visible. The aim is a stronger understanding of what happened and which conclusions the evidence can support."
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "id": "thank-you",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "06 / Keep in touch"
            }
          ]
        },
        {
          "type": "heading",
          "level": 2,
          "id": "thank-you-heading",
          "content": [
            {
              "type": "text",
              "text": "Thank you."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Questions or observations? Let’s keep the conversation going."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Website "
            },
            {
              "type": "link",
              "url": "https://scrubnet.org/",
              "content": [
                {
                  "type": "text",
                  "text": "scrubnet.org ↗"
                }
              ]
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Connect on LinkedIn "
            },
            {
              "type": "link",
              "url": "https://www.linkedin.com/in/charles-vayssie/",
              "content": [
                {
                  "type": "text",
                  "text": "linkedin.com/in/charles-vayssie ↗"
                }
              ]
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "My Chrome extension"
                }
              ]
            },
            {
              "type": "heading",
              "level": 3,
              "id": "scrubbox-promo-heading",
              "content": [
                {
                  "type": "text",
                  "text": "SEO Scrubbox"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Check indexability, compare source and rendered SEO signals, and explore a site with visual crawling."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "link",
                  "url": "https://scrubnet.org/seo-scrubbox.html",
                  "content": [
                    {
                      "type": "text",
                      "text": "Explore SEO Scrubbox ↗"
                    }
                  ]
                },
                {
                  "type": "text",
                  "text": " "
                },
                {
                  "type": "link",
                  "url": "https://chromewebstore.google.com/detail/seo-scrubbox/bpmnoonigkfiidagbibkbfcoikiclmic",
                  "content": [
                    {
                      "type": "text",
                      "text": "Get it on Chrome ↗"
                    }
                  ]
                }
              ]
            }
          ]
        },
        {
          "type": "figure",
          "image": null,
          "caption": [
            {
              "type": "strong",
              "content": [
                {
                  "type": "text",
                  "text": "Scan to explore Scrubnet"
                }
              ]
            },
            {
              "type": "text",
              "text": " Tools, articles and crawler research."
            }
          ]
        },
        {
          "type": "aside",
          "content": [
            {
              "type": "heading",
              "level": 3,
              "id": "thank-you-context-heading",
              "content": [
                {
                  "type": "text",
                  "text": "Continue the conversation"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Scrubnet brings together technical SEO tools, articles and public crawler research. The website and LinkedIn links provide ways to explore the work and share related observations and questions."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "For practical browser-based checks, SEO Scrubbox includes indexability inspection, source-versus-rendered signal comparison and visual crawling. Its product page explains the features, and the Chrome Web Store link opens the existing extension listing."
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "structuredData": [
    {
      "@context": "https://schema.org",
      "@graph": [
        {
          "@type": "WebPage",
          "@id": "https://scrubnet.org/talks/access-logs-and-freshness-signals.html#webpage",
          "url": "https://scrubnet.org/talks/access-logs-and-freshness-signals.html",
          "name": "Access Logs & Freshness Signals: What Bots Are Really Doing",
          "description": "Tech SEO North presentation by Charles Vayssié: using access logs, crawl data and Search Console to investigate bot activity and freshness signals.",
          "isPartOf": {
            "@id": "https://scrubnet.org/#website"
          },
          "author": {
            "@id": "https://scrubnet.org/#founder"
          },
          "publisher": {
            "@id": "https://scrubnet.org/#org"
          },
          "breadcrumb": {
            "@id": "https://scrubnet.org/talks/access-logs-and-freshness-signals.html#breadcrumbs"
          },
          "primaryImageOfPage": {
            "@type": "ImageObject",
            "url": "https://scrubnet.org/scrubberduck.webp",
            "width": 1024,
            "height": 1024
          },
          "inLanguage": "en-GB"
        },
        {
          "@type": "BreadcrumbList",
          "@id": "https://scrubnet.org/talks/access-logs-and-freshness-signals.html#breadcrumbs",
          "itemListElement": [
            {
              "@type": "ListItem",
              "position": 1,
              "name": "Home",
              "item": "https://scrubnet.org/"
            },
            {
              "@type": "ListItem",
              "position": 2,
              "name": "Talks",
              "item": "https://scrubnet.org/talks.html"
            },
            {
              "@type": "ListItem",
              "position": 3,
              "name": "Tech SEO North",
              "item": "https://scrubnet.org/talks/access-logs-and-freshness-signals.html"
            }
          ]
        },
        {
          "@type": "Person",
          "@id": "https://scrubnet.org/#founder",
          "name": "Charles Vayssié",
          "url": "https://scrubnet.org/about-us.html",
          "jobTitle": "Founder, Technical SEO Specialist and Web Developer",
          "worksFor": {
            "@id": "https://scrubnet.org/#org"
          },
          "sameAs": [
            "https://www.linkedin.com/in/charles-vayssie/",
            "https://x.com/Scrubnet_"
          ]
        },
        {
          "@type": "Organization",
          "@id": "https://scrubnet.org/#org",
          "name": "Scrubnet Ltd",
          "url": "https://scrubnet.org/",
          "logo": {
            "@type": "ImageObject",
            "url": "https://scrubnet.org/scrubberduck-72.webp",
            "width": 72,
            "height": 72
          }
        },
        {
          "@type": "WebSite",
          "@id": "https://scrubnet.org/#website",
          "url": "https://scrubnet.org/",
          "name": "Scrubnet",
          "publisher": {
            "@id": "https://scrubnet.org/#org"
          },
          "inLanguage": "en-GB"
        }
      ]
    }
  ]
}
