{
  "format": "scrubnet-page",
  "version": 1,
  "language": "en-GB",
  "title": "Public Crawler Feeds | Scrubnet Observatory",
  "description": "Explore the Scrubnet Observatory's public machine-readable feeds and observe verified search and AI crawler discovery, recrawling and request distribution by format.",
  "source": "https://scrubnet.org/llms.html",
  "representations": {
    "html": "https://scrubnet.org/llms.html",
    "markdown": "https://scrubnet.org/llms.md",
    "text": "https://scrubnet.org/llms-page.txt",
    "json": "https://scrubnet.org/llms.json"
  },
  "languageAlternates": {
    "en-GB": "https://scrubnet.org/llms.html",
    "fr-FR": "https://scrubnet.org/fr/flux-publics.html"
  },
  "content": [
    {
      "type": "section",
      "label": "Scrubnet public crawler feeds",
      "content": [
        {
          "type": "heading",
          "level": 1,
          "content": [
            {
              "type": "text",
              "text": "Public Crawler Feeds"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "A public data surface within the Scrubnet Crawler Observatory"
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "content": [
        {
          "type": "heading",
          "level": 2,
          "content": [
            {
              "type": "text",
              "text": "Why the Observatory publishes these feeds"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Scrubnet publishes authorised source-derived HTML, JSON, TXT and Markdown feeds so crawler behaviour can be observed on a consistent public surface. Stable structures, source links and timestamps make it easier to describe discovery, recrawling and request distribution across search, AI and archive bots."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "These feeds are part of the Scrubnet Crawler Observatory: a specialist research capability within the wider toolkit and knowledge hub for technical SEO specialists and web developers."
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "content": [
        {
          "type": "heading",
          "level": 2,
          "content": [
            {
              "type": "text",
              "text": "Questions we observe"
            }
          ]
        },
        {
          "type": "list",
          "ordered": false,
          "items": [
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "Which crawlers discover feeds without direct submission?"
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "How quickly do bots return after a feed or source page changes?"
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "How are verified requests distributed across HTML, JSON, TXT and Markdown?"
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "How do verified and unverified crawler requests differ?"
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "text",
                      "text": "How can observable fetch patterns inform technical SEO and GEO analysis?"
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "content": [
        {
          "type": "heading",
          "level": 2,
          "content": [
            {
              "type": "text",
              "text": "Explore the data"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Start with "
            },
            {
              "type": "code",
              "text": "/feed/sitemap.xml"
            },
            {
              "type": "text",
              "text": " to discover active feeds and their "
            },
            {
              "type": "code",
              "text": "<lastmod>"
            },
            {
              "type": "text",
              "text": " values. Then use the "
            },
            {
              "type": "link",
              "url": "https://scrubnet.org/dashboard.php",
              "content": [
                {
                  "type": "text",
                  "text": "live logs dashboard"
                }
              ]
            },
            {
              "type": "text",
              "text": " to filter requests by path, bot, verification status and date, or compare requests by crawler and format."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "link",
              "url": "https://scrubnet.org/feed/sitemap.xml",
              "content": [
                {
                  "type": "text",
                  "text": "Open Feed Index"
                }
              ]
            },
            {
              "type": "text",
              "text": " "
            },
            {
              "type": "link",
              "url": "https://scrubnet.org/dashboard.php",
              "content": [
                {
                  "type": "text",
                  "text": "View Live Logs"
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "content": [
        {
          "type": "heading",
          "level": 2,
          "content": [
            {
              "type": "text",
              "text": "Technical access"
            }
          ]
        },
        {
          "type": "list",
          "ordered": false,
          "items": [
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "strong",
                      "content": [
                        {
                          "type": "text",
                          "text": "Discovery:"
                        }
                      ]
                    },
                    {
                      "type": "text",
                      "text": " fetch "
                    },
                    {
                      "type": "code",
                      "text": "https://scrubnet.org/feed/sitemap.xml"
                    },
                    {
                      "type": "text",
                      "text": "."
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "strong",
                      "content": [
                        {
                          "type": "text",
                          "text": "Freshness:"
                        }
                      ]
                    },
                    {
                      "type": "text",
                      "text": " compare "
                    },
                    {
                      "type": "code",
                      "text": "<lastmod>"
                    },
                    {
                      "type": "text",
                      "text": " and re-fetch changed feeds only."
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "strong",
                      "content": [
                        {
                          "type": "text",
                          "text": "Conditional requests:"
                        }
                      ]
                    },
                    {
                      "type": "text",
                      "text": " send "
                    },
                    {
                      "type": "code",
                      "text": "If-Modified-Since"
                    },
                    {
                      "type": "text",
                      "text": " or "
                    },
                    {
                      "type": "code",
                      "text": "If-None-Match"
                    },
                    {
                      "type": "text",
                      "text": " where supported."
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "strong",
                      "content": [
                        {
                          "type": "text",
                          "text": "Formats:"
                        }
                      ]
                    },
                    {
                      "type": "text",
                      "text": " use the linked HTML, JSON, TXT or Markdown representation needed for your analysis."
                    }
                  ]
                }
              ]
            },
            {
              "content": [
                {
                  "type": "paragraph",
                  "content": [
                    {
                      "type": "strong",
                      "content": [
                        {
                          "type": "text",
                          "text": "Provenance:"
                        }
                      ]
                    },
                    {
                      "type": "text",
                      "text": " retain the source URL and timestamps when analysing or citing records."
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "content": [
        {
          "type": "heading",
          "level": 2,
          "content": [
            {
              "type": "text",
              "text": "Methodological limits"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Access logs confirm that a request reached a Scrubnet feed. They do not prove that content was indexed, ranked, used for model training, retrieved in an answer or cited to a user. Findings are descriptive observations from Scrubnet’s environment and do not establish why a crawler behaved in a particular way."
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "content": [
        {
          "type": "heading",
          "level": 2,
          "content": [
            {
              "type": "text",
              "text": "ScrubberDuck"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "strong",
              "content": [
                {
                  "type": "text",
                  "text": "ScrubberDuck/1.0"
                }
              ]
            },
            {
              "type": "text",
              "text": " is the robots-aware, low-rate feed compiler that creates and refreshes authorised participating-site feeds. It supplies the observation surface; it is not a search engine or ranking crawler."
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "image",
              "url": "https://scrubnet.org/scrubberduck-200.webp",
              "alt": "ScrubberDuck, Scrubnet’s research crawler"
            }
          ]
        }
      ]
    },
    {
      "type": "section",
      "content": [
        {
          "type": "heading",
          "level": 2,
          "content": [
            {
              "type": "text",
              "text": "Research and crawler teams"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "To contribute a site, suggest an observational question or discuss access to supporting data, email "
            },
            {
              "type": "link",
              "url": "mailto:contact@scrubnet.org",
              "content": [
                {
                  "type": "text",
                  "text": "contact@scrubnet.org"
                }
              ]
            },
            {
              "type": "text",
              "text": "."
            }
          ]
        }
      ]
    }
  ],
  "structuredData": [
    {
      "@context": "https://schema.org",
      "@type": "BreadcrumbList",
      "@id": "https://scrubnet.org/llms.html#breadcrumbs",
      "itemListElement": [
        {
          "@type": "ListItem",
          "position": 1,
          "name": "Home",
          "item": "https://scrubnet.org/"
        },
        {
          "@type": "ListItem",
          "position": 2,
          "name": "Observatory",
          "item": "https://scrubnet.org/observatory.html"
        },
        {
          "@type": "ListItem",
          "position": 3,
          "name": "Public Crawler Feeds",
          "item": "https://scrubnet.org/llms.html"
        }
      ]
    }
  ]
}