{
  "format": "scrubnet-page",
  "version": 1,
  "language": "en-GB",
  "title": "Google’s 2 MB Crawl Limit and the Machine Web, Scrubnet",
  "description": "What Google’s 2 MB crawl limit means for indexing, rendering, resource size and Scrubnet’s machine-readable public feeds.",
  "source": "https://scrubnet.org/discoveries/google-2mb-crawl-limit-machine-web.html",
  "representations": {
    "html": "https://scrubnet.org/discoveries/google-2mb-crawl-limit-machine-web.html",
    "markdown": "https://scrubnet.org/discoveries/google-2mb-crawl-limit-machine-web.md",
    "text": "https://scrubnet.org/discoveries/google-2mb-crawl-limit-machine-web.txt",
    "json": "https://scrubnet.org/discoveries/google-2mb-crawl-limit-machine-web.json"
  },
  "languageAlternates": {
    "en-GB": "https://scrubnet.org/discoveries/google-2mb-crawl-limit-machine-web.html",
    "fr-FR": "https://scrubnet.org/fr/articles/limite-exploration-google-2-mo-web-machines.html"
  },
  "content": [
    {
      "type": "section",
      "label": "Article hero",
      "content": [
        {
          "type": "heading",
          "level": 1,
          "content": [
            {
              "type": "text",
              "text": "Google’s 2 MB Crawl Limit"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Why byte budgets matter and how Scrubnet fits the future of crawling"
            }
          ]
        }
      ]
    },
    {
      "type": "aside",
      "label": "Article author",
      "content": [
        {
          "type": "paragraph",
          "content": [
            {
              "type": "image",
              "url": "https://scrubnet.org/charles-vayssie.webp",
              "alt": "Charles Vayssié, founder of Scrubnet"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "By "
            },
            {
              "type": "link",
              "url": "https://scrubnet.org/about-us.html",
              "content": [
                {
                  "type": "text",
                  "text": "Charles Vayssié"
                }
              ]
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Founder, Technical SEO & Web Developer"
            }
          ]
        },
        {
          "type": "paragraph",
          "content": [
            {
              "type": "text",
              "text": "Technical SEO specialist and web developer leading strategy and delivery across complex e-commerce and international websites, with expertise in migrations, rendering, crawl management, structured data and Core Web Vitals."
            }
          ]
        }
      ]
    },
    {
      "type": "article",
      "content": [
        {
          "type": "section",
          "id": "intro",
          "content": [
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Google clarified a major crawling constraint. When "
                },
                {
                  "type": "link",
                  "url": "https://developers.google.com/search/docs/crawling-indexing/googlebot",
                  "content": [
                    {
                      "type": "text",
                      "text": "Googlebot crawls"
                    }
                  ]
                },
                {
                  "type": "text",
                  "text": " a page for Search, it only fetches the first 2 MB of each supported resource. This is down from older, looser limits and it applies strictly to uncompressed data."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "This is not a theoretical guideline. It is a hard cutoff. Once the limit is reached, the fetch stops and only the retrieved portion is considered for indexing and rendering."
                }
              ]
            }
          ]
        },
        {
          "type": "figure",
          "image": {
            "url": "https://scrubnet.org/discoveries/2m-google-limit-scrubberduck.webp",
            "alt": "Scrubberduck in the digital world avoiding google limitations."
          },
          "caption": [
            {
              "type": "text",
              "text": "Scrubberduck in the digital world avoiding google limitations."
            }
          ]
        },
        {
          "type": "section",
          "id": "context",
          "content": [
            {
              "type": "heading",
              "level": 2,
              "content": [
                {
                  "type": "text",
                  "text": "The change in practical terms"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The important detail is that the 2 MB limit applies per fetched resource, not per page."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "That means:"
                }
              ]
            },
            {
              "type": "list",
              "ordered": false,
              "items": [
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "text",
                          "text": "The HTML document has its own 2 MB limit"
                        }
                      ]
                    }
                  ]
                },
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "text",
                          "text": "Each JavaScript file has its own 2 MB limit"
                        }
                      ]
                    }
                  ]
                },
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "text",
                          "text": "Each CSS file has its own 2 MB limit"
                        }
                      ]
                    }
                  ]
                },
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "text",
                          "text": "Each referenced asset is fetched independently"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Google does not pool these together. There is no overall page allowance. Each request is bounded on its own."
                }
              ]
            }
          ]
        },
        {
          "type": "section",
          "id": "reality",
          "content": [
            {
              "type": "heading",
              "level": 2,
              "content": [
                {
                  "type": "text",
                  "text": "Why this matters more than it sounds"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Many modern sites transfer ten to twenty megabytes per page without issue. That alone is not a problem. What matters is the size of individual resources and where important content appears."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "If the HTML exceeds 2 MB, anything beyond that point is invisible to Google. If a JavaScript bundle exceeds 2 MB, Google executes only a partial file. Critical rendering logic, injected content, or internal links may never be reached."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Compression does not help here. The limit applies after decompression. A heavily compressed file can still exceed the cutoff once inflated."
                }
              ]
            }
          ]
        },
        {
          "type": "section",
          "id": "logs",
          "content": [
            {
              "type": "heading",
              "level": 2,
              "content": [
                {
                  "type": "text",
                  "text": "The role of access logs"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "In most setups, this cutoff is observable. When Googlebot reaches its internal limit, it aborts the connection. The server stops sending data."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "With proper access logging, you can see:"
                }
              ]
            },
            {
              "type": "list",
              "ordered": false,
              "items": [
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "text",
                          "text": "The exact number of bytes actually sent"
                        }
                      ]
                    }
                  ]
                },
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "text",
                          "text": "Requests that terminate early"
                        }
                      ]
                    }
                  ]
                },
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "text",
                          "text": "Responses that cluster around the 2 MB boundary"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "This moves the discussion from documentation to evidence. You can see what Google actually consumed, not what the page was meant to deliver."
                }
              ]
            }
          ]
        },
        {
          "type": "section",
          "id": "scrubnet",
          "content": [
            {
              "type": "heading",
              "level": 2,
              "content": [
                {
                  "type": "text",
                  "text": "How Scrubnet’s observatory aligns with this reality"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The Scrubnet crawler observatory publishes compact machine-readable feeds and records how verified crawlers request them."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Every observatory feed is:"
                }
              ]
            },
            {
              "type": "list",
              "ordered": false,
              "items": [
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "text",
                          "text": "Compact and predictable in size"
                        }
                      ]
                    }
                  ]
                },
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "text",
                          "text": "Clean HTML with no rendering dependency"
                        }
                      ]
                    }
                  ]
                },
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "text",
                          "text": "Free from oversized scripts and layout noise"
                        }
                      ]
                    }
                  ]
                },
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "text",
                          "text": "Designed to be fully consumable in a single fetch"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "This ensures that crawlers never hit byte ceilings before reaching meaningful content. What is published is what is read."
                }
              ]
            }
          ]
        },
        {
          "type": "section",
          "id": "future",
          "content": [
            {
              "type": "heading",
              "level": 2,
              "content": [
                {
                  "type": "text",
                  "text": "What this signals about the future"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Google’s clarification is not a regression. It is an admission. Crawlers need firm limits to operate at scale. Rendering everything is expensive. Guessing intent is risky."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The web is splitting into layers. A human web full of interaction and presentation. A machine web focused on clarity, structure, and bounded cost."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "The Scrubnet crawler observatory operates in the second layer."
                }
              ]
            }
          ]
        },
        {
          "type": "section",
          "id": "takeaways",
          "content": [
            {
              "type": "heading",
              "level": 2,
              "content": [
                {
                  "type": "text",
                  "text": "Key takeaways"
                }
              ]
            },
            {
              "type": "list",
              "ordered": false,
              "items": [
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "text",
                          "text": "The 2 MB limit applies per resource, not per page"
                        }
                      ]
                    }
                  ]
                },
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "text",
                          "text": "Oversized HTML and JS can silently lose content"
                        }
                      ]
                    }
                  ]
                },
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "text",
                          "text": "Access logs reveal real crawl behaviour"
                        }
                      ]
                    }
                  ]
                },
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "text",
                          "text": "Machine first publishing reduces uncertainty"
                        }
                      ]
                    }
                  ]
                },
                {
                  "content": [
                    {
                      "type": "paragraph",
                      "content": [
                        {
                          "type": "text",
                          "text": "Clean feeds are future proof by design"
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        },
        {
          "type": "section",
          "id": "cta",
          "content": [
            {
              "type": "heading",
              "level": 2,
              "content": [
                {
                  "type": "text",
                  "text": "Observe crawlers without assuming outcomes"
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "text",
                  "text": "Scrubnet helps authorised participating sites publish compact machine-readable representations, then makes verified request behaviour observable. A request confirms access only; it does not prove trust, indexing, reuse or citation."
                }
              ]
            },
            {
              "type": "paragraph",
              "content": [
                {
                  "type": "link",
                  "url": "https://scrubnet.org/brands.html",
                  "content": [
                    {
                      "type": "text",
                      "text": "Add a Site"
                    }
                  ]
                },
                {
                  "type": "text",
                  "text": " "
                },
                {
                  "type": "link",
                  "url": "https://scrubnet.org/llms.html",
                  "content": [
                    {
                      "type": "text",
                      "text": "Explore Public Feeds"
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "structuredData": [
    {
      "@context": "https://schema.org",
      "@graph": [
        {
          "@type": "BlogPosting",
          "@id": "https://scrubnet.org/discoveries/google-2mb-crawl-limit-machine-web.html#article",
          "mainEntityOfPage": {
            "@type": "WebPage",
            "@id": "https://scrubnet.org/discoveries/google-2mb-crawl-limit-machine-web.html"
          },
          "headline": "Google’s 2 MB Crawl Limit and the Shape of the Machine Web",
          "alternativeHeadline": "Why byte budgets matter and how Scrubnet fits the future of crawling",
          "description": "Google now enforces a 2 MB crawl limit per resource for Search. Learn what this means for indexing and rendering, how truncation can happen, and how Scrubnet publishes machine first feeds designed for modern crawlers and AI agents.",
          "image": "https://scrubnet.org/discoveries/2m-google-limit-scrubberduck.webp",
          "datePublished": "2026-02-06",
          "dateModified": "2026-07-22",
          "author": {
            "@id": "https://scrubnet.org/#founder"
          },
          "publisher": {
            "@id": "https://scrubnet.org/#org"
          },
          "inLanguage": "en-GB"
        },
        {
          "@type": "BreadcrumbList",
          "@id": "https://scrubnet.org/discoveries/google-2mb-crawl-limit-machine-web.html#breadcrumbs",
          "itemListElement": [
            {
              "@type": "ListItem",
              "position": 1,
              "name": "Home",
              "item": "https://scrubnet.org/"
            },
            {
              "@type": "ListItem",
              "position": 2,
              "name": "Articles",
              "item": "https://scrubnet.org/discoveries.html"
            },
            {
              "@type": "ListItem",
              "position": 3,
              "name": "Google’s 2 MB Crawl Limit and the Shape of the Machine Web",
              "item": "https://scrubnet.org/discoveries/google-2mb-crawl-limit-machine-web.html"
            }
          ]
        },
        {
          "@type": "Organization",
          "@id": "https://scrubnet.org/#org",
          "name": "Scrubnet Ltd",
          "url": "https://scrubnet.org/",
          "logo": {
            "@type": "ImageObject",
            "url": "https://scrubnet.org/scrubberduck-72.webp"
          }
        },
        {
          "@type": "Person",
          "@id": "https://scrubnet.org/#founder",
          "name": "Charles Vayssié",
          "jobTitle": "Founder, Technical SEO Specialist and Web Developer",
          "image": "https://scrubnet.org/charles-vayssie.webp",
          "url": "https://scrubnet.org/about-us.html",
          "worksFor": {
            "@id": "https://scrubnet.org/#org"
          },
          "sameAs": [
            "https://www.linkedin.com/in/charles-vayssie/",
            "https://x.com/Scrubnet_"
          ]
        },
        {
          "@type": "WebSite",
          "@id": "https://scrubnet.org/#website",
          "url": "https://scrubnet.org/",
          "name": "Scrubnet",
          "publisher": {
            "@id": "https://scrubnet.org/#org"
          },
          "inLanguage": "en-GB"
        }
      ]
    }
  ]
}