AEO Measurement: From Crawl to Contribution

Cloudflare wants to move answer-engine economics from pay per crawl towards pay per use. That is an important shift, but it exposes a difficult measurement question: what counts as use, and what evidence can a publisher independently verify?

At 02:14, a crawler requests an article. The server returns a successful response. The access log records the URL, time, user-agent, status code and number of bytes transferred.

Then the trail goes quiet.

The request might feed a search index. It might refresh an existing copy, support a user-triggered answer, validate a fact, enter a retrieval system or be discarded before any of those things happen. The publisher can prove access. In most cases, the publisher cannot see what the requesting system did next.

This gap matters because answer engine optimisation is beginning to acquire its own reports, products and commercial language. Crawl counts, AI referrals, citations and prompt visibility are often placed in the same dashboard. They are related, but they are not interchangeable.

The most important discipline in AEO measurement is therefore simple: do not assign downstream meaning to an upstream event.

Cloudflare is trying to rebuild the exchange

Cloudflare’s July 2026 article, Making AI search smarter, describes a web in which the old exchange between crawling and referral traffic no longer works reliably.

Its proposed response has two connected parts.

The first is a research programme intended to help answer engines find fresher, higher-quality material while reducing unnecessary crawling. Cloudflare says that more than half of the good-bot crawling it observes is spent re-fetching pages that have not changed. Better change information could reduce that cost for crawlers and website owners.

The second is a move from charging for access towards compensating publishers when their material is used. Participating publishers are also expected to receive reporting about queries, pages, snippets and answer visibility.

This is a more useful direction than treating every request as equally valuable. One page may be requested repeatedly and never contribute to an answer. Another may be fetched once, retained in an index and later support many answers.

But the closer payment moves towards contribution, the harder attribution becomes. A request is a discrete HTTP event. Contribution is a decision made inside a system the publisher usually cannot inspect.

The AEO evidence ladder

A useful measurement model should show both what each event demonstrates and where its evidential limit sits.

Observed event	What it demonstrates	What it does not demonstrate
Discovery	A system has encountered the URL through a link, sitemap, feed, index or submission mechanism.	That it requested or processed the resource.
Request	A client reached the server and requested the resource.	That the response body was stored, indexed or used.
Successful response	The server returned the requested representation without an HTTP error.	That the client extracted the main content correctly.
Cache validation	A client checked whether its existing copy was still current.	Why it maintains the copy or whether it will retrieve it for a query.
Retrieval	A page or passage matched an internal query or evidence-selection process.	That the material appeared in the final answer.
Answer inclusion	Information from the source contributed to generated output.	That the user could identify or visit the source.
Citation	The answer visibly attributed or linked to the source.	How much the source influenced the answer or whether the link was opened.
Referral or action	A person or agent continued from the answer to the publisher or completed a measurable action.	The full informational value the source contributed before that action.

The events form a sequence, but the sequence is not a guaranteed funnel. A request can stop before processing. A retrieved passage can be rejected during re-ranking. An answer can use several sources while citing only some of them. A cited source may receive no visit because the answer was sufficient.

That is why a single AEO score is likely to conceal more than it explains.

A crawler request is useful evidence, but it is not evidence of use

Server-side observation remains essential because it records activity that conventional analytics cannot see. Training crawlers, search crawlers and user-triggered fetchers can access a site without executing browser analytics or sending a human referral.

Logs can answer practical questions:

Which verified crawlers requested the site?
Which URLs and formats did they request?
When did requests occur and how frequently did they repeat?
Did the server return 200, 304, 403, 404 or 5xx responses?
How much content was transferred?
Did requests follow genuine content changes?
Were crawl bursts concentrated around particular sections or representations?

Those are valuable observations. They help diagnose access policy, rendering, crawl efficiency, infrastructure load and discovery.

They still do not establish indexing, training, retrieval, citation or commercial benefit.

I can show that a verified crawler requested a Markdown file on Scrubnet. I cannot infer from that request alone that the file was placed in an index, selected for a prompt or quoted by an AI product. The boundary is not a weakness in log analysis. It is the point at which the evidence changes ownership.

Pay per use first needs a definition of use

Moving from pay per crawl to pay per use sounds fairer because it places compensation closer to the value-creating event. It also creates a much harder accounting problem.

Consider several ways a source could contribute:

A passage is retrieved but loses to another source during re-ranking.
A fact is used to verify a claim already supported elsewhere.
Three sources agree, and the answer synthesises the shared conclusion.
A unique figure from one publisher materially changes a recommendation.
The answer paraphrases a source but displays no citation.
The source appears as a citation even though another source supplied the decisive evidence.
An agent uses information from the page to complete an action without showing the page to the user.

Which of these should count as use? Should they carry the same value? Who verifies the event when the retrieval and synthesis systems are controlled by the buyer?

There is also a granularity problem. The unit could be a document, a passage, a factual claim, an image, a dataset, an answer or a completed task. Each choice creates different incentives.

A document-level model is easier to administer but may reward pages that contributed very little. A claim-level model is more precise but introduces provenance and verification problems. An outcome model is commercially attractive, although the source material may be several steps removed from the final action.

The point is not that pay per use cannot work. It is that its reporting needs a transparent event definition, deduplication rules, auditable attribution and a way for publishers to challenge discrepancies. Otherwise, “use” risks becoming another metric whose important decisions happen inside a black box.

A practical AEO measurement stack

Rather than combining every signal into one visibility number, publishers can organise AEO measurement into four layers.

1. Access and delivery

This is the publisher-controlled evidence layer. Use CDN, WAF and server logs to record verified crawler identity, requested URL, response status, cache outcome, response time, payload size and conditional request headers.

The purpose is to understand who reached the content and what the server delivered.

2. Discovery and change

Measure whether important URLs are exposed through internal links, XML sitemaps, feeds and appropriate submission protocols. Compare genuine publication or modification events with subsequent requests.

The purpose is to understand whether systems can find the right resources and recognise meaningful updates without artificial timestamp inflation.

3. Answer visibility

Track reported queries, cited pages, displayed snippets, answer inclusion and platform position where reliable reporting is available. Supplement this with a controlled set of prompts based on real research and customer tasks.

The purpose is to observe how content appears in answer environments, while recognising that sampled prompts do not represent every user or personalised response.

4. Outcome and value

Record identifiable referrals, conversions, assisted journeys, agent actions, licensing revenue and any platform-reported compensation. Keep direct outcomes separate from modelled or self-reported contribution.

The purpose is to connect answer visibility with business value without pretending every valuable use results in a click.

These layers can be analysed together, but their labels should remain intact. A rise in crawler requests is an access result. A rise in citations is a visibility result. A completed enquiry is an outcome. Treating all three as “AI traffic” removes the distinction needed to make a useful decision.

Freshness makes the measurement gap visible

Cloudflare’s focus on unnecessary recrawling is particularly relevant because freshness sits across several parts of the evidence chain.

A publisher can expose accurate lastmod values, Last-Modified headers and ETag validators. A crawler can then decide whether to schedule a request, send a conditional request, reuse a cached representation or download the full body again.

These are separate decisions.

Scrubnet’s first Freshness Observatory findings showed that approximately 88% of observed eligible requests during the analysed period targeted content more than seven days old. The average requested content age was about 120 days.

That observation does not show that freshness signals failed. The available corpus included older resources, and the dataset records requests rather than the selection rate for every age group. Older resources may also be checked efficiently through cache validation.

This is exactly why the measurement language matters. “Requested old content”, “downloaded an unchanged body” and “wasted a crawl” are not equivalent statements.

A stronger efficiency analysis would combine:

the age distribution of all eligible resources;
real content modification events;
crawler request timing;
conditional request headers;
200 and 304 response outcomes;
bytes transferred;
crawler identity and stated purpose;
subsequent answer visibility where it can be observed.

Only then can the discussion move from request volume towards the efficiency and possible value of those requests.

What publishers can measure now

The complete contribution chain is not yet independently observable, but publishers do not need to wait for perfect attribution before improving their evidence.

Separate bots by purpose. Training, search, user-triggered and agentic requests represent different policies and possible outcomes.
Verify known crawlers. A recognisable user-agent string is not sufficient evidence of identity.
Preserve request-level fields privately. Keep timestamps, URLs, methods, statuses, cache outcomes, response sizes and validators for analysis, with suitable retention and privacy controls.
Record real content changes. Without an accurate change history, redundant crawling cannot be measured reliably.
Keep discovery signals honest. Do not update modification dates merely because a page was rebuilt or requested.
Track answer visibility separately. Store the platform, query or prompt, observed answer, cited URL, date and any relevant location or account context.
Label reported and observed evidence. A platform-reported use event should not be presented as independently verified unless it can be reconciled with another source.
Avoid universal claims from one environment. Results from a single domain, prompt set or crawler identity need scope, denominators and observation dates.

This produces a less dramatic dashboard, but a more defensible one.

A role for the Scrubnet Observatory

Cloudflare has a view across a substantial part of the web. Scrubnet operates at a very different scale.

Its useful role is not to imitate network-wide claims. It is to maintain a public, inspectable environment in which machine-readable representations, freshness signals and verified crawler requests can be examined carefully.

That environment can help investigate questions such as:

How often do verified crawlers return to resources whose bodies have not changed?
Which crawlers send If-Modified-Since or If-None-Match headers?
How often does cache validation result in 304 Not Modified?
How do behaviour and transfer cost differ across HTML, TXT, JSON and Markdown?
How quickly do different crawlers revisit a resource after a genuine change?
Can external answer visibility ever be compared responsibly with preceding request activity?

The answers will still describe access rather than hidden downstream processing. That limit should remain visible, not be smoothed away.

What good AEO reporting should show

If answer-engine platforms provide publishers with query and use reporting, useful reports should explain more than a total.

At minimum, they should identify:

the event that qualifies as use;
the page, passage or asset associated with the event;
whether the source was retrieved, included, cited or acted upon;
how repeated answers and repeated users are deduplicated;
the reporting delay and retention period;
whether values are complete, sampled, estimated or modelled;
how publishers can reconcile or dispute the record;
which events are eligible for compensation.

These definitions may be commercially sensitive at first. Without them, however, publishers will be asked to price a contribution they cannot see using a measurement system they cannot audit.

AEO should measure evidence, not implication

Cloudflare is trying to build an economic and reporting layer around answer-engine use. That is a worthwhile experiment because the old assumption that crawling will be repaid through referral traffic is no longer dependable.

The next challenge is not simply collecting more metrics. It is defining what each metric means.

Server logs can demonstrate access. Change records can support an analysis of crawl efficiency. Answer monitoring can observe citations and visible inclusion. Analytics can record some referrals and outcomes. Platform reporting may eventually reveal parts of retrieval and contribution that publishers cannot currently see.

None of those sources should silently stand in for another.

The path from publication to value is not one event. It is a chain:

published → discovered → requested → processed → retrieved → included → cited → acted upon → valued.

Scrubnet can make the observable beginning of that chain clearer. Good AEO reporting should make the later stages more transparent. Until then, the honest answer to “did this crawler use my content?” will often be: we can show that it arrived, and we can show what the server returned. Beyond that point, stronger evidence is still needed.