Cloudflare’s crawler can now refuse its own request when a publisher’s declared reuse rules are stricter than the crawl job.
An August 31 Browser Rendering update adds a practical check to Cloudflare’s /crawl endpoint. A caller declares whether it wants content for reference or full use. Cloudflare reads the site’s Content Signals. If the requested use is more permissive than the publisher’s rule, the crawl returns HTTP 400.
The new parameter is contentUse
Cloudflare’s August 31 changelog says Browser Rendering’s crawl endpoint now respects the Content Signals use directive. The crawl API documentation lists two values for contentUse: reference and full. The default is full.
That default deserves attention. A developer who omits the field is not making a neutral request. The crawler treats it as full use, then compares that request with the site’s signal.
| Requested use | Site allows | Expected result |
|---|---|---|
| reference | reference | Crawl can proceed |
| reference | full | Crawl can proceed |
| full | reference only | HTTP 400 conflict |
| field omitted | reference only | HTTP 400 because default is full |
A 400 response is a policy conflict, not a broken page
HTTP 400 usually sends developers toward malformed JSON, invalid selectors, or a bad URL. In this flow it can also mean the declared content use conflicts with the publisher’s rule. That distinction belongs in logs and dashboards. Otherwise an operator may retry a policy refusal as if it were a flaky render.
I would store four fields for every crawl: target URL, requested contentUse, response status, and the site’s resolved signal. A retry should preserve the original use. Silently downgrading from full to reference can produce content that later enters the wrong pipeline.
Content Signals are machine-readable preferences
The Content Signals specification gives publishers a machine-readable way to express preferences about uses such as search, AI input, and AI training. The exact vocabulary and delivery mechanism matter more than a vague line in a terms page because crawlers can evaluate the rule before collecting content.
Cloudflare also offers managed robots.txt controls. Robots directives, Content Signals, access controls, and contracts solve different parts of the problem. Treat them as layers, not interchangeable switches.
This is not universal AI crawler enforcement
The update changes one Cloudflare crawler implementation. It does not prove that every crawler reads Content Signals. It does not block direct requests outside this endpoint. It does not settle copyright questions, and an HTTP 400 response is not a legal judgment.
That caveat is the whole reason the change is useful. Policy language becomes more valuable when a tool turns it into a predictable technical outcome, but site owners still need to test other agents and traffic paths. Our analysis of Google’s generative AI controls covers a different system where opting out also carries visibility tradeoffs.
A five-minute integration test
- Choose one page with a restrictive Content Signal and one permissive control page.
- Call
/crawlwithcontentUse: "reference"against both. - Repeat with
contentUse: "full". - Repeat once with the field omitted to verify the default.
- Confirm that your job queue records a policy conflict separately from render and network failures.
Run the test in staging before changing a production crawler. If the content later feeds search, retrieval, model input, or training, attach the requested use to the artifact so it does not lose its origin after ingestion.
Publishers should also keep the rule aligned with the promises made in their content workflow. Our guide to SEO with AI explains why provenance and human review still matter after a crawler has permission to collect a page.
My verdict
This is a small API change with a good operational idea: the crawler declares its intended use, the site declares its boundary, and the request fails when those declarations disagree. I would copy that pattern even outside Cloudflare. Make purpose an input to collection, not a note someone reads after the data has moved.
Cloudflare’s changelog and crawl API documentation were verified on September 1, 2026. The enforcement described here applies to Cloudflare Browser Rendering’s crawl endpoint.
