---
title: "Cloudflare Sets Deadline for AI Firms to Distinguish Crawlers, Paving Way for Publisher Compensation"
source: "TechCrunch：AI（RSS）"
source_tier: "t2"
url: "https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/"
permalink: "https://feed.astralor.com/items/itm_920ebcbadff5dd5a94d6b24d"
published: "2026-07-01T17:48:37.000Z"
score: 53
channel: "ai"
tags: ["AI agents","AI Training","Cloudflare","Publisher Payment","Web Crawler"]
---

# Cloudflare Sets Deadline for AI Firms to Distinguish Crawlers, Paving Way for Publisher Compensation

## At a glance

Cloudflare issues a policy requiring AI companies to separate web crawlers for search from those for AI training and agents by September 15, or face default blocking on many publisher sites. The move enables publishers to identify and charge for content used by AI, addressing concerns over uncompensated scraping.

## Key points

- Cloudflare requires AI firms to separate search crawlers from AI training/agent crawlers by September 15.
- Non-compliant crawlers risk default blocking on many publisher sites.
- The policy aims to help publishers track content usage and enable payments.

## Full text

Cloudflare is giving AI companies until September 15 to separate web crawlers used for search from those used for AI training and agents, or risk being blocked by default on many publisher sites.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_920ebcbadff5dd5a94d6b24d