Publishers have always had their content crawled. For years, that was simply part of how the internet worked. Search engines indexed a page and, in return, publishers had a chance of getting a reader through the door.AI is starting to change that deal.
A crawler can now take information from an article and feed it into an AI product capable of answering a user’s question directly. The publisher has created the information, but the person searching for it may never need to visit the original website.
New figures suggest European publishers are seeing a particularly high level of this activity.
Research from TollBit, reported by Digiday, looked at AI bot activity across 3,906 publishers and found that the median European site received four times as many AI scrapes as its North American equivalent.
What’s coming back the other way is considerably smaller.
The research found European publishers received around one human referral from an AI application for every 179 AI bot visits. AI applications accounted for only 0.05% of external referrals to the European publishers studied during the first half of 2026.
At the same time, scraping is heading in the opposite direction. TollBit recorded almost 20% more AI scrapes on European publisher sites in June than it had in January.
Those numbers don’t necessarily mean Europe is universally being targeted more aggressively. Different security companies are seeing slightly different patterns of web scraping detection..
Cloudflare data shows North American publishers receiving more bot requests in absolute terms, for example, while bots make up a greater share of overall traffic to European media and publishing websites. DataDome has also found significant differences from publisher to publisher.
What does seem clear is that AI crawlers are becoming a meaningful part of publisher traffic.
Why would European content be attractive?
Language could be part of the answer.
An AI product that wants to be useful in Germany, France, Spain or Italy needs more than English-language information translated into another language. It needs local sources, local reporting and an understanding of the subjects people in those markets are actually talking about. That potentially makes Europe’s fragmented media landscape extremely useful.
There is some supporting evidence here. European country domains have become a larger part of the Common Crawl dataset over time, suggesting greater demand for material from outside the traditional English-language web.
But the more interesting question isn’t really why AI companies want the content.
Of course they want it. Publishers spend considerable amounts of money producing information that is current, original and useful. The question is what publishers get in return.
A Google search that used a publisher’s content to determine its rankings could still deliver a click. That click could become an advertising impression, newsletter registration, subscription or another page view. An AI answer can remove that step altogether. Ask a question, get the information and move on.
For the reader, that’s wonderfully convenient. For the company that paid to produce the information, the economics look rather different.
The next argument could be about value
If AI platforms continue using publisher content while sending relatively little traffic back, the relationship may eventually need a different commercial model.
Some publishers will inevitably ask whether access to their journalism should be licensed rather than treated as an automatic part of being on the open web. Others may decide certain crawlers are welcome while restricting those that offer little obvious value in return.
That could make bot management a commercial decision as much as a technical one.
Publishers will need to know which crawlers are accessing their sites, how frequently they visit and what content they’re taking. Without that visibility, it’s difficult to decide which automated services should be allowed, restricted or blocked.
The question may therefore shift from “How do we stop AI bots?” to something more complicated: “Which AI bots are we prepared to let in, and what should we expect in return?”
For publishers, that distinction could become increasingly important as AI-driven discovery grows.
Publishers now have a difficult choice
Blocking crawlers sounds like the obvious response, but it isn’t quite that simple.
Publishers may want their reporting represented in AI services, particularly if those platforms become an important way people discover information. Blocking everything could mean giving up visibility in a channel that is still developing.
Allow everything, however, and publishers risk seeing more of their work consumed without receiving meaningful traffic in return.
There is also the question of whether requests to block crawlers are respected. TollBit’s research found a notable level of attempted scraping despite publishers indicating that content should not be accessed.
That pushes the conversation beyond traffic and into something much more commercial.
Publishers have already lived through one version of the platform economy, where search engines and social networks became enormously influential in deciding how audiences discovered their journalism.
AI could create the next one. The difference this time is that the platform doesn’t necessarily need to send the audience anywhere.
