TP The business of what comes next Thursday 19 November 2026 · updated 01:11 Get the weekly briefing
Tomorrow’sPublisher.
Editor’s Pick Policy & Rights Editors Picks

All flashes and flames: the data doesn't support a publishers' war on bots

At Flashes & Flames' AI forum this week, a panel of publishers and policymakers made the case for identifying bots, shutting out the unlicensed ones and asking Parliament for a law. Tomorrow's Publisher tested that case against the evidence. It does not hold. The bots cannot be kept out, the law is unlikely to pass, it could not be enforced abroad if it did, and the publishers who pay the price are the British ones.

Ivan Massow · 9 min read

A doorman stops a paying reader at a short velvet rope.

Cartoon · AI-generated

The verdict in brief

Can publishers stop the bots? No. They can stop the polite ones.

Will the law change? Probably not. It is a Private Member's Bill from the opposition benches, and the probability of it or a successor becoming law by the end of 2028 is 37%.

Could it be enforced overseas? Barely. The probability of any publisher collecting money from a foreign operator through the courts by the end of 2028 is 24%.

Who gets hurt? UK companies that obey the rules, and UK readers whose own software gets turned away.

Is there money in licensing? A little. Enough to prove the cheques exist, nowhere near enough to replace what has been lost.

A subscriber pays for a news site, then asks an AI assistant to search its archive and prepare a briefing. The site refuses the assistant. The reader has paid, and has been told that the tool they chose to read with is not welcome. Inside the trade this is called protecting journalism. The customer has a shorter word for it.

That reader was missing from the main stage at Stationers' Hall on 18 November. The session, "AI, public policy and the platforms", put five people in front of a sold-out room: Paul Hood of the House of Commons, Tami Hoffman of the Guardian, Emily Shelley of the Press Association, Paul Shanley of the Associated Press and Sajeeda Merali of the PPA. That is Parliament, a national newspaper, two news agencies and a trade body. Nobody on the platform builds an agent, runs a search product or speaks for the people who now send software to read on their behalf. It was a panel of sellers discussing how to make buyers behave. Programme

Their plan had four parts: identify the bots, decide which uses to permit, block the rest and charge for access. It was received warmly, as it always is. The publishing trade leans heavily towards protection, and it is surrounded by advisers who are paid to tell it to bolt the doors. This piece takes the plan at its word and checks each part against the evidence.

The irony of the venue

It is worth pausing on where this conversation took place, because the building makes the argument better than any panellist did.

The Stationers' Company was given authority over the printing trade by royal charter in 1557. For a century and a half it decided who in England was allowed to copy. Its own records note that books it disapproved of were burned at its hall in June 1599. It had the law, the Crown and a bonfire on its side, and it still lost control of copying to a technology that got cheaper every year.

So the news industry gathered, in the hall of the last body to hold a legal monopoly on copying, to discuss how to get one back. Company history

The Stationers had the law, the Crown and a bonfire. They still lost control of copying.

Publishers cannot stop the bots

Start with the plain fact. A publisher cannot reliably keep a determined bot off its website. It can keep out the bots that agree to be kept out.

The standard tool is a file called robots.txt, which sits on a website and tells automated visitors where they may go. It is a notice on the door. It is not a lock, and the internet standard that defines it says so explicitly: the instructions are not a form of access control. A bot that ignores the notice walks straight in. IETF standard

Real locks exist. Paywalls, logins and limits on how many pages one visitor can request will all refuse a crude scraper. The trouble is the other side's toolkit, which is free, public and better maintained than most publishers' websites. Playwright and Puppeteer drive a real web browser automatically, so the visit looks human. Crawlee manages the crawling. A project called curl_cffi advertises that it can impersonate a browser's digital fingerprint. Scrapling lists "stealth" among its features. Apify will rent out the whole operation as a service, anti-bot handling included. Playwright, Puppeteer, Crawlee, curl_cffi, Scrapling, Apify

Then there are residential proxies, which route a bot's requests through ordinary household broadband connections. The FBI says these networks are built partly from people who signed up and partly from devices that were hijacked. To the publisher, the scraper looks like a family in Swindon. FBI advisory

The industry's best answer is identification. Cloudflare's Web Bot Auth lets a bot prove who it is with a cryptographic signature, which is a real improvement on taking its word. But it only works on bots that choose to sign. The scheme identifies the honest and tells a publisher nothing about the rest. Cloudflare documentation

And the website is only one copy. India's Sach Kahoon found its entire editions being passed around WhatsApp and Telegram after readers rebuilt them from page images. NewsGuard found 37 websites using AI to rewrite mainstream reporting back in 2023. Once the journalism is out, it is out. INMA case study, NewsGuard

A publisher can make scraping harder and more expensive. It cannot make it stop, and the cost of trying never ends. That cost is somebody else's revenue, which may explain the enthusiasm with which the defence is sold.

The scheme identifies the honest and tells a publisher nothing about the rest.

The law will not arrive

If technology cannot do it, the hope is that Parliament will. The vehicle is the Automated Online Software (Access and Transparency) Bill. It would make bot operators declare who they are and what they want, set up a register, and let courts order UK broadband providers to block persistent offenders.

It is a Private Member's Bill. It was introduced by Damian Hinds, a Conservative MP on the opposition benches, with the backing of the News Media Association. The government has not adopted it. Its second reading was first listed for 16 October and is now listed for 15 January 2027. Bills of this kind, without ministers behind them, seldom become law, and Press Gazette, which is hardly hostile to the cause, says this one is unlikely to. Newspaper bosses have asked Andy Burnham to take it up. Bill page, InPublishing, Press Gazette

37% The probability, on Noah's forecast, that this Bill or a successor receives Royal Assent by 31 December 2028. Put the other way round, there is a 63% probability that it does not.

The American equivalent, the Stealth Bot Prohibition Act, is parked in a House committee.

33% The probability, on Noah's forecast, that comparable US federal legislation becomes law in the same period.

The industry is planning around a law that will probably not exist. And if it did, it would oblige bots to fill in a form. It would not block a single hidden request, and it would not send a publisher a penny. US status

It would bind Britain and nobody else

Suppose the Bill passes anyway. Who obeys it?

A British AI start-up with a London office, UK customers and a board of directors will register, declare its purpose and pay its lawyers. So will the large American firms, which already identify their crawlers and already sign licences. An operator in a jurisdiction that does not care, with no office, no assets and no customers in Britain, will do none of those things and will carry on scraping.

The Bill mentions overseas operators. Mentioning them is the easy part. To collect money abroad, a publisher needs a foreign court to enforce a British judgment, and the main international treaty on enforcing judgments, the 2019 Hague Convention, specifically excludes intellectual property. Hague Convention, Article 2

24% The probability, on Noah's forecast, that any publisher anywhere actually receives money by 31 December 2028 from a UK, US or EU court judgment over unauthorised automated access against an operator based outside that jurisdiction. Even then, the payer is likely to be a large company with assets within reach, not an anonymous scraper.

The result is a law that works on the people who were already behaving. British developers get a compliance burden their overseas competitors do not. British publishers get a register of the bots that were never the problem. It is a handicap for the home side, described as protection for it.

British publishers get a register of the bots that were never the problem.

The usual defence of blocking is that it has become selective, and that is true. OpenAI runs one crawler for training its models and another for search. Google now offers a control, live worldwide since the end of August, that takes a site out of AI Overviews and AI Mode while leaving it in the ordinary results, with no ranking penalty. OpenAI, Google

Now look at what is being kept. Google's own documentation says opting out removes a site's links, impressions and traffic from those AI features. And Pew's study of 900 American adults found that when an AI summary appeared, people clicked an ordinary result on 8% of visits, against 15% when it did not. The AI layer is where search is going. The blue links are what is left behind. A publisher that opts out keeps its seat in the half of the theatre that is emptying. Pew Research Center

70% The probability, on Noah's forecast, that Google delivers a further new control for UK news publishers by 31 December 2027, letting them leave AI Overviews without a ranking penalty. More switches are coming. None of them brings the traffic back.

Then there is what the industry calls the headless web: software that reads, compares and buys on a person's behalf, without anyone looking at a page. The Reuters Institute's October survey found that 14% of people across nine countries had used an AI agent for a real task in the previous week. Weekly use of AI for news rose from 6% to 9% in a year. It is a minority, and it is growing by half annually. Reuters Institute

A site cannot tell a scraper from a customer's assistant unless it has built a way to do so. Blend Commerce sent an automated shopping browser to 200 UK and US retailers and was turned away twice by 29 of them. Those retailers had refused a visitor that might have been carrying a customer's money. Publishers with subscriptions, affiliate links and specialist data are making the same mistake with more enthusiasm. Blend Commerce

Agents do misbehave. When they abused The RealReal's item-hold feature, the retailer tightened the transaction and left discovery open. It fixed the product. It did not ban the customer. The Verge

The blocking instinct also damages things nobody meant to hit. The Guardian restricted the Internet Archive's access as a precaution, without having found any scraping through it, and the New York Times blocked the Archive's crawler outright. The Stationers' Company, for all its faults, signed the 1610 agreement with the Bodleian that began the tradition of depositing publications for posterity. Four centuries on, in the same hall, its successors are making their own record harder to keep. Nieman Lab

The money is real and small

Licensing is the part of the plan that works, up to a point. PLS, the publishers' collecting society, says more than 800 publishers are in its scheme covering AI use in the workplace, and payments have been going out since March. More than 350 have signed up for its broader AI-product licensing, about which PLS itself says significant revenue should not be expected in the short term. PLS

77% The probability, on Noah's forecast, that by 31 December 2028 at least two independent UK publishers each receive two consecutive AI royalty payments from the same collective scheme. This is the most confident forecast in the piece, and the bar could hardly be lower. Two small cheques, twice, would clear it.

A publisher receives a grand ceremonial cheque for a tiny royalty.
Cartoon · AI-generated

Nobody has published a figure suggesting these royalties will replace lost advertising or subscriptions, and the organisation collecting them is telling its members not to expect it.

The best evidence against the panel's plan was sitting on the panel. The Associated Press sent its senior director of revenue strategy. AP sells a product called AP Intelligence, which packages its journalism as structured data for AI systems and corporate monitoring tools. It is a new product, built for machine customers, sold on purpose. AP has not said whether it makes a profit. It is still the only idea in the room that involved making something. AP

The scorecard

Five forecasts, one conclusion

Question Probability
UK bot-identification law passes by end of 2028 37%
US federal equivalent passes by end of 2028 33%
Any publisher collects from a foreign operator via the courts by end of 2028 24%
Google ships a further AI opt-out for UK publishers by end of 2027 70%
Two independent UK publishers get two consecutive AI royalty payments by end of 2028 77%

The things publishers are counting on are unlikely. The things that are likely are small.

What publishers should do instead

The courts will not supply the missing certainty. The EU's Advocate General Szpunar advised on 3 November that an AI service retrieving and reusing press content in real time can infringe publishers' rights in certain circumstances, and in the same opinion separated the facts a publication reports from the expression it owns. Facts remain free. Court summary

Some of what the panel proposed is worth having. Signed bots are useful. Usage reporting, such as the telemetry standard published by the SPUR coalition, is useful. Collective licensing puts real if modest money in real accounts. These are sensible housekeeping. SPUR

They are not a strategy, and the war on bots is not one either. It rests on a wall that leaks, a Bill with a one-in-three chance, enforcement that stops at Dover and royalties that buy lunch. It will be obeyed by British firms and ignored by everyone the industry is actually afraid of. And every month spent on it is a month not spent building for the reader who now arrives as software: the archive an agent can search, the data product a machine can buy, the publication that did not exist last year.

The Stationers spent 150 years defending the right to control copying, and lost. The publishers who met in their hall this week do not have 150 years.

Forecasts are from Noah Predict and run from 19 November 2026. Full method and sources: Claim audit and source notes

Advertisement · Noah

Source: Tomorrow’s Publisher

Make sense of what's next.

A weekly briefing for people shaping the future of publishing.

Free. Email requests are handled by our team.