Skip to Content
New Old Web home
New Old Web home

Open Web

A Short History of Paying for News: Why We’re Building Allegro Audience

The commerce layer has always been core to publishers and must be treated as a first-class part of the technology ecosystem. News organizations have been charging for regular access to the news for hundreds of years.

July 1, 2026

Analyzing 5,818 Publishers’ robots.txt Files: Most Non-profit News Organizations Allow AI Bots, OpenAI Most Commonly Blocked

Robots.txt is a common code format that allows website owners to instruct and direct crawlers, scrapers, spiders, and other automated systems that identify themselves as a unique user agent. Once used to green or red light search engines from accessing a site’s content, publishers are now relying on robots.txt for something completely new: Managing web…

October 15, 2025