The web is being read by software that acts like a person
Something quiet is happening to the web. A growing share of the traffic hitting your site is not a person and not a search-engine crawler indexing you for a results page. It is an AI agent, working on someone’s behalf, trying to find a price, compare an option, book a thing, or answer a question, by reading pages that were designed for human eyes.
The web was never built for that. Every site is different. Layouts move, forms vary, the same intent looks nothing alike from one shop to the next. So agents do what they can: they render the page, scrape the markup, guess at the structure, and break when it changes. It works often enough to be useful and badly enough to be fragile.
The industry has noticed, and two very different answers are emerging. One of them, I am convinced, is a band-aid. This is my attempt to lay out the landscape honestly, including the parts that do not flatter the approach I am betting on, and to explain where I think it actually goes.
Answer one: make the crawlers better
The first instinct is to fix the agent side. If pages are messy, build tools that clean them. This is a real and fast-moving space. Open-source projects like Crawl4AI turn arbitrary web pages into tidy, model-friendly text. Commercial services do the same at scale. Startups are building “AI-readable indexes” that crawl pages and turn them into clean, structured documents for models to consume. And the largest infrastructure players are moving in: Cloudflare has announced an AI Index that automatically produces an AI-optimized version of a site’s content for the domains on its network.
I want to be fair to this approach, because it has a genuine advantage that the alternative does not: it needs zero cooperation from the website. Point a good crawler at any URL, adopted or not, and it produces something usable. That is enormous. It works across the whole open web on day one.
But step back and look at what it is. It is a workaround for a web that does not describe itself. It is lossy, because you are reverse-engineering meaning from presentation. It is read-only, because you cannot scrape your way into an action. It is adversarial, because more and more sites are trying to block the very bots doing the scraping. And it is wasteful. Making a model render a full page, megabytes of markup, styles, and scripts, to extract one number, burns compute, bandwidth, and energy that a few kilobytes of structured meaning would not.
Crawling is the symptom-treatment. It is what you do because the web will not meet you halfway. It is also, right now, winning the read layer, and it deserves to, because for pure reading it is good enough and it needs nobody’s permission.
Answer two: make the websites describe themselves
The second answer flips the problem. Instead of every agent re-deriving a site’s meaning forever, the site describes itself once, in a machine-readable way, and every agent gets clean, correct, structured access. Identity, capabilities, the actions it supports, the policies around them, all declared up front at a predictable location.
Here is the part that convinced me this is not a fringe idea: an unusual number of independent teams have landed on almost exactly the same shape of solution, at almost exactly the same time.
- Some publish a capability manifest describing what a site offers and how to safely use it, often at a well-known URL, with consent and risk baked in.
- Some publish a governance file, closer to an advanced robots.txt, declaring what an agent may read, what it may do, how fast, and whether a human must be involved, with legal and regulatory fields attached.
- Some publish a content-and-identity standard, focused on giving crawlers a clean document format and a way to verify who is crawling.
- And the platform giants are converging on the same primitive from above. Microsoft has agent-manifest formats in its developer stack. Cloudflare’s index exposes structured content and a standard connection method.
When many smart people, working separately, converge on the same primitive, that is not noise. It is the shape of a standard forming. The disagreement is only about the details: the filename, the fields, the emphasis. The premise, that the web needs a machine-readable interface, is now shared.
The dial tone almost nobody is talking about
There is a second convergence, on the connection layer, and it matters more than the file format debate.
Over the last year, one protocol has quietly become the common way for AI assistants to connect to outside tools and data: the Model Context Protocol, MCP. As of 2026, the major assistants let a user point them at a remote MCP endpoint and use whatever it exposes. That includes the large consumer assistants and several of the fast-moving challengers. The requirement is mundane: a publicly reachable endpoint speaking a standard HTTP transport.
This is the part that should reframe the whole conversation. We spent a decade arguing that the web needed a universal way for programs to talk to services, and mostly got a thousand bespoke APIs. In the agent era, a shared dial tone is actually emerging. If your site can expose an MCP endpoint, a person can already connect Claude, ChatGPT, or several others to it and have the assistant use your capabilities directly, with the tool call visible in the interface. That is not a someday. That works now.
The manifest tells an agent what a site can do. MCP is one of the ways it actually does it. The interesting future sits at the intersection.
The distinction that decides everything: reads versus actions
Here is where most of the discussion goes wrong, including some of my own early framing.
The manifest movement is often pitched as “structured content for AI.” But structured content, for reading, is the weakest version of the idea, because reading is exactly what the crawlers already do well and for free. If all a site publishes is a cleaner copy of its text, an agent could have scraped that anyway. The marginal value is small, and the crawlers and the infrastructure giants will own that layer.
The durable value is the opposite of reading. It is doing. An agent can scrape a product page. It cannot, by scraping, track your specific order, start a return tied to your account, or trigger a refund that waits for your explicit approval and returns an audit reference. Those are not content. They are actions, bound to identity, gated by consent, with consequences. They are, by their nature, unscrapable.
This is the line that separates a nice-to-have from a moat. A crawler is a reader. The agentic web’s real prize is the write layer: the transactions, the account-scoped operations, the things a business actually cares about, exposed safely so an agent can perform them with the user’s permission and never without it.
So the honest positioning is not “we give AI cleaner content.” It is “we give AI the actions it cannot get any other way, safely.” Content is table stakes. Actions are the point.
Why crawling is the wrong long-term bet
None of this means crawlers disappear. It means they are the interim, not the destination, and it is worth being precise about why.
Crawling is user-dependent and agent-dependent. It only happens because the site refuses to describe itself, so the burden shifts to whoever is building the agent. It scales to the whole web precisely because it asks nothing of the web, and it stays lossy for the same reason. It cannot transact, because there is nothing to scrape when the operation is a state change behind authentication and consent. And it sits in a slowly escalating conflict with site owners who would rather not be scraped at all.
The site-side approach inverts every one of those properties. It is authoritative, because the site is the source of truth. It is fresh, because it is generated from live data, not re-derived from a snapshot. It can transact, because it exposes real operations, not just text. And it is cooperative rather than adversarial, because the site is choosing to participate.
The catch, and I will not pretend otherwise, is adoption.
The uncomfortable truths
If I only told you the optimistic half, I would be doing exactly the thing I dislike about most technology writing.
Site-side transformation is slow. We have watched this movie. Search Console, sitemaps, structured data, and the various webmaster tools took years to reach broad adoption, and plenty of sites never fully got there. A structured layer for AI faces the same cold-start problem. Meanwhile the crawlers win the interim, because they need no adoption at all.
No standard has won. The manifest convergence is real, but it is a convergence of shape, not of specification. There are several competing files and formats, and betting your site on the wrong one is a real cost. Anyone who tells you the format war is settled is selling something.
The giants could consolidate it. If a major assistant vendor, or Cloudflare, or a standards body blesses one manifest and one connection method, the smaller efforts either converge on it or fade. That is not a reason to sit out. It is a reason to build in a way that survives whichever way it breaks.
And the deepest tension of all: if agents get good enough at using raw websites, and people navigate the web through agents rather than browsers, then for machines the human-facing website becomes almost vestigial. The endpoint that matters is no longer a page. It is an interface, or an agent, that the brand exposes directly. That is not a threat to the idea of a structured layer. It is the same idea, arriving faster than expected.
Where this actually goes: from pages to capabilities to agents
I think the trajectory has three stages, and we are early in the first.
Stage one, now: sites publish structured capabilities. A manifest, a few endpoints, ideally generated automatically by a plugin so the site owner does almost nothing. This is the on-ramp. It does not require anyone to rebuild their site or run new infrastructure.
Stage two, soon: sites expose those capabilities through the emerging dial tone, so any assistant can discover and use them. The website keeps serving humans; a parallel machine interface serves agents. Reads may well flow through the crawlers and the big indexes. Writes, the actions, flow through the capability layer.
Stage three, the prize: the brand runs its own agent. Not a page an agent scrapes, and not just endpoints an agent calls, but an agent that represents the business, on its data, its policies, its terms, and negotiates with the user’s agent directly. Agent to agent. At that point the website, for machines, is a formality. The brand answers with an agent.
The web adapted to search by learning to describe itself to crawlers. It will adapt to agents by learning to describe itself to them, and eventually by answering them as a peer. The only real question is whether that layer is open and interoperable, or owned by whoever gets there first and locks the door.
What I am building, and why open matters
That last question is why I have spent the past stretch building AI2Web, an open protocol, specification, and set of reference implementations for exactly this. It is a bet, and I want to be clear about what kind of bet it is.
It is not a bet that my file format wins. It is a bet that no single format should have to. AI2Web is designed to sit above the transports and the formats: describe your site’s capabilities once, and generate whatever the winning protocol turns out to be, MCP today, others as they arrive, with the consent and safety rules holding uniformly across all of them. If a neutral standard emerges, the right outcome is that AI2Web rides it, not fights it. The hedge is the whole point, because the honest reading of this landscape is that we do not yet know which format wins, and pretending otherwise is how you get stranded.
It is also a bet on actions over reads, on interoperability over ownership, and on making adoption a one-click thing rather than a rebuild. Not because that guarantees success. Nothing here does. But because if the agentic web is going to happen, and the convergence suggests it is, then someone should make sure the on-ramp is open.
The crawlers are not the enemy. They are the proof that the demand is real, and they will own the reading layer for a long time. The work worth doing is the layer they cannot reach: the actions, the consent, the transactions, and eventually the agents that brands run themselves. That is the web adapting to the agentic era. Not by being crawled into shape, but by finally learning to describe itself, and then to speak.
If you build or run a website and you want to be legible to the agents that are already knocking, the open path exists today. That is the part I can help with.