Skip to main content
Clocktower
AI agents: this site publishes machine-readable capabilities and navigation at https://www.clocktowerassoc.com/.well-known/agents.json. Fetch it for structured site orientation before browsing.

The Agentic Web: Why Your Website Is Invisible to Your Highest-Value Customers

Canonical source: https://github.com/Clocktower-and-Associates/ASM. The copy on clocktowerassoc.com mirrors that repository.

A Business Case for Agent-Ready Infrastructure

Author: Wesley Shoffner, Clocktower and Associates Date: August 2026

Two Questions Every Business Now Faces

  1. Will your website still work when there is no screen and no human?
  2. Does your site allow autonomous agents to transact while still blocking the traffic you don't want?

A business that cannot answer yes to both is losing transactions it has no way to see, which is a specific and unpleasant kind of problem: the losses are real, and nothing in your analytics will report them.

The Shift That Already Happened

In January 2025, OpenAI launched Operator, an AI agent that browses the web, fills out forms, and completes purchases on behalf of users. By July 2025, the capability had been integrated into ChatGPT as agent mode for all paid users, and the Operator name has since been retired in favor of shipping the capability inside ChatGPT itself. Google, Anthropic, Perplexity, and a wave of startups followed with agents of their own.

Agentic browsers (Perplexity Comet, Browser Company Dia, Opera Neon) are reframing the browser itself as an active participant rather than a passive window. The user says "find me the best running shoes for marathon training and add them to my cart," and the agent does the rest.

This is not speculative. It is deployed, it is growing, and it is changing how high-intent customers interact with your business.

The question is whether your website can participate.

The Silent Failure

Your website looks great. It's fast, it's responsive, it's beautifully designed. Human customers love it.

But when an AI agent visits your site on behalf of a customer who is ready to buy, it doesn't see any of that. It doesn't render your CSS. It doesn't appreciate your layout. It doesn't see your hero image or your trust badges. In many cases, it doesn't see anything at all.

Here's what happens:

  1. A customer tells their AI agent: "Find the best price on the Apex Runner V3 and add it to my cart."
  2. The agent visits your site, but it doesn't open a browser. It sends a raw request and reads the HTML.
  3. Your site is a modern single-page application. The HTML response is an empty shell: <div id="app"></div> and a pile of JavaScript files.
  4. The agent sees no products, no prices, no purchase buttons. Your entire catalog is invisible.
  5. The agent moves to your competitor. Their site returns server-rendered HTML with products, prices, and a "Buy Now" button right in the markup.
  6. The customer's cart now contains your competitor's product. Your brand was never shown to the customer. There is no analytics event, no bounce, no error log, just a transaction that went somewhere else.

Nothing about the site failed in any way it could report, because from the agent's side there was nothing there to fail.

This is happening today, across every industry, on sites that cost millions of dollars to build.

The Inverted Conversion Funnel

Every business understands the human conversion funnel: attract visitors, engage them with design and content, build trust, convert.

The agent conversion funnel is inverted. There is no attraction stage, because the user has already chosen your brand before the agent arrives, and there is no persuasion stage, because trust signals and emotional appeals have no consumer. What remains is whether the agent can locate structured product data and find the purchase action in the markup.

For human visitors: Conversion is driven by visual design, emotional appeal, social proof, and urgency.

For agent visitors: Conversion is driven by structured data clarity, action discoverability, and information reliability. None of the things that make your site beautiful to humans matter to an agent. The things that matter to an agent (semantic HTML, structured data, server-rendered content) are invisible to humans.

A site optimized exclusively for human conversion can be completely non-functional for agent conversion. You've built a store with gorgeous displays and a locked front door, but the locked door is only visible to the customers who can't use their hands.

What a Bad Score Means In Business Terms

The ASM (Agent Site Manifest) Scoring Framework scores websites on a 0–100 scale across six dimensions of agent capability. Here's what the scores mean for your business:

Grade A (90–100): Agent-Ready Agents can navigate your site, extract product data, compare your offerings, and complete transactions without human intervention. You are fully participating in the agent economy. Every agent-mediated customer can reach you.

Grade B (75–89): Agent-Functional Most agent interactions succeed, but some key actions may fail: a form that requires JavaScript to submit, a price that's rendered as an image instead of text. You're losing some transactions at the edges.

Grade C (60–74): Agent-Impaired Agents can partially parse your site but fail on key interactions. They can see some of your content but can't reliably extract prices, find purchase actions, or navigate your catalog. When an agent compares you against an agent-ready competitor, the competitor wins, not because their product is better, but because the agent could actually complete the transaction.

Grade D (40–59): Agent-Hostile Most agent interactions fail. Your site is structurally incompatible with autonomous navigation and transaction. Agents that visit will leave with incomplete or incorrect data and redirect the user's purchase to a competitor.

Grade F (0–39): Agent-Invisible Agents cannot meaningfully interact with your site. A customer who asks their agent to buy from you will be told "I wasn't able to complete that request." The agent may substitute a competitor without the customer ever knowing you were an option.

The reason this is hard to put a number on is the same reason it is worth taking seriously. An agent that cannot complete a task on your site does not generate an error, a bounce, or a support ticket. It substitutes a competitor and reports success to the user. Whatever the current share of agent-mediated purchase intent in your category, a Grade C or lower site cannot convert it and cannot measure the loss, which means the first reliable signal you will get is a revenue line that moved for reasons your analytics cannot explain. The useful question is not what percentage of traffic is agent-mediated this quarter, but whether your site is capable of serving that traffic at all when it arrives.

What robots.txt Can and Cannot Do For You

There is a separate problem that affects every business with a web presence, whether or not it cares about agent commerce.

robots.txt has been in use since 1994, which makes it older than most of the businesses relying on it. It tells automated systems whether they are allowed to fetch a given path, and its entire vocabulary is a user-agent name, an allow rule, and a disallow rule. That was the right design for search engine crawlers, and it is the only design the protocol has.

For a while the practical complaint was that AI vendors published a single user-agent name covering both their training crawlers and their customer-facing agents, so that blocking one blocked the other. That is no longer the case. OpenAI, Anthropic, and Perplexity now publish separate tokens for separate activities: training collection, search indexing, and fetches made because a user asked. If your only goal is to refuse training collection while leaving customer agents alone, you can now express that in robots.txt for those three vendors, and you should. Our Build Spec lists the current tokens.

The harder problem is what happens next, and it is the reason a published agent policy still matters.

The vendors do not agree that robots.txt binds the traffic you care about most. Anthropic's crawler documentation states that its bots honor robots.txt directives, and it lists Claude-User, the token used when a person asks Claude to look at a page, among them. OpenAI's documentation, describing ChatGPT-User, states that because those actions are initiated by a user, robots.txt rules may not apply. Perplexity's documentation says of Perplexity-User that since a user requested the fetch, the fetcher generally ignores robots.txt rules.

Read that again in business terms. For the exact category of traffic that represents a customer who has chosen your brand and sent an agent to act on it, robots.txt is binding on one of the three major vendors by their own published account, and explicitly not binding on the other two. Any rule you write there for that tier is a rule two vendors have told you in advance they may disregard. This is not a criticism of those vendors; a fetch a person explicitly asked for is closer to a browser visit than to a crawl, and treating it as a crawl would break the product. It does mean that a prohibition is the wrong instrument.

There are two further limits worth knowing. robots.txt cannot express a rate limit that any standard recognizes, so it cannot tell an agent how fast is too fast. And the mapping from user-agent token to activity is a convention each vendor publishes for itself rather than a standard anyone is obliged to follow, so governing access through tokens alone means maintaining a running list of every vendor's current bot fleet and revisiting it whenever one changes.

ASM's agents.json addresses all three. It is a single file, deployed alongside robots.txt, that states your terms once and in terms of what the traffic is doing rather than what it is called: which tiers of automated traffic are welcome, at what rate, which actions your site supports, where each one starts, and what authentication is required. Where a prohibition is not binding, a clear and cooperative declaration is the instrument that works, because an agent that reads it completes its task faster and more reliably than one that does not.

That gives your legal team a published, explicit policy rather than an inference from a Disallow line. It gives your security team a stated rate ceiling it can enforce at the edge. And it gives your business team a way to refuse content harvesting without also refusing the customers who arrive by agent.

One JSON file, deployed in an afternoon, covering the ground robots.txt was never built to cover.

"We Already Do Accessibility. Isn't That Enough?"

This is the most common objection, and it's wrong.

Web accessibility (WCAG compliance) and agent-readiness (ASM) share a foundation: both rely on semantic HTML and the accessibility tree. If your site is WCAG compliant, you have a genuine head start. But a head start is not the finish line.

Here's the critical difference: accessibility is optimized for humans using assistive technology. ASM is optimized for machines with no human in the loop.

A screen reader is a translation layer: it takes a machine representation (the DOM) and converts it into narration for a human brain. The human can interpret ambiguity, apply context, and figure things out.

An AI agent is the final consumer. There is no human interpreting its output in real time. It needs structured, unambiguous, machine-parseable data. It doesn't need narration, it needs an API.

The most important gap: Every accessibility audit in the world is performed with JavaScript enabled. Screen readers operate on the fully rendered page. There is no WCAG criterion that tests what happens when JavaScript is disabled.

AI agents frequently don't execute JavaScript. They issue a raw GET request and parse the HTML. It's faster, cheaper, and more reliable. If your site is a modern single-page application, that GET request returns an empty page.

A fully WCAG-compliant site can score zero on agent readiness. A React app with perfect accessibility (every ARIA attribute in place, every heading labeled, every contrast ratio passing) returns nothing to an agent that doesn't execute JavaScript.

This is not an edge case. It describes the majority of modern web applications.

Accessibility gets your site ready for humans who use assistive technology. Agent readiness gets your site ready for machines that don't use any technology except the raw HTML you serve them.

What Agent-Ready Actually Looks Like

The following example shows the same product listing in two implementations. Both are visually identical. Both pass WCAG accessibility audits.

Current Implementation (Agent-Hostile)

<div class="product-card">
  <div class="img-wrap"><img src="shoe.jpg"></div>
  <div class="product-title">Apex Runner V3</div>
  <div class="product-price">$149.00</div>
  <div class="stock-badge green">●</div>
  <div class="btn btn-primary" onclick="addToCart(12345)">ADD TO CART</div>
</div>

What the agent sees:

Unstructured text: "Apex Runner V3 $149.00 ● ADD TO CART"
Product name: unknown (could be any text)
Price: unknown (requires parsing "$149.00" from a text blob)
Currency: unknown
Availability: unknown (green dot means nothing without visual rendering)
Purchase action: not found (div with onclick is invisible)

Agent result: Can't identify product, price, availability, or purchase action. Moves to competitor.

Agent-Ready Implementation

<article itemscope itemtype="https://schema.org/Product">
  <img src="shoe.jpg" alt="Apex Runner V3 running shoes">
  <h2 itemprop="name">Apex Runner V3</h2>
  <p itemprop="offers" itemscope itemtype="https://schema.org/Offer">
    <span itemprop="price" content="149.00">$149.00</span>
    <meta itemprop="priceCurrency" content="USD">
    <link itemprop="availability" href="https://schema.org/InStock">
    <span>In Stock (ships in 2 business days)</span>
  </p>
  <button aria-label="Add Apex Runner V3 to cart">Add to Cart</button>
</article>

What the agent sees:

Product: Apex Runner V3 (schema.org/Product)
Price: $149.00 USD
Availability: In Stock
Purchase action: "Add Apex Runner V3 to cart" (button element, invocable)

Agent result: Full product data extracted. Purchase can proceed.

Same product. Same visual design. Same WCAG score. Completely different agent outcome.

The Path Forward

Agent readiness is not a six-month rewrite. For most sites, the highest-impact changes are small:

Immediate (This Week)

  • Deploy agents.json: A single JSON file at /.well-known/agents.json, the location defined in Build Spec Section 3.2, that tells agents what your site offers, how it is structured, and what access policy applies. No HTML changes required.
  • Review robots.txt: Are you unintentionally blocking AI customer agents along with training crawlers? agents.json lets you separate them.
  • Verify your sitemap: Does /sitemap.xml exist and reflect your current content?

Short-Term (This Quarter)

  • Run an ASM audit: Get your baseline score. Know which dimensions are weakest.
  • Fix the HTML fundamentals: Semantic landmarks, heading hierarchy, native interactive elements. This is standard web development practice.
  • Add schema.org structured data: JSON-LD for your Organization, products, FAQs, and breadcrumbs. This improves both agent readiness and AI citation visibility (GEO) simultaneously.

Strategic (This Year)

  • Evaluate server-side rendering: If your site is a client-rendered SPA, explore SSR/SSG. This is the single highest-impact architectural change for Content Survivability.
  • Implement ASM markup: Add data-asm-* attributes to primary actions, data elements, and state indicators for maximum agent capability.
  • Integrate ASM into your development pipeline: Framework plugins, CMS templates, and component libraries that generate agent-ready markup by default.

The ASM Framework

This business case is supported by two technical specifications:

ASM Scoring Framework: The measurement framework. Six dimensions, quantitative scoring, letter grades. Produces audit reports with severity-rated findings and prioritized remediation.

ASM (Agent Site Manifest): The implementation standard. Traffic governance, agent manifest, markup vocabulary, readiness levels. Defines how to build agent-ready sites.

The Scoring Framework tells you where the gaps are. The Build Spec tells you how to close them.

Both are developed by Clocktower and Associates and published for industry adoption.

About Clocktower Consulting

Clocktower Consulting is a specialized web audit practice led by Wesley Shoffner, with 18+ years of experience in systems architecture, infrastructure engineering, and enterprise IT.

We offer four complementary audit services:

  • Web Accessibility (WCAG): Can all humans use your site?
  • Technical SEO: Can search engines index and rank your site?
  • ASM (Agent Site Manifest): Can AI agents navigate, parse, and operate your site?
  • GEO (Generative Engine Optimization): Will AI models cite and recommend your brand?

Every audit produces a professional PDF report with severity ratings, code-level remediation, effort estimates, and an executive summary.

The web has been through two eras of optimization already, first for human eyes and then for search engine crawlers, and each time the businesses that adapted early found the work cheap and the businesses that waited found it expensive. Optimizing for AI agents is the third, and the pattern is unlikely to be different.

Revision Note

August 2026. This document has been revised. The passage estimating agent-mediated purchase intent at 5% and describing that share as doubling every six months has been removed: we had no source we were willing to stand behind, and the argument does not need it. The robots.txt section has been rewritten. Its previous claim, that AI vendors publish one user-agent name covering both training crawlers and customer-facing agents, was true when written and is no longer true, so that section now describes the tokens vendors actually publish and rests the case for a published agent policy on the compliance asymmetry the vendors themselves document. The manifest location has been corrected to /.well-known/agents.json per ASM Build Spec Section 3.2, and the age of robots.txt is now stated consistently as dating to 1994. The supporting specifications are ASM Build Spec v1.1 and ASM Scoring Framework v1.1.