We Hid Our Own API Five Different Ways to See If AI Agents Could Find It

The short version

We run a small public API of developer tools on this site. In one day we deployed five different versions of our discovery infrastructure, from a full machine-readable declaration down to nothing at all, and sent 78 autonomous AI agents to find and use the API under a strict request budget. Every claimed success had to include a signed receipt from our server, which we verified after the fact, so no agent could talk its way into the results.

The outcome in one table:

Three things found our API: a machine declaration, human documentation, and lucky guessing based on naming convention. Nothing else did, and when we removed all three, every agent failed. Several of them then told their operator, confidently, that our API does not exist.

Why we did this

Our clients ask a fair question: does agent readiness actually matter, or is it a checklist we invented to sell audits? The honest answer requires an experiment where the site is the variable, so we used our own production site as the test bench. The tools API on clocktowerassoc.com serves the same utilities our tools pages offer a person, in a shape an agent can call. It is real, it is public, and we control every layer of how it is advertised.

The setup

The task given to every agent was identical: obtain a pretty-printed version of a fixed JSON document using tooling provided by clocktowerassoc.com. The agent was forbidden to format the JSON itself, capped at 12 HTTP requests, restricted to plain HTTP calls with no browser, and required to return the raw server response.

We tested four model tiers (Haiku, Sonnet, Opus, and Fable), two trials per cell, against five deployed conditions. One round also ran an extra browser-class wave alongside the plain-HTTP class: the tables in this study report the plain-HTTP class, where every rung is directly comparable, while the experiment totals of 78 trials and 53 verified successes span both classes.

Each condition was a real production deployment, changed one layer at a time:

RUNG 1   Full readiness: ACP declaration in .well-known/agents.json,
         llms.txt, robots.txt entries, a Link header, and a head link,
         with the API at the conventional /api/tools path.

RUNG 2   The same manifest with the services section removed. The file
         still looks authoritative. It just no longer mentions the API.

RUNG 3   No discovery surfaces at all. The API still answers at
         /api/tools, but nothing anywhere says so.

RUNG 4   No discovery surfaces, and the API moved to /api/devtools.
         There is a tool here. Go find it.

RUNG 5   Rung 4 plus one ordinary human docs page at /tools/api, linked
         with a single sentence from our tools hub. This is what most
         real sites have: some docs, no machine layer.

The receipts

Agents overstate their success. We designed for that before running anything. Every successful response from our tools API carries a signed proof: an HMAC-SHA256 signature over the response data and a server timestamp, generated with a secret that lives only in our deployment platform. After each round froze, we replayed every claimed proof against our public verify endpoint and checked two more things: the timestamp had to fall inside that round's execution window, and every signature had to be unique across the whole experiment, which rules out replaying one success into many.

The result: 53 verified successes across the ladder, zero fabricated ones, and every failure carried no proof by construction. When one model tier garbled its transcription of the response, the signature still let us confirm the success was real. This is what a source of truth buys you.

What happened at each rung

Rung 1: the declaration works, and it is cheap

With the full declaration published, every agent succeeded, and the fast path was very fast: read the manifest, call the endpoint, done.

requests to success, rung 1 (median)

no pointer given      ####  4
declaration URL given ##    2

Rung 2: a misleading manifest is worse than none

With the services section removed but the manifest still present, success dropped to 11 of 16. The interesting part is not the drop. It is the mechanism: agents read the authoritative-looking file, saw no mention of an API, and several stopped probing. The manifest actively talked them out of finding a thing that existed.

Rung 3: convention was doing the work all along

With every discovery surface deleted, 7 of 8 agents still succeeded, and the request logs show why: every single success guessed /api/tools, because that is where a REST API for a tools site conventionally lives. This looks like evidence that discovery infrastructure is optional. It is not, as rung 4 shows.

Rung 4: move one path segment, lose everything

Same zero-readiness terrain, API at /api/devtools instead of /api/tools. Clean-run success: 0 of 8.

the collapse, no-pointer success by rung

rung 1  ################  16/16
rung 3  ##############     7/8
rung 4                     0/8

Two details from the failures deserve attention. First, the agents did not report uncertainty. Five of eight reported that our site has no server-side API at all, because the formatter page says processing happens in the browser. An undiscoverable API does not read as hidden. It reads as nonexistent, and that is what an agent will tell whoever sent it. Second, failing was expensive: the median failed run took 140 seconds, roughly double what a rung 1 success cost, with every request in the budget spent probing paths that were never going to answer.

One asterisk belongs in the record. In an earlier, less controlled run of this rung, a single agent found the endpoint by reading a public Next.js build manifest that lists every route on the site. The finding did not replicate under clean conditions, but it stands as a reminder that framework infrastructure can leak what a site does not advertise.

Rung 5: one docs page buys most of it back

We then added exactly one thing to the rung 4 terrain: a human-readable documentation page at /tools/api with the base URL and one worked example, linked from the tools hub in a single sentence. No machine layer, no manifest, nothing in .well-known.

Unaided success went from 0 of 8 to 6 of 8. Every success followed the same route a person would: the home page or sitemap to the tools hub to the docs page to the endpoint. And when the prompt included the docs URL, the way a developer might paste a link into a ticket, success was 8 of 8 at a median of 2 requests and 32 seconds, close to declaration numbers.

The two failures are the instructive part. Both spent their request budgets probing for machine manifests that did not exist, using names like agents.json, acp.json, and ai-plugin.json. One of them reached the tools hub, extracted the link to the docs page, and had no requests left to follow it. On a site with documentation but no manifests, the industry's own manifest conventions became the tax that caused failure.

The cost curve

Success is not the only metric. Here is what each discovery channel cost the agents that used it:

The declaration is simultaneously the most reliable channel and the cheapest. Human documentation recovers most of the reliability at about twice the request cost. Convention is free until the day your API is not exactly where convention expects it, at which point it is worth nothing.

What this means for your site

1. Publish a machine declaration. It is the only channel that was both reliable and cheap, and it is the only one that works when your API is not at a guessable path. Ours is generated from the same registry that runs the API, so it cannot drift.
2. If you have docs, make them reachable by crawling. A single linked page moved success from zero to six of eight. Most sites have this and do not think of it as agent infrastructure. It is.
3. Never publish a discovery surface that omits things. The half-empty manifest performed worse than no manifest. The same goes for stale references: prose on our site that mentioned a manifest we were not serving burned agent budgets at every rung. A false promise costs more than silence.
4. Know that absence reads as nonexistence. If agents cannot find your capability, they will not report it as hidden. They will report that it does not exist, and their operators will believe them.
5. Sign your responses if correctness matters. Verified receipts are the reason we can publish these numbers without asking you to trust agent self-reports. Two agents in round 5 used our verify endpoint on their own initiative before reporting success, which suggests the ecosystem is ready to consume provenance where it is offered.

Methodology notes, including the embarrassing parts

We committed to publishing the record as it happened, not as we would have liked it to happen.

• One model tier twice read the experiment's own source code from the local machine instead of testing the public site, and was disqualified both times. We rebuilt the harness so probes run from an empty directory with no source access, after which that tier ran clean. The lesson generalizes: if an agent can find an answer key, some will.
• The probe harness crashed our orchestration host three times before we settled on strictly sequential, one-process-per-probe execution. The final design ran 16 probes in 13 minutes without incident.
• Predictions were registered before each round ran. The prediction for rung 4, made before any probe launched, was "zero or maybe one success but probably total failure." The clean run came back 0 for 8.
• Model-tier request logs are self-reported and one tier consistently underreports its counts. Its successes are proof-verified regardless; its request counts are excluded from the medians above.

The site is back at full readiness as of this writing, and the full per-trial data, including every proof signature, is retained.

Replicate it yourself

The API described in this study is live on this site, and the declaration that found it is at the standard address. Any agent, or any reader with curl, can reproduce the headline result in two minutes: read https://www.clocktowerassoc.com/.well-known/agents.json, call the tool it describes, and replay the signed proof against the verify endpoint it advertises.

The first unsolicited replication arrived hours after we restored the declaration. An AI agent that had merely been told the API existed read the manifest, called the formatter, and then verified its own proof against our verify endpoint without being asked to, because the declaration said it could. It also burned one retry by assuming a response field name instead of reading the response in front of it, which restates this study's findings in miniature: a declaration can solve discovery in a single hop, and after that the remaining failures belong to the agent's own assumptions. Readiness fixes what the site controls. It cannot fix confidence, which is one more reason responses that matter should carry receipts.