Charlotte Code Audit, June 2026

Why this document is public

Charlotte is an open-source MCP server our founder maintains as an independent project, and in June 2026 we put it through the same kind of review we sell to clients. This is that report, edited only to remove internal working notes and to keep the language consistent with the rest of the site. The findings, their severities, and the remediation guidance are unchanged from the version that went to the maintainer.

We publish it because a firm that audits other people's systems should be willing to show what an audit of our founder's own code looks like, including the parts that are unflattering.

Scope and method

Version audited: v0.6.3, plus one unreleased commit, against a clean working tree on main.

Method: five parallel deep-read passes over the codebase, covering the core engine, the tools layer, the test suite, hardening and robustness, and project health. Every source file was read in full, and every finding was checked against the code before it was written down. One finding, the iframe bounds bug in Theme 2, was confirmed empirically against a live Chromium run rather than by reading alone.

Two findings were verified against the source but could not be confirmed empirically in the audit session, and they are flagged inline where they appear.

Status of the findings

The audit examined the released v0.6.3, so at the time of the audit every finding below was present in the version users could install. The project's changelog records fixes for most of the major findings under a version that has been prepared but not tagged or published. Nothing in this report should be read as "already shipped" until that release exists.

[PENDING VERIFICATION] The remediation state above is taken from the project changelog rather than from a released package. Confirm against the published release before this document goes live.

Summary

Charlotte has a coherent thesis, which is token-efficient structured pages for agents, and the thesis is executed as a system rather than as a slogan. Empty-field stripping, compact JSON, the interactive summary, consecutive-element collapsing, the output-file escape hatches, and the detail levels all pull in the same direction. The README's tool and profile counts are exactly correct against the code, which is unusual. Changelog discipline, CI design, and npm packaging are all above the bar for a solo open-source project.

The problems cluster into five themes rather than scattering as isolated bugs.

1. Silent failure is the dominant defect class. At least seven tools can report success while doing nothing, or while doing the wrong thing. For an agent-facing tool this is the worst available failure mode, because an agent can recover from an error but not from a lie.
2. One verified, user-visible correctness bug. Same-origin iframe element bounds are double-offset, so a coordinate click on iframe content lands in the wrong place.
3. The test suite largely tests the wrong layer. Of 43 tools, only 4 are ever exercised through their real MCP handlers. Most integration tests reimplement the production logic and then test the reimplementation.
4. There is no recovery path from a browser crash. The server wedges permanently until it is restarted.
5. --no-sandbox is hardcoded into every launch, including bare-metal installs, and the README never mentions it.

None of this is fatal to the project. Most of the major findings are days rather than weeks of work, and several of them, including the error-code sweep and the static-server boundary check, are mechanical.

What the audit found sound

These were verified against the code, not taken on trust.

• Atomic ID swap (src/renderer/renderer-pipeline.ts:148 with element-id-generator.ts:101-106). A fresh generator is created per render and swapped in via replaceWith() only on success, so a failed render never leaves half-populated ID state.
• Three-step stale-ID recovery in resolveElement (src/tools/tool-helpers.ts:178-211). It checks the map, re-renders once, then fails with a conservative "did you mean" suggestion. This is best-in-class agent UX for hash-ID staleness.
• The typing guard from issue #180 (src/tools/interaction-helpers.ts:37-86) is a model fix. Pure validation is hoisted above all browser work, the overhead factor is explained in a comment, the cap is documented in the schema so the model sees it before failing, and the error message carries three concrete remedies. It is boundary-tested in tests/unit/tools/typing-duration-guard.test.ts:50-56.
• resolveOutputPath (src/tools/tool-helpers.ts:407-445) does a lexical containment check before any side effect, then a realpath-based symlinked-parent check after mkdir, with symlink-escape tests at both the unit and integration layers. This is more thorough than most production servers. One leaf-symlink gap remains, recorded as H-8.
• Ignored-node reparenting in accessibility-extractor.ts:113-137 preserves accessibility tree connectivity through ignored nodes in linear time, and the nullable backendDOMNodeId invariant is honored at every consumer, with no violations found anywhere in the tree.
• The dialog race in waitForPossibleNavigation (interaction-helpers.ts:188-203). Recognizing that a JS dialog blocks the action promise indefinitely, and racing against it with a .catch() on the abandoned promise, is careful and non-obvious work.
• Stdout discipline is clean. There are zero console.* calls in src/, and the logger writes structured JSON to stderr only, so the stdio transport is safe.
• Tool counts check out. 42 registered tools plus the meta-tool gives 43, every group member matches a registration, and all six README profile counts are exact.
• CI is real and well built (.github/workflows/ci.yml). It runs on pull requests, splits static-check and test jobs, uses npm ci --ignore-scripts to skip the Chromium download in the lint job, and sets concurrency cancellation and timeouts.
• The npm tarball is clean, verified with npm pack --dry-run, with a working bin shebang and the version single-sourced from package.json. The v0.6.3 changelog entry is an honest postmortem of the stale-dist incident, with a prepublishOnly remediation.
• Docker posture is reasonable: a non-root user, no exposed ports, and dumb-init for signal handling. The static dev server binds to 127.0.0.1 explicitly.
• The best-tested area is state/ (differ, snapshot store, artifact store) at roughly 98.5% coverage with genuinely behavior-driven suites, and meta-tool.test.ts pins notification batching precisely.

Theme 1: silent failures

These findings share a root cause, which is that the result of an in-page or CDP operation is never inspected, so failure looks identical to success. An agent acting on the output then compounds the error.

Recommendation. Make "verify the effect" a house rule. Every Runtime.callFunctionOn and Runtime.evaluate call should check exceptionDetails, and every state-mutating tool should confirm the resulting state where that is cheap to do, such as the selected value, the checked state, or the navigation response object.

Theme 2: iframe bounds are double-offset

Major, empirically confirmed. For same-process (same-origin) iframes, getFrameSession() returns the page's own CDP session, so DOM.getBoxModel already returns coordinates in the main-frame viewport, but extractSingleFrame adds the content offset again. Verified against tests/fixtures/pages/iframe-parent.html, where the coordinates Charlotte reports for a button inside the iframe differ from the Puppeteer ground truth by exactly the iframe's content offset applied a second time. The offset is correct for out-of-process iframes, which have their own session and frame-local coordinates. The existing integration test at tests/integration/iframe.test.ts:117-132 only asserts a greater-than-or-equal relationship, so it passes either way. Any coordinate click on iframe content lands in the wrong place.

Fix. Apply the content offset only when the frame's session differs from the main page session, and tighten the integration test to assert coordinates within a few pixels of boundingBox().

Where. src/renderer/frame-discovery.ts:71-85 and src/renderer/layout-extractor.ts:73-80.

Theme 3: lifecycle and recovery

Theme 4: the test suite tests the wrong layer

Of 43 tools, only 4 (fill_form, dialog, one navigate path, and one observe path) are ever invoked through their real MCP handlers. The rest are tested by reimplementing the handler logic inside the test with raw CDP calls, and one test says so in its own comment at tests/integration/modifier-click.test.ts:74-76. Coverage corroborates this: src/tools sits at roughly 35.7% of lines, interaction.ts at 7.7% of functions, and the session.ts handler code at zero.

This has practical consequences. The one real bug ever caught by manual sandbox testing, back and forward returning null, lived in handler code, which is exactly the layer the automated suite bypasses. Findings S-1 and S-3 above survive for the same reason, because the tests assert that a call does not throw rather than asserting its effect.

Key findings:

• Critical. tests/integration/interaction.test.ts:97-155, session.test.ts (which builds a _deps object and never uses it, so every test calls Puppeteer directly and asserts nothing about Charlotte), and evaluate.test.ts (which tests Chromium's Runtime.evaluate rather than src/tools/evaluate.ts).
• Major. No end-to-end agent journey exists. Nothing drives navigate, observe, find, click, type, and submit through mcpClient.callTool. protocol.test.ts is excellent and is the template to copy, but it stops at a single round trip.
• Major (flakiness). There are 31 fixed setTimeout sleeps in dialog.test.ts alone, in the 100 to 200 millisecond range, which will fire on loaded CI runners. The codebase ships pollUntilCondition and the tests do not use it.
• Minor. tests/unit/browser/page-manager.test.ts launches a real Chromium despite living under tests/unit/, which contradicts the project's own contributor guidance and makes the unit suite require a browser.
• Minor. There are no direct unit tests for content-extractor, layout-extractor, accessibility-extractor, or frame-discovery, and the documented invariants around nullable backendDOMNodeId and role naming have no pinning tests.
• Minor (CI). There is no coverage run or threshold in CI, Puppeteer's Chromium is not cached so it is re-downloaded every job, the matrix runs Node 22 only while engines promises 20 and above, and tests/docker-smoke-test.mjs, a decent 303-line smoke test, is never invoked by any workflow.
• Teardown is otherwise solid. Every integration file closes its browser, HTTP servers use port 0, and no orphaned Chromium processes are possible by construction. The fixture pages are genuinely representative, covering three-level iframes, pseudo-elements, sequential dialogs, beforeunload, and popups.

Top fix. Extend the in-memory MCP client pattern from protocol.test.ts and the fill_form suite across all tool groups, and add one agent-flow.test.ts that drives the sandbox site purely through callTool. That single change converts roughly 40% of the integration suite from testing its own reimplementations to testing the product.

Theme 5: hardening

The severities here are calibrated to a locally-run developer tool, not to a public network service.

Theme 6: per-render CDP cost

Every interaction tool ends in a render, so render latency is the product's latency.

Theme 7: API design for agent consumers

• Major, error-code taxonomy misuse. SESSION_ERROR is used for at least seven pure argument-validation failures, covering scroll amount, key and keys exclusivity, a missing wait condition, an upload file that does not exist, a screenshot argument conflict, and two dev-mode validations. INVALID_ARGUMENT exists and is used exactly once. An agent that treats SESSION_ERROR as "the browser is broken, restart it" gets the wrong recovery signal. The sweep is mechanical, low risk, and high payoff.
• Major, single-use selector IDs. The dom- IDs returned by charlotte_find in selector mode are registered on the live generator, which every render wipes when it swaps the generator, and every interaction ends in a render. A second use fails with a misleading suggestion, and they never work with fill_form because it pre-renders. Either persist them across swaps, re-run the originating selector on resolve, or document the single-use semantics.
• Minor. The fill_form checkbox and radio semantics toggle rather than set, so sending a true value to an already-checked box unchecks it, when agents express desired state rather than toggling. The documented claim that no fields are changed if one field is invalid is also violated at the mutation stage, because a mid-list failure leaves earlier fields applied.
• Minor. charlotte_toggle is charlotte_click minus the dialog and navigation guards plus a sleep, with no type validation, which leaves two overlapping tools where the more specific one is the worse one.
• Minor. charlotte_key sequences lack the duration guard added for issue #180, so a long key sequence with a per-key delay can hang for minutes, and a full-speed type of a very large string is similarly unguarded because per-character round trips still cost a few milliseconds each.
• Minor. Partially enabled groups are invisible. Under the browse profile, 6 of 13 interaction tools are disabled, but the server instructions only list fully disabled groups, so an agent that needs fill_form has no discoverability path short of calling charlotte_tools on a hunch.
• Minor. Three tools bypass formatPageResponse. Two of them return pretty-printed, unstripped JSON, and dialog nests its payload under a unique key. Adding an optional merge parameter to formatPageResponse would fix all three.
• Minor. Timeout responses from wait_for embed the unstripped representation, which makes the error path the most verbose response in the codebase, in exactly the place where clarity matters most.
• Nit. charlotte_screenshot is always full-page, with no viewport-only option.
• Nit. In submit-button detection, the properties check is dead code because CDP accessibility properties never include a type field, so detection degrades to looking for the word "submit" in the label, and a form whose button reads "Sign in" gets a null submit target.

Theme 8: element ID design

This is a design decision worth making deliberately rather than a bug.

• The 16-bit hash, rendered as four hex characters, gives roughly a 50% chance of at least one collision on a page with about 300 elements. Collisions are handled with numeric suffixes, but the suffix depends on traversal order, so if the base element disappears, the former suffixed element silently becomes the base ID. An agent's cached ID then resolves to a different element, and the differ reports phantom add and remove pairs. Widening to six or eight hex characters costs very little in tokens and reduces collisions by orders of magnitude, and salting the disambiguator into the hash stops suffixed IDs from migrating onto the base.
• Because the accessible name is part of the composite key, a label change re-IDs the element, which makes the differ's label and text change branches mostly dead code. A button going from "Follow" to "Following" reports a remove and an add rather than a change. Either document that as intended, or pair removed and added entries by type and DOM path in a second pass.
• Sibling index and nearest labelled container are content-derived, so inserting one list item re-IDs every following sibling of the same role. That is inherent to the design, but it is undocumented as a source of churn.
• Reclassified file inputs keep the button prefix while the type-prefix map says otherwise. The IDs are stable but misleading, and the similarity matcher, which is prefix-driven, will never match them.

Theme 9: documentation and release drift

• Major. The README's benchmark section is labeled with a version it was not measured on, and the underlying raw results are from two considerably older versions. One headline comparison also silently excludes the one site where the tool does not come out ahead. The harness is real and committed, so the right remediation is to re-run it against the current release and relabel the table, or to caveat the existing numbers honestly. A separate cost claim elsewhere in the README has no backing data anywhere in the repository and should be cut or substantiated.
• Major. server.json declares a version several minor releases behind what is published on npm. If that file feeds the MCP registry, the registry advertises a months-old version. Bump it and add it to the release checklist.
• Minor. The README links a comparison tool to the wrong GitHub organization.
• Minor. Tool counts are stale in prose. The launch guide claims 39 tools and two detail levels when there are 43 and three, and the contributing guide claims 30. Either stop committing counts to prose, or generate them.
• Minor. The contributing guide never mentions the lint and format-check commands, but CI fails pull requests on both, which guarantees friction on a first contribution.
• Minor. Running the full Chromium integration suite in prepublishOnly is the right instinct after the stale-dist incident, but it makes hotfix publishing hostage to a flaky integration test on one machine. Consider building and running unit tests locally, and moving publishing to a tag-triggered GitHub Action, which eliminates the entire stale-local-build failure class.
• Nit. One README heading renders incorrectly because of a missing blank line, the root deployment config appears to be dead depending on dashboard settings, the site subdirectory README is unmodified framework boilerplate, the sandbox test log is a stale point-in-time record, one release has no git tag so its Docker publish never ran, and one merged pull request is missing from the unreleased changelog section.

Strategic observations

These are about direction rather than individual defects.

1. The testing strategy reimplements and then asserts. This is the one structural mistake in the project. Copying production logic into tests produces a suite that passes and that reports reasonable coverage of the renderer, but that cannot catch handler bugs, and the silent-failure findings survived precisely because of it. The right pattern already exists in the codebase, in protocol.test.ts and the fill_form suite, and it simply never propagated. This is the highest-value engineering investment available.
2. Mutation paths assume success. The architecture renders after every action and reports the new state, which creates the impression that an agent will notice when a page did not change. In practice agents trust a tool's success signal far more than they re-derive state from a diff, so tools need to verify their own effects.
3. ID stability was traded for ID readability. Putting the accessible name in the hash key and using four hex characters optimizes for short, human-readable IDs, at the cost of identity churn from label changes and collision migration, which quietly degrades the differ, one of the project's headline features. The fix is cheap; the decision just needs to be made consciously.
4. --no-sandbox became a convenience default. It was the path of least resistance during the Docker work and then leaked into every install mode undocumented. For a tool whose pitch includes sending agents to arbitrary URLs, sandbox-off-by-default is the wrong side of that trade.
5. Tools have proliferated at the margins. Toggle versus click, wait-for versus find plus polling, and three response shapes for the same data are each individually defensible, but together they create surface area for agent confusion. The profile discipline is good, and the same discipline should be applied to overlapping tools.

What the project got right strategically is worth stating too. The token-efficiency thesis is executed end to end rather than asserted, the accessibility-tree-first representation is the correct bet against screenshot and DOM dumps, the profile and meta-tool system is well designed and internally consistent, and the project health fundamentals, meaning the changelog, CI, packaging, and issue templates, are better than most funded projects manage.

Prioritized remediation plan

Now, because an agent will hit these this week.

1. The iframe bounds double-offset in Theme 2, which is verified, user-visible, and a small fix.
2. The charlotte_select exception check (S-1), plus a sweep of all callFunctionOn sites.
3. charlotte_submit moving to requestSubmit() (S-2).
4. A re-render before resolve in the wait_for exists path (S-4), and real feedback for function results and exceptions in evaluateCondition (S-5).
5. Browser-crash recovery: a try/catch in closeTab, and an onDisconnected hook that resets the page manager (L-1).

Next, within a focused week.

6. The error-code sweep to INVALID_ARGUMENT, which is mechanical.
7. The dialog-race guard on type, toggle, select, key, and fill_form (L-2).
8. Network.enable plus session reuse in charlotte_network, with a test that asserts blocking actually happens (S-3).
9. Batched file-input reclassification, constant-time form-field lookup, and node ID deduplication (P-1 through P-3).
10. Response-size caps on the pipeline arrays, full content, and evaluate results (H-2, H-3).

Then, structurally.

11. Propagate the in-memory MCP client pattern across the integration suite, add one end-to-end agent-flow test, add coverage to CI with a threshold on the tools directory, and cache Chromium in CI.
12. Sandbox on by default with a documented opt-out (H-1), and a real seccomp profile in the compose file (H-6).
13. Widen the element ID hash and salt the disambiguators (Theme 8).
14. Re-run the benchmark harness against the current release, fix the README labeling, bump server.json, and sweep the stale counts (Theme 9).
15. Decide and document the agent-trust boundary for the paths charlotte_configure accepts (H-7).