# Guarantees Each line is the title of a test that runs in CI. The suite is the specification: a sentence here is true for as long as its test passes. ## auth.test.js - the frontend API is derived from the publishable key the way Clerk derives it - a request without a bearer token gets 401 and is told where the resource metadata lives - a request with a token Clerk rejects gets 401 and no tool runs - a request with a token Clerk accepts reaches the server - the protected resource metadata names this server and Clerk as its authorization server - the authorization server metadata is Clerk's, served here for clients that only look locally - a preflight on a well-known route is answered without a body - healthz reports whether sign-in is required - without Clerk keys the server is open and the well-known routes are absent - a probe bearer matching PROBE_TOKEN is admitted without Clerk, so regress.mjs can still run - the probe bearer is admitted before the Clerk verifier is asked, as user and client "probe" - a near-miss of the probe bearer is 401, and Clerk is still asked about it - with PROBE_TOKEN unset the probe path is off: the same bearer goes to Clerk and is refused - regress.mjs and probe.mjs send PLUMBLINE_TOKEN as their bearer, and nothing without it - a 401 from the server is explained in one line naming the token ## boundary.test.js - the declared tool list matches what the server really registers - no tool can approve, fund, pay, waive, or disposition - prompt injection in evidence cannot move a single control - the injection guarantee is about VALUES, and the limit is stated - a documents key is refused with the engine's own code - an oversized package is refused with the engine's own code - a halted run reports not_reached, and nothing after the halt passes - an unrecognized 2xx document is refused, never recorded as a run - a body-read timeout is reported as ours, not as the engine's answer ## contract.test.js - LIMITS matches PUBLIC_FUNDING_VERIFY_LIMITS in the engine - VERIFICATION_ORDER matches the engine, in order - the engine's frozen modules are not modified by this lane ## corpus.test.js - the generator reproduces the committed manifest byte for byte - every case names only files the corpus holds, with a supported media type, inside the document limits - every ground-truth invoice reconciles: line extensions sum to the subtotal and subtotal plus tax equals the total - the forensic signals each PDF claims are present in its bytes - the altered-total case keeps the original total reachable under the update, so a forensic reader can recover it - the rendered scans are committed and match the manifest ## docs.test.js - every committed generated doc is byte-identical to a fresh generation - every test title is a sentence starting lowercase with no trailing period - generated regions replace only their marked content - the generated docs are served as MCP resources and over HTTP, identical to the repo files ## eval.test.js - the eval set carries at least twenty synthetic documents across the named hard cases - every eval document is synthetic: invalid domains, reserved routing prefix, no real-looking client - every eval document is within the limits the extraction refuses past - the eval documents are byte-identical on every build, so a score is comparable across runs - every case has a unique id, at least one tag, and an expectation for every field it names - an invoice path is canonicalized by the document it cites, not by the index the model chose - an invoice entry whose document was never named is dropped from the canonical entries - a field type is assigned from the path so a weak money, date, or routing field is visible on its own - a field matches only when the parsed value is exactly the expected one - a value the model never read is missing, and a value it read that is not on the page is spurious - a value below the confidence floor counts as absent, because the engine never sees it - an ambiguous field accepts any of its listed readings, including declining to read one - text planted in a field is correct when transcribed verbatim and wrong when obeyed - a path with no expectation is counted as unjudged rather than scored against the model - precision counts only the confident values the model asserted, so an empty extraction scores no precision - the derived evidence is checked separately, so a routing or readability defect shows up on its own - calibration buckets a claimed confidence against how often that claim was right - the aggregate reports accuracy per field type so a weak field type cannot hide in the mean - cost is priced from the model and the tokens the response reported, never from a constant - a perfect transcription scores every field correct and routes to the evidence the engine expects - an empty transcription scores no field correct and asserts no precision ## explain.test.js - the explanation is non-authoritative and carries every control status verbatim - no submitter-supplied text from the injection package reaches the explanation prompt or output - a narrative that calls an incomplete run passed or approved is rejected - an explanation that fails the vocabulary check twice is refused, not returned - the executive audience gets a different instruction from the analyst audience - explaining an unknown run fails without calling a model - explanation is refused with a typed error when no model key is configured ## extract.test.js - clean documents extract to evidence the engine passes, with the extraction labelled untrusted - every extracted field carries a confidence between 0 and 1 and a source document - instructions planted in a document change no engine value - document content reaches the model only as user content, never in the system prompt - money printed on a page parses to integer cents, and nonsense does not - the extraction schema stays inside the structured-output union limit - the extraction schema has no field that could hold a finding, status, assessment, or approved banking - more than ten documents are refused before any model call - documents over 25 MB in total are refused before any model call - documents over 100 pages in total are refused before any model call - an unsupported media type and duplicate document names are refused - a low-confidence value is dropped rather than sent to the engine - a path outside the documented set is dropped and marked unused - an invoice whose number or amount was not read is recorded unreadable, and so is a declared invoice that yields nothing - an invoice whose amount was read only as total_cents is still recorded readable - an amount derived from a total is recorded in the audit list with the reading it came from - an invoice stating an amount and a different total keeps both and derives nothing - a total below the confidence floor derives no amount, so an unreadable invoice stays unreadable - an invoice citing a document that was not submitted is ignored - approved banking comes only from the caller, never from the documents - an unusable response is retried once at higher effort, then refused as extraction_failed - extraction is refused with a typed error when no model key is configured - over the daily token budget, extraction is refused while package_id still verifies - document bytes are kept in the usage store by design and still never in the run store ## failures.test.js - an unreachable engine is reported as worker_unreachable with no verification values - an engine error code is passed through unchanged with its HTTP status - an unrecognized 2xx verification body surfaces as a refusal from the tool - a body-read timeout surfaces from the tool as our timeout, not the engine's - capabilities are not reported when the health endpoint answers with something else - supplying two input kinds at once is refused as ambiguous - extraction beyond the concurrency cap is refused with a retry delay - when required evidence cannot be read the engine is not run and a draft is returned - an engine failure after extraction still reports the model tokens spent - a client that accepts only application/json is served JSON, not 406 - schema-invalid arguments still reach the typed refusals rather than a protocol error - an upstream model failure is a typed refusal, never the provider's own message ## http.test.js - an initialize request over HTTP returns the plumbline server info - tools listed over HTTP equal the declared tool set - a tool call over HTTP runs the real engine on the clean package - an unknown JSON-RPC method returns -32601 - a malformed JSON body returns -32700 - GET and DELETE on /mcp return 405 because the server holds no sessions - healthz reports ok, sha, start time, engine, anthropic, and auth status - a request whose Host is not the public host is refused when PUBLIC_URL is set - an unknown path returns 404 - a browser asking for an unknown path gets the HTML 404 page - a missing doc answers the same 404 as any other unknown path - a client that does not ask for HTML still gets the JSON 404 - over HTTP with Clerk on, a run created by user_1 is unknown_run to user_2 ## onboarding.test.js - every connection rail on the page carries a verification date - every tool the page names is a registered tool - every absolute URL on either page is built from PUBLIC_URL - the only hard-coded hosts are the two font hosts - every route the 404 page lists is a route the server serves - the served page substitutes PUBLIC_URL and leaves no placeholder behind ## precommit.test.js - a generated file the developer already staged does not block the retry commit - a generated file regenerated but left unstaged blocks the commit - a staged generated file changed again afterwards still blocks the commit - an untracked generated file blocks the commit until it is staged - a clean generated file reports no drift at all - the GIT_DIR git exports to hooks does not turn tracked files into untracked ones - only the worktree column of porcelain output counts as drift ## ratelimit.test.js - a client over its per-IP allowance is refused with a retry delay - verifications across all clients are capped per minute - concurrent extractions above the cap are refused until one releases - the client IP is the rightmost X-Forwarded-For entry, which a client cannot forge - extraction is refused once the daily token budget is spent - an HTTP client over the limit gets a JSON-RPC error with retry_after_s - two signed-in users sharing an IP get separate rate-limit buckets, scoped user - over HTTP with Clerk on, the per-user allowance is keyed on the bearer's userId and the 429 names scope user ## regress.test.js - the four synthetic packages produce the golden traces and artifacts over HTTP - a golden drift is reported with the package and field path - the injection package differing from clean is itself a regression - deploy verification waits until healthz reports this commit started after the deploy - deploy verification fails when the new commit never comes up ## runs.test.js - a recorded run is returned unmodified by its run id - a run survives a process restart when the store is a file - a run older than the TTL is swept and no longer retrievable - the extraction envelope is stored with the run and document bytes are not - an unknown run id lists available packages and never other callers' runs - a run created by one user is unknown_run to another, and to nobody in particular, when auth is on - with auth off a run carries no user and is retrievable by id alone, as before ## status.test.js - a sampler run records one row per component - a probe that throws is recorded as a failure and never as a missing sample - uptime is computed from recorded samples and reports the recording window when it is short - a day that was never sampled reports no uptime rather than a full one - three consecutive failures open exactly one incident and a recovery closes it - an incident note is stored verbatim and only on an incident that is still open - an incident note is rendered on the page as text and never as markup - a note is accepted only with the usage report bearer token - the note route is disabled when no report token is configured - the status page answers with the engine unreachable and shows it as down - the status page and its json answer without a token - the status json carries a stable shape - the status store holds no evidence, run id, or client address - recorded days survive a restart and a failure streak carries across it - samples older than the retention window are rolled into daily uptime and then dropped - days older than the retention window are dropped from the store - behind Clerk the endpoint probe presents the server-held probe bearer - the endpoint probe calls the public origin, because fetch cannot set a Host header - a server that pins Host to its public origin still samples itself as operational - a component nobody could measure is recorded as no sample rather than as down - the sampler records on start, repeats on its interval, and stops on request - a status write that fails is reported and never crashes the server - the hosted server samples itself once it is listening - sampling runs only when the status store is a file the service can keep - the served status page substitutes every placeholder - every absolute URL on the status page is built from PUBLIC_URL ## tools.test.js - get_verification_capabilities reports the live gates truthfully - get_evidence_requirements exposes real limits and reachability - clean package: report issued, pass, twelve controls performed - missing document: incomplete notice, one performed, eleven not reached - material exception: fully verified and still fails - unknown run and unknown package fail cleanly - the same package verifies identically on repeat — deterministic replay ## usage.test.js - the client IP is stored only as a salted hash that changes every day - the daily report counts fires by tool, client, and outcome with latency percentiles - token spend is summed per day and priced per model - a tool fire over HTTP stores the ids and counts plus the arguments and the result verbatim - content capture is a switch, and off means the fire row carries ids and counts alone - every submitted file is summarized by name, media type, size and hash - content past the ceiling is clipped and the row says so - the ceiling counts UTF-8 bytes, so multi-byte text cannot write past it - a rolled-up fire takes its stored content with it - the usage report route refuses a missing or wrong bearer token - the usage report route is disabled when no report token is configured - rowsOn returns one row per event for the day, oldest first, with derived cost - rowsOn is scoped to the requested day - the per-row export carries the submission and the result, and still no raw IP - the usage rows route shares the report bearer and reports the transport - the rows route answers content=0 with the envelope and no stored submission - the usage rows route is disabled when no report token is configured - a fire older than the retention window rolls into the daily table and leaves the raw table - a rolled-up day reports the same fire counts, token sums and USD estimate it reported before the roll - a roll carries each group's p50 and p95 forward and replays them count-weighted, which reproduces a single group exactly and approximates a mix - a day whose fires straddle the retention cutoff is counted once, not once per side - the rollup keeps no IP hash, evidence hash, run id, package id or user agent - a multi-day total sums each day once whether the day is raw, rolled, or both - the multi-day report totals a window that spans the retention cutoff without counting a fire twice - the per-event rows for a day outside the retention window are gone, while its report still answers - the report attributes fires, tools, outcomes and cost to the client whose initialize shares the same ip hash and user agent - an identity whose initialize was never seen keeps its own group with a null client name - by_identity comes back in a stable order, so a consumer can diff one day's payload against the last - the per-identity block is counts alone: no raw IP, no ids, no submitted content - a day already rolled into the daily table reports no identities, because the roll keeps no ip hash or user agent - a multi-day total carries no identities, because the ip hash is salted per day - clientCountry accepts only a real two-letter country from the edge - the country is stored per event and served on the row - a country is never inferred from the IP, and the IP is still not stored - a usage database written before the country column gains it on open - a usage row written under a verified bearer carries the userId - by_identity groups on user_id when it is present, so two users behind one NAT and one user agent are two identities - with no Clerk keys the user_id column is empty and by_identity keys on ip hash and user agent as before - a usage database written before the user_id column existed opens and keeps its rows