ACVP Lab Workflow: From Vector Set to Evidence

Two NIST programmes are routinely blurred together. The Cryptographic Algorithm Validation Program (CAVP) validates implementations of approved algorithms and their components; NIST calls algorithm validation “a prerequisite of cryptographic module validation” [CAVP]. The Cryptographic Module Validation Program (CMVP), run jointly by NIST and the Canadian Centre for Cyber Security, validates whole cryptographic modules to FIPS 140-3, using independent NVLAP-accredited Cryptographic and Security Testing (CST) laboratories [FIPS 140-3].

Where this Hub stands: it runs neither programme. It executes selected public reference vectors, standards tests, conformance cases and probes, and its results are never an ACVTS verdict or a CAVP/CMVP certificate (see the notice at the top of this page).

The module side (security levels, the Modules In Process queue, and the CMVP routes for adding ML-KEM or ML-DSA to an already-validated module) is taught in FIPS 140-3 Certification.

ACVP testing starts with the client, not the server. A registration is the client’s description of the capabilities of one or more algorithm / mode / revision combinations [ACVP spec §3.2.2]. ACVTS generates test cases that match those capabilities and hands the inputs to the implementation [CAVP]. Each algorithm in the registration becomes one test vector set (identified by vsId), made of test groups (tgId) that share properties, made of test cases (tcId) [ACVP spec §5.4].

The registration below is the ML-DSA sigGen example from the ACVP ML-DSA sub-specification, with its hash list shortened. Every array is a promise: the server will generate test groups for combinations of these values, and the implementation has to answer all of them [ACVP ML-DSA sigGen registration] [ACVP ML-DSA sub-spec].

{
  "algorithm": "ML-DSA",
  "mode": "sigGen",
  "revision": "FIPS204-tr1",
  "prereqVals": [{ "algorithm": "SHA", "valValue": "123456" }],
  "capabilities": [{
    "parameterSets": ["ML-DSA-44", "ML-DSA-65", "ML-DSA-87"],
    "messageLength": [{ "min": 8, "max": 65536, "increment": 8 }],
    "hashAlgs": ["SHA2-256", "SHA2-512", "SHAKE-256"],
    "contextLength": [{ "min": 0, "max": 2040, "increment": 8 }]
  }],
  "deterministic": [true, false],
  "externalMu": [true, false],
  "signatureInterfaces": ["external", "internal"],
  "preHash": ["pure", "preHash"],
  "keyFormats": ["expanded", "seed"]
}

• deterministic: [true, false] commits the implementation to both signing variants, including hedged groups that carry the server’s randomness [ACVP ML-DSA sub-spec].
• signatureInterfaces: ["internal"] commits it to answering ML-DSA.Sign_internal on a formatted message, and externalMu: [true] to accepting a message representative μ computed outside the module [ACVP ML-DSA sigGen registration] [FIPS 204 Alg. 7].
• prereqVals names prerequisite algorithm validations (here SHA) [ACVP ML-DSA sigGen registration].
• The revision is part of the identity: the sub-specification lists ML-DSA / sigGen as both FIPS204 and FIPS204-tr1, “a new test revision against the same standard” [ACVP ML-DSA supported revisions].

The ACVP specification’s minimal message flow has six exchanges [ACVP spec §12.4]:

1. 1

   Register capabilities POST /testSessions

   The registration lists each algorithm / mode / revision and the options the implementation supports. The server answers with a test session and one vector-set URL per algorithm.

2. 2

   Download each prompt GET /testSessions/{id}/vectorSets/{vsId}

   The prompt (the spec also calls it the request) holds test groups (tgId) of test cases (tcId). Exactly one prompt exists per vector set.

3. 3

   Submit the response POST …/vectorSets/{vsId}/results

   The client returns the implementation’s output for every test case, linked to the prompt by vsId.

4. 4

   Read the disposition GET …/vectorSets/{vsId}/results

   The server reports passed, fail, incomplete, expired, missing, unreceived or error for the vector set and for each test case.

5. 5

   Certify the session PUT /testSessions/{id}

   Allowed only when the session is both publishable and passed. The request associates the module and its operational environment (OE) and lists any prerequisite validations.

6. 6

   Retrieve the validation GET /requests/{requestId}

   The certification is a request; once approved it yields the identifier of the validation record.

Sources for the steps: [ACVP spec §5.4] [ACVP spec §12.16.4] [ACVP spec §12.17.4].

Map this onto the Hub’s claim ladder and the boundary is exact: generating a response to an issued prompt is rung 7; the disposition is rung 8; a published validation record is rung 9. The Hub’s ACVP-format prototype stops at local import and response export — it has no ACVTS session, credentials, submission or verdict retrieval.

A test type describes the format of a test rather than its intention; several tests of the same type can cover different assurances [ACVP ML-DSA sub-spec]. The vocabulary is defined per algorithm, in each ACVP sub-specification:

FIPS 203, 204 and 205 each split their algorithms into an external interface, which generates its own randomness, and an internal interface that takes the randomness as an input. The internal interfaces are the ones CAVP tests, and each standard says they should not be offered to applications other than for testing, because the random values shall be generated by the module [FIPS 203 §6] [FIPS 204 §6] [FIPS 205 §9].

FIPS 204 treats the variants differently for security, not only for testing: hedged signing helps mitigate side-channel attacks, the deterministic variant “should not be used on platforms where side-channel attacks are a concern and where they cannot be otherwise mitigated”, and implementing the hedged variant alone is sufficient for interoperability [FIPS 204 §3.4].

ACVP talks to an implementation through whatever interface the test harness can reach [ACVP spec §4.2, §5]. For the Hub that interface is PKCS#11 v3.2, on two WASM engines (C++ and Rust). An ACVP test group can only be answered if PKCS#11 can carry exactly the inputs the group supplies — nothing coerced, nothing defaulted.

ACVP builds negative cases into the vector sets. An implementation that answers “valid” to everything fails them:

• ML-DSA sigVer: the server modifies a valid signature — modified message, modified commitment, modified z, modified hint — and includes at least 3 tests per modification type [ACVP ML-DSA sub-spec].
• SLH-DSA sigVer adds modified R, SIG_FORS and SIG_HT, and signatures that are too long or too short [ACVP SLH-DSA sub-spec].
• ML-KEM VAL: modified ciphertexts must go through implicit rejection and still return a key; invalid encapsulation and decapsulation keys must be detected [ACVP ML-KEM sub-spec] [FIPS 203 Alg. 18, §7.2, §7.3].

Two kinds of negative testing sit outside ACVP. Requirements ACVP does not test — the sub-specifications list zeroization, DRBG strength and incorrect-length ML-KEM/ML-DSA inputs [ACVP ML-KEM sub-spec] [ACVP ML-DSA sub-spec]. And API behaviour: wrong key type, wrong state, short buffers and other PKCS#11 return codes, which OASIS profile cases and product probes exercise [PKCS#11 Profiles v3.2].

Vocabulary the Hub holds itself to: one happy-path case is “sampled”, never “covered”; positive, negative, boundary and error-path coverage are reported separately, so a passing round-trip can never turn a negative-coverage cell green.

The Hub gives every result exactly one of eight evidence classes, each with the only claim a passing result of that class may make [PQC Today evidence policy]. The classes are the Hub’s policy, not a NIST taxonomy:

Running the same case on two implementations or two platforms is valuable — a disagreement is always a finding. Agreement is weaker than it looks:

• Shared layers. In the Hub’s ACVP-format prototype, the C++ and Rust engines are driven by one dispatcher and one response builder. A defect in that shared code makes both engines agree and both be wrong, so dual-engine agreement is differential evidence, not independent validation. External expected values — a NIST sample, a published KAT — are what break the tie.
• Byte vs semantic equivalence. Hedged signatures differ on every run by design [FIPS 204 §3.4]; the comparison policy must be declared per operation.
• Environment identity. ACVP binds testing to an operational environment — “the specific hardware and/or software the client’s cryptographic implementation uses to run” — and certification associates the module with its OE [ACVP spec §5.4, §12.16.4]. A result from WebAssembly in a browser says nothing about a native build, a different CPU, or a hardware accelerator until that target runs the same cases and records its own identity.
• Honest statuses. Cross-target reports keep pass, fail, unsupported, not run and not comparable apart; the last three never add up to a pass.

The Hub accepts a new expected-value vector only with its provenance. Every vector file under src/data/acvp/ must have an entry in the vector manifest, and the gate fails until it does. The manifest’s own header documents the procedure:

1. Add the vector file under src/data/acvp/.
2. Generate a skeleton entry with the maintainers’ vector-manifest tool — it records the file’s SHA-256 and one case record per test case.
3. Fill every TODO: the evidence class; the source (for NIST ACVP-Server material the commit, upstream path, retrieval date and upstream SHA-256); each case’s operation, parameters and positive/negative expectation; and a lineage record for any subset, rename or change of operation.
4. The maintainers then run the vector-manifest check and regenerate the validation counts.

Propose the test with the validation contribution issue template: it asks for the algorithm and parameters, the immutable source, the licence note and the expected behaviour, which is what a reviewer checks above.

Not a coder? Open a Data Suggestion issue with the source URL and the exact case, and the maintainer does the manifest work.