ACVP Lab Workflow: From Vector Set to Evidence

Two NIST programmes are routinely blurred together. The Cryptographic Algorithm Validation Program (CAVP) validates implementations of approved algorithms and their components; NIST calls algorithm validation “a prerequisite of cryptographic module validation” [CAVP]. The Cryptographic Module Validation Program (CMVP), run jointly by NIST and the Canadian Centre for Cyber Security, validates whole cryptographic modules to FIPS 140-3, using independent NVLAP-accredited Cryptographic and Security Testing (CST) laboratories [FIPS 140-3].

• What is tested?

  Algorithm validation (CAVP)

  An algorithm implementation, by comparing its outputs with known or expected answers [CMVP Management Manual §1.4]

  Module validation (CMVP)

  A defined module against FIPS 140-3’s eleven requirement areas — module specification, interfaces, roles/services/authentication, software/firmware, operating environment, physical, non-invasive, SSP management, self-tests, life-cycle assurance, mitigation of other attacks — at one of four security levels [FIPS 140-3]

• Which algorithms?

  Algorithm validation (CAVP)

  Approved algorithms; NIST points to SP 800-140C and SP 800-140D for the lists [CAVP] [SP 800-140Cr2]

  Module validation (CMVP)

  The approved algorithms the module claims; the lab confirms each complies with all requirements of its standard, including those CAVP tests do not reach [CMVP Management Manual §2.6.2]

• How is it run?

  Algorithm validation (CAVP)

  Black-box, through the Automated Cryptographic Validation Testing System (ACVTS): the implementation is never sent to NIST [CAVP]

  Module validation (CMVP)

  A CST laboratory tests the module, reviews documentation and source, and submits a report to the CMVP validation authorities [CMVP Management Manual §2.6.2]

• What is published?

  Algorithm validation (CAVP)

  An algorithm validation certificate on the CAVP validation list [CAVP]

  Module validation (CMVP)

  A module certificate; its entry links to the CAVP certificates of the algorithms the module uses [CMVP Management Manual §4.9.3.5]

Where this Hub stands: it runs neither programme. It executes selected public reference vectors, standards tests, conformance cases and probes, and its results are never an ACVTS verdict or a CAVP/CMVP certificate (see the notice at the top of this page).

The module side (security levels, the Modules In Process queue, and the CMVP routes for adding ML-KEM or ML-DSA to an already-validated module) is taught in FIPS 140-3 Certification.

ACVP testing starts with the client, not the server. A registration is the client’s description of the capabilities of one or more algorithm / mode / revision combinations [ACVP spec §3.2.2]. ACVTS generates test cases that match those capabilities and hands the inputs to the implementation [CAVP]. Each algorithm in the registration becomes one test vector set (identified by vsId), made of test groups (tgId) that share properties, made of test cases (tcId) [ACVP spec §5.4].

The registration below is the ML-DSA sigGen example from the ACVP ML-DSA sub-specification, with its hash list shortened. Every array is a promise: the server will generate test groups for combinations of these values, and the implementation has to answer all of them [ACVP ML-DSA sigGen registration] [ACVP ML-DSA sub-spec].

{
  "algorithm": "ML-DSA",
  "mode": "sigGen",
  "revision": "FIPS204-tr1",
  "prereqVals": [{ "algorithm": "SHA", "valValue": "123456" }],
  "capabilities": [{
    "parameterSets": ["ML-DSA-44", "ML-DSA-65", "ML-DSA-87"],
    "messageLength": [{ "min": 8, "max": 65536, "increment": 8 }],
    "hashAlgs": ["SHA2-256", "SHA2-512", "SHAKE-256"],
    "contextLength": [{ "min": 0, "max": 2040, "increment": 8 }]
  }],
  "deterministic": [true, false],
  "externalMu": [true, false],
  "signatureInterfaces": ["external", "internal"],
  "preHash": ["pure", "preHash"],
  "keyFormats": ["expanded", "seed"]
}

• deterministic: [true, false] commits the implementation to both signing variants, including hedged groups that carry the server’s randomness [ACVP ML-DSA sub-spec].
• signatureInterfaces: ["internal"] commits it to answering ML-DSA.Sign_internal on a formatted message, and externalMu: [true] to accepting a message representative μ computed outside the module [ACVP ML-DSA sigGen registration] [FIPS 204 Alg. 7].
• prereqVals names prerequisite algorithm validations (here SHA) [ACVP ML-DSA sigGen registration].
• The revision is part of the identity: the sub-specification lists ML-DSA / sigGen as both FIPS204 and FIPS204-tr1, “a new test revision against the same standard” [ACVP ML-DSA supported revisions].

The ACVP specification’s minimal message flow has six exchanges [ACVP spec §12.4]:

1. 1

   Register capabilities POST /testSessions

   The registration lists each algorithm / mode / revision and the options the implementation supports. The server answers with a test session and one vector-set URL per algorithm.

2. 2

   Download each prompt GET /testSessions/{id}/vectorSets/{vsId}

   The prompt (the spec also calls it the request) holds test groups (tgId) of test cases (tcId). Exactly one prompt exists per vector set.

3. 3

   Submit the response POST …/vectorSets/{vsId}/results

   The client returns the implementation’s output for every test case, linked to the prompt by vsId.

4. 4

   Read the disposition GET …/vectorSets/{vsId}/results

   The server reports a disposition for the vector set and for each test case: passed, fail, incomplete, expired, missing or unreceived. “error” applies to the vector set only, not to individual test cases.

5. 5

   Certify the session PUT /testSessions/{id}

   Allowed only when the session is both publishable and passed. The request associates the module and its operational environment (OE) and lists any prerequisite validations.

6. 6

   Retrieve the validation GET /requests/{requestId}

   The certification is a request; once approved it yields the identifier of the validation record.

Sources for the steps: [ACVP spec §5.4] [ACVP spec §12.16.4] [ACVP spec §12.17.4].

Map this onto the Hub’s claim ladder and the boundary is exact: generating a response to an issued prompt is rung 7; the disposition is rung 8; a published validation record is rung 9. The Hub’s ACVP-format prototype stops at local import and response export — it has no ACVTS session, credentials, submission or verdict retrieval.

A test type describes the format of a test rather than its intention; several tests of the same type can cover different assurances [ACVP ML-DSA sub-spec]. The vocabulary is defined per algorithm, in each ACVP sub-specification:

• SHA-1 / SHA-2

  Test type

  AFT

  What the implementation must do

  Hash single, multiple and partial blocks of random data — checks chunking and padding [ACVP SHA sub-spec]

• SHA-1 / SHA-2

  Test type

  MCT

  What the implementation must do

  Monte Carlo Test: from one seed, run 100 × 1,000 chained hashes where each digest feeds the next message; aimed at state, allocation and error-handling faults [ACVP SHA sub-spec]

• SHA-1 / SHA-2

  Test type

  LDT

  What the implementation must do

  Large Data Test: hash a multi-gigabyte message described, not transmitted, by the prompt [ACVP SHA sub-spec]

• ML-KEM / keyGen

  Test type

  AFT

  What the implementation must do

  Generate a key pair from a provided seed (ML-KEM.KeyGen_internal) [ACVP ML-KEM sub-spec]

• ML-KEM / encapDecap

  Test type

  AFT

  What the implementation must do

  Encapsulate with a server-chosen m (ML-KEM.Encaps_internal) and return c and k [ACVP ML-KEM sub-spec]

• ML-KEM / encapDecap

  Test type

  VAL

  What the implementation must do

  Decapsulate valid or modified ciphertexts (ML-KEM.Decaps_internal), or decide whether an encapsulation key (FIPS 203 §7.2) or decapsulation key (§7.3) is valid [ACVP ML-KEM sub-spec]

• ML-DSA / keyGen, sigGen, sigVer

  Test type

  AFT

  What the implementation must do

  Key pair from a seed; signatures compared with the server’s known signature; validity decisions on modified signatures [ACVP ML-DSA sub-spec]

• SLH-DSA / keyGen, sigGen, sigVer

  Test type

  AFT

  What the implementation must do

  Key pair from SK.seed, SK.prf and PK.seed; signatures from provided keys and randomness; validity decisions on modified signatures [ACVP SLH-DSA sub-spec]

FIPS 203, 204 and 205 each split their algorithms into an external interface, which generates its own randomness, and an internal interface that takes the randomness as an input. Tests that must supply the random input use the internal interfaces; the current ACVP ML-DSA specification supports testing both the internal and the external interface [ACVP ML-DSA sub-spec §6.1.2]. Each standard says they should not be offered to applications other than for testing, because the random values shall be generated by the module [FIPS 203 §6] [FIPS 204 §6] [FIPS 205 §9].

• ML-KEM.Encaps

  Randomness

  32-byte m drawn inside the module; Encaps_internal(ek, m) takes it as input [FIPS 203 Alg. 17, 20]

  Consequence for testing

  Byte-exact testing needs the internal interface with the server’s m.

• ML-DSA.Sign

  Randomness

  Default hedged: rnd from an RBG. Optional deterministic: rnd = 32 zero bytes; that is the only difference [FIPS 204 §3.4, Alg. 2, Alg. 7]

  Consequence for testing

  Deterministic signatures can be compared byte for byte; hedged ones only if the test can inject rnd.

• SLH-DSA sign

  Randomness

  Default hedged: opt_rand = addrnd. Deterministic: opt_rand = PK.seed [FIPS 205 §9.2, Alg. 19]

  Consequence for testing

  Same split as ML-DSA.

FIPS 204 treats the variants differently for security, not only for testing: hedged signing helps mitigate side-channel attacks, the deterministic variant “should not be used on platforms where side-channel attacks are a concern and where they cannot be otherwise mitigated”, and implementing the hedged variant alone is sufficient for interoperability [FIPS 204 §3.4].

ACVP talks to an implementation through whatever interface the test harness can reach [ACVP spec §4.2, §5]. For the Hub that interface is PKCS#11 v3.2, on two WASM engines (C++ and Rust). An ACVP test group can only be answered if the interface takes exactly the inputs the group supplies — nothing coerced, nothing defaulted. Where standard PKCS#11 has no field for an input, this HSM accepts it as an optional test input outside the standard, and every result that used one says so.

• ML-KEM decapsulation (VAL)

  Interface path

  Import dk, then C_DecapsulateKey with CKM_ML_KEM (ML-KEM.Decaps) [PKCS#11 v3.2 §6.68]

  Hub disposition

  Executed; k is the derived key’s CKA_VALUE

• ML-KEM encapsulation (AFT)

  Interface path

  Standard PKCS#11 has no field for m: C_EncapsulateKey takes a public-key handle and a template [PKCS#11 v3.2 §5.18.8]. This HSM accepts m (32 bytes) as an optional test input, the CKM_ML_KEM mechanism parameter [FIPS 203 Alg. 17]

  Hub disposition

  Executed on both engines in the Playground suite and in Step 3: c and k are byte-compared with NIST’s, and each result is labelled “test input supplied (outside the standard)”. Step 3 first checks that the engine really uses the supplied m (the same m twice gives the same output, a different m a different ciphertext); if not, nothing is answered

• ML-KEM key checks (VAL)

  Interface path

  None: in the PKCS#11 v3.2 text read for this module the ML-KEM mechanism covers ML-KEM.Encaps and ML-KEM.Decaps [PKCS#11 v3.2 §6.68], and the FIPS 203 §7.2/§7.3 input checks have no operation of their own [FIPS 203 §7.2, §7.3]

  Hub disposition

  Executed in Step 3 from what these engines do when the key is imported (PKCS#11 itself does not require a key check at import; these engines run one): accepted means valid, refused with exactly CKR_ATTRIBUTE_VALUE_INVALID means invalid, and any other result is an error, never a guess. evidence.json keeps the return code for every key

• ML-DSA sigVer, external, pure / preHash

  Interface path

  CKM_ML_DSA (ML-DSA.Verify) or CKM_HASH_ML_DSA_<hash>, context in CK_SIGN_ADDITIONAL_CONTEXT [PKCS#11 v3.2 §6.67.5–6.67.7]

  Hub disposition

  Executed. SHA2-512/224 and SHA2-512/256 have no CKM_HASH_ML_DSA_<hash> mechanism [PKCS#11 v3.2 §6.67]; the generic CKM_HASH_ML_DSA takes them as its hash (CKM_SHA512_224 / CKM_SHA512_256) with the caller computing PH(M). Step 3 runs these through that route too, labelled “computed outside the engine”

• ML-DSA internal interface on M′ (externalMu false)

  Interface path

  CKM_ML_DSA implements FIPS 204 Algorithms 2 and 3 on the message M [PKCS#11 v3.2 §6.67.5], but the internal interface only ever signs μ = SHAKE256(tr ‖ M′, 64) [FIPS 204 Alg. 7, line 6]. The harness computes μ from the public key and M′ and uses the external-μ mechanism below — no extension

  Hub disposition

  Playground suite: executed on both engines (sigGen byte-match and sigVer). Step 3 runs the sigVer groups the same way, labelled “computed outside the engine”

• ML-DSA internal interface with external μ

  Interface path

  No PKCS#11 v3.2 mechanism. The Hub’s engines use a non-standard mechanism code, 0x403c, for it; FIPS 204 allows μ to be computed in a different module [FIPS 204 Alg. 7, line 6]

  Hub disposition

  Executed (Playground suite and ACVP-format prototype) where an engine advertises 0x403c, labelled vendor-defined in evidence.json

• ML-DSA sigGen, deterministic

  Interface path

  Import sk, sign with hedgeVariant CKH_DETERMINISTIC_REQUIRED [PKCS#11 v3.2 §6.67.5]

  Hub disposition

  Byte-comparable with NIST’s deterministic sigGen sample (Playground suite)

• ML-DSA sigGen, hedged

  Interface path

  Standard PKCS#11 has no field for rnd: CKH_HEDGE_REQUIRED demands hedging but offers no input for it [PKCS#11 v3.2 §6.67.5]. This HSM accepts rnd as an optional test input in an extended signing structure: the standard CK_SIGN_ADDITIONAL_CONTEXT fields, then pRandom, ulRandomLen and messageEncoding

  Hub disposition

  Playground suite: byte-comparable with NIST’s hedged sigGen, labelled “test input supplied (outside the standard)”

• SLH-DSA sigGen hedged, and sign / verify on the internal interface

  Interface path

  Standard PKCS#11 has no field for addrnd and no internal-interface mechanism [PKCS#11 v3.2 §6.69]. This HSM accepts addrnd and messageEncoding = internal (slh_sign_internal / slh_verify_internal) in the same extended structure [FIPS 205 §9.2–9.3, Alg. 19–20]

  Hub disposition

  Playground suite: executed on both engines, labelled “test input supplied (outside the standard)”

• ECDSA sigGen (AFT, NIST’s k)

  Interface path

  Standard CKM_ECDSA takes no parameter and draws k itself [PKCS#11 v3.2 §6.3]. This HSM accepts k as the optional CKM_ECDSA parameter

  Hub disposition

  Playground suite: r ‖ s byte-compared with NIST’s, labelled “test input supplied (outside the standard)”

• ML-DSA keyGen (AFT)

  Interface path

  CKM_ML_DSA_KEY_PAIR_GEN with the seed ξ supplied as CKA_SEED. The specification lists CKA_SEED among the attributes the mechanism contributes, so accepting a caller’s seed is engine behaviour, not a PKCS#11 requirement [PKCS#11 v3.2 §6.67.4]

  Hub disposition

  Byte-comparable public key (Playground suite), engine-specific path

ACVP builds negative cases into the vector sets. An implementation that answers “valid” to everything fails them:

• ML-DSA sigVer: the server modifies a valid signature — modified message, modified commitment, modified z, modified hint — and includes at least 3 tests per modification type [ACVP ML-DSA sub-spec].
• SLH-DSA sigVer adds modified R, SIG_FORS and SIG_HT, and signatures that are too long or too short [ACVP SLH-DSA sub-spec].
• ML-KEM VAL: modified ciphertexts must go through implicit rejection and still return a key; invalid encapsulation and decapsulation keys must be detected [ACVP ML-KEM sub-spec] [FIPS 203 Alg. 18, §7.2, §7.3].

Two kinds of negative testing sit outside ACVP. Requirements ACVP does not test — the sub-specifications list zeroization, DRBG strength and incorrect-length keys and signatures for ML-DSA [ACVP ML-DSA sub-spec] (the ML-KEM sub-specification repeats the ML-DSA list, a copy error, as it names signatures [ACVP ML-KEM sub-spec]). And API behaviour: wrong key type, wrong state, short buffers and other PKCS#11 return codes, which OASIS profile cases and product probes exercise [PKCS#11 Profiles v3.2].

Vocabulary the Hub holds itself to: one happy-path case is “sampled”, never “covered”; positive, negative, boundary and error-path coverage are reported separately, so a passing round-trip can never turn a negative-coverage cell green.

The Hub gives every result exactly one of eight evidence classes, each with the only claim a passing result of that class may make [PQC Today evidence policy]. The classes are the Hub’s policy, not a NIST taxonomy:

• NIST ACVP-Server reference sample
  nist-acvp-reference-sample

  Meaning

  Expected values copied from the public NIST ACVP-Server repository with immutable source identity.

  Permitted claim

  “Passes this public NIST ACVP-Server reference sample”

• ACVTS-issued vector
  acvts-issued-vector

  Meaning

  Prompt issued to an authorized ACVTS test session and executed without changing the prompt.

  Permitted claim

  “Executed ACVTS-issued vector set <id>; no validation claim until accepted”

• Published standard KAT
  published-standard-kat

  Meaning

  Expected values printed in a cited standard or consensus RFC.

  Permitted claim

  “Passes the cited standard's example/KAT”

• Independent oracle
  independent-oracle

  Meaning

  Expected values produced by a separately identified implementation such as OpenSSL.

  Permitted claim

  “Agrees with <oracle/version> for this case”

• Cross-implementation differential
  cross-implementation-differential

  Meaning

  Two implementations agree for the same inputs.

  Permitted claim

  “C++ and Rust agree for this case”

• Functional round-trip
  functional-round-trip

  Meaning

  Output produced by an implementation is consumed by the same or paired implementation.

  Permitted claim

  “Completes this round-trip; no external correctness claim”

• OASIS profile case
  oasis-profile-case

  Meaning

  An identified OASIS PKCS#11 Profiles test case is replayed.

  Permitted claim

  “Passes this named OASIS profile case”

• Product mechanism probe
  product-mechanism-probe

  Meaning

  A PQC Today-authored test of a cited PKCS#11 behavior.

  Permitted claim

  “Passes this product-authored probe”

Running the same case on two implementations or two platforms is valuable — a disagreement is always a finding. Agreement is weaker than it looks:

• Shared layers. In the Hub’s ACVP-format prototype, the C++ and Rust engines are driven by one dispatcher and one response builder. A defect in that shared code makes both engines agree and both be wrong, so dual-engine agreement is differential evidence, not independent validation. External expected values — a NIST sample, a published KAT — are what break the tie.
• Byte vs semantic equivalence. Hedged signatures differ on every run by design [FIPS 204 §3.4]; the comparison policy must be declared per operation.
• Environment identity. ACVP binds testing to an operational environment — “the specific hardware and/or software the client’s cryptographic implementation uses to run” — and certification associates the module with its OE [ACVP spec §5.4, §12.16.4]. A result from WebAssembly in a browser says nothing about a native build, a different CPU, or a hardware accelerator until that target runs the same cases and records its own identity.
• Confirm the execution path. An “acceleration enabled” setting does not prove that a test used the accelerator. A library may choose different implementations for different operations, or fall back to software. To claim hardware-path coverage, retain evidence showing which implementation handled the tested operation—for example, an implementation-specific execution trace or counter. If that evidence is unavailable, say that the execution path was not confirmed.
• Honest statuses. Cross-target reports keep pass, fail, unsupported, not run and not comparable apart; the last three never add up to a pass.

The Hub accepts a new expected-value vector only with its provenance. Every vector file under src/data/acvp/ must have an entry in the vector manifest, and the gate fails until it does. The manifest’s own header documents the procedure:

1. Add the vector file under src/data/acvp/.
2. Generate a skeleton entry with the maintainers’ vector-manifest tool — it records the file’s SHA-256 and one case record per test case.
3. Fill every TODO: the evidence class; the source (for NIST ACVP-Server material the commit, upstream path, retrieval date and upstream SHA-256); each case’s operation, parameters and positive/negative expectation; and a lineage record for any subset, rename or change of operation.
4. The maintainers then run the vector-manifest check and regenerate the validation counts.

Propose the test with the validation contribution issue template: it asks for the algorithm and parameters, the immutable source, the licence note and the expected behaviour, which is what a reviewer checks above.

Not a coder? Open a Data Suggestion issue with the source URL and the exact case, and the maintainer does the manifest work.