QA Evidence Workflow

Use this page when you need to move from individual run evidence to a bundled artifact you can bring to a release review or attach to a QA record.

BenchCI produces structured verification evidence for each hardware test run. Workspaces can package multiple runs into a release evidence bundle — a self-contained ZIP with per-run evidence, a requirement coverage matrix, review history, LCOV summaries, and a hash manifest.

What BenchCI does not do: The bundle is a structured record. It is not a product certification and it does not replace your company’s sign-off procedure. Whether a bundle meets a specific customer or release requirement is a judgment call made by your team and your QA process.

What is a release evidence bundle?

A release evidence bundle packages the runs you choose into a single ZIP file. It includes:

release-evidence-v1.2.3.zip
  release-summary.json      ← bundle metadata and coverage counts
  release-summary.html      ← human-readable bundle summary report
  coverage-matrix.json      ← full per-run requirement/test rows
  coverage-matrix-latest.json
                            ← latest result per requirement/test pair
  coverage-matrix.csv       ← full matrix, spreadsheet-friendly
  coverage-matrix.html      ← standalone full coverage table
  HASHES.txt                ← SHA256 of every file in the bundle
  runs/
    <run_id>/
      evidence.json
      evidence.html
      manifest.json
      external-artifacts/
        ...

Inline artifacts imported from external JUnit/CTRF workflows are included under the corresponding run when present. release-summary.json and release-summary.html also include the review state, lock timestamp, hash-linked review events, included-run DUT identity, fault summary counts, and LCOV coverage summary.

Each bundle is workspace-permissioned. Only members of your workspace can list, download, or view bundles.

How to create a bundle (CLI)

The benchci releases command group manages bundles.

1. Find the run IDs you want to include

benchci runs list

Run IDs appear in the RUN ID column. You can also copy run IDs from the dashboard Runs page.

Only terminal runs (status done or failed) can be included. Up to 100 runs per bundle.

2. Create the bundle

benchci releases create "Firmware v1.2.3 Release" \
  --version "1.2.3" \
  --runs 4c1e9b7a2fd3,7b3fa1c8e921 \
  --description "Pre-release validation runs against REQ-BOOT and REQ-COMM"

--version and --description are optional. The bundle name is required (positional) and --runs is required.

The CLI prints the bundle ID on success:

Bundle created: bnd_a3f9c2e1b047
Status: ready

Bundle generation is synchronous and completes before the command returns. For typical release bundles (2–20 runs) this takes a few seconds.

3. Download the bundle

benchci releases download bnd_a3f9c2e1b047

The ZIP is written to the current directory as benchci_release_bnd_a3f9c2e1b047.zip unless --out is provided.

Other CLI commands

benchci releases list              # list all bundles in your workspace
benchci releases show bnd_a3f9c2e1b047   # show bundle metadata
benchci releases report bnd_a3f9c2e1b047 --format html
benchci releases report bnd_a3f9c2e1b047 --template iec-62304-style --format pdf

How to create a bundle (Dashboard)

  1. Open the Releases view in the BenchCI dashboard.

  2. Click Create bundle.

  3. Enter the bundle name, optional version and description, and paste or type the run IDs to include.

  4. Click Create. The bundle appears in the list with status ready when generation completes.

  5. Click Download to save the ZIP.

The Releases view also shows the coverage summary, included runs, DUT and LCOV context, review state/history, and report downloads.

Review and sign-off

New bundles start in draft.

The dashboard supports this review flow:

draft or rejected
        ↓ submit with comment
under_review
        ↓ approve or reject with comment
approved or rejected

Workspace developers, admins, and owners can submit a bundle for review. Workspace owners and admins can approve or reject it. Each action requires a comment and creates a hash-linked review event. The bundle ZIP is regenerated after review changes so the downloaded package contains the current review history.

Approved bundles are locked and cannot be removed through the normal dashboard or API delete path. Owners and admins may remove non-approved bundles. A run included in any release bundle cannot be removed until that bundle is removed.

Human-readable release reports

Generate HTML or PDF reports from the same evidence used by the bundle:

benchci releases report bnd_a3f9c2e1b047 \
  --template generic-qa \
  --format html \
  --out release-report.html

benchci releases report bnd_a3f9c2e1b047 \
  --template iso-26262-style \
  --format pdf \
  --out release-report.pdf

Available templates are:

  • generic-qa

  • iec-62304-style

  • iso-26262-style

Reports include release metadata, review state/comments, run summaries, DUT identity, firmware hashes, LCOV summaries, traceability rows, fault summaries, and external-result source labels. A draft, under-review, or rejected report is labeled with that state.

These reports are evidence review aids. They are not official assessments, certification, or regulatory approval.

How requirement coverage works

The coverage matrix is built from traceability metadata in your suite.yaml.

Connect tests to requirements

Add requirement_ids to each test case:

tests:
  - name: boot_ok
    test_case_id: TC-BOOT-001
    requirement_ids:
      - REQ-BOOT-001
      - REQ-BOOT-002
    steps:
      - expect_uart:
          node: dut
          transport: console
          contains: "[BOOT] OK"
          within_ms: 3000

  - name: comm_latency
    test_case_id: TC-COMM-001
    requirement_ids:
      - REQ-COMM-001
    steps:
      - send_uart:
          node: dut
          transport: console
          data: "PING\n"
      - expect_uart:
          node: dut
          transport: console
          contains: "PONG"
          within_ms: 1000

test_case_id and requirement_ids are optional at the test level. You can also set requirement_ids at the suite level to apply to all tests in the suite.

See Suite Configuration for the full traceability field reference.

How the matrix is built

When you create a bundle, BenchCI parses the evidence from each selected run and writes two machine-readable views:

  • coverage-matrix.json contains the full per-run matrix: one row per (run_id, requirement_id, test_case_id).

  • coverage-matrix-latest.json contains the latest result per (requirement_id, test_case_id) pair. This is the default view used by the dashboard and /coverage API.

  • Result states are pass, fail, or not_run.

  • If a requirement appears in multiple runs, the most recently created run wins.

not_run means the traceability record exists but the test did not produce a passing or failing result, for example because it was skipped or interrupted by an infrastructure failure. Requirements that never appear in the selected runs’ evidence are absent from the matrix; BenchCI does not infer a master requirement list from source files that were not part of the indexed evidence.

Coverage rows include:

  • requirement_id

  • test_case_id

  • test_name

  • result and latest_result (pass / fail / not_run). For full rows, result is the per-run result; latest_result is kept for backward compatibility.

  • run_id and run_status of the run that produced the row

  • DUT identity fields and attached LCOV summary when available

  • firmware_sha256 and suite_version from that run

  • last_run_at timestamp

Coverage is bundle-scoped

The matrix reflects only the runs you included in the bundle. It is a snapshot, not a live cross-run view.

If a traced test is present but was skipped or did not reach a valid test result, its requirement/test pair can show as not_run. If a requirement is missing entirely, confirm that it was included in the submitted suite and preserved in the selected runs’ traceability evidence.

How to share the bundle with your QA team

The downloaded ZIP is self-contained. You can:

  • Attach it to a QA review ticket or release review checklist.

  • Upload it to a shared drive or document management system.

  • Open coverage-matrix.html in a browser — it is a standalone full per-run table with no external dependencies.

  • Import coverage-matrix.csv into a spreadsheet for QA managers who prefer tabular review.

  • Use coverage-matrix-latest.json when an automated consumer only needs the latest result per requirement/test pair.

  • Attach HASHES.txt alongside the ZIP to let reviewers verify the bundle was not modified after generation.

  • Preserve the approved ZIP rather than recreating a new bundle when the review event history is part of the release record.

The release-summary.json file at the top of the ZIP contains the bundle metadata, review state/history, coverage and LCOV counts, and run list. It is the starting point if you are building automated processing on top of bundles.

Workflow example

A typical pre-release validation workflow:

# Trigger CI validation runs across your suite
benchci run --cloud --bench-id dut-stm32 \
  --suite suite.yaml --artifact build/v1.2.3.elf

# Run finishes: get the run ID from output or dashboard
benchci runs list

# Create the release bundle
benchci releases create "STM32 v1.2.3 pre-release" \
  --version "1.2.3" \
  --runs 4c1e9b7a2fd3,7b3fa1c8e921,9d2ae3f01b44

# Download the ZIP
benchci releases download bnd_a3f9c2e1b047

# Generate a human-readable review report
benchci releases report bnd_a3f9c2e1b047 \
  --template generic-qa \
  --format pdf \
  --out stm32-v1.2.3-review.pdf

What to do when coverage is incomplete

If the matrix shows not_run for requirements you expected to cover:

  1. Confirm the suite ran those tests. Check results.json in the per-run evidence — are the test cases present?

  2. Confirm requirement_ids is set on those tests. Open the inputs/suite.yaml from a run’s evidence ZIP to see what was actually submitted.

  3. Check if the run failed early. If a run failed before a test executed, that test will not appear in the coverage data.

  4. Add a run that covers the missing requirement. Create a new bundle that includes that run.

Coverage data is only as complete as the traceability metadata in your suite.yaml files.