BenchCI Documentation

Hardware CI for embedded systems

BenchCI lets embedded teams flash firmware, run tests on real devices, return results to CI, and preserve review-ready verification evidence.

benchci run --cloud --bench-id my-bench --suite suite.yaml --artifact build/fw.elf

No simulation. No manual serial-terminal testing. Real hardware in the loop.

The problem BenchCI solves

Most embedded validation still depends on manual bench work:

  • flash firmware by hand

  • open a terminal

  • send commands manually

  • watch logs

  • copy results into a ticket or release note

That is fine for early bring-up.

It is not enough for repeatable CI, shared hardware, pull-request validation, release gates, or remote teams.

What BenchCI gives you

BenchCI turns hardware validation into a repeatable CI workflow:

CI builds firmware
        ↓
BenchCI schedules a hardware run
        ↓
Agent flashes and tests a real device
        ↓
Results, logs, artifacts, and evidence return to CI/dashboard
        ↓
Optional release review packages can collect runs, coverage, identity, and sign-off history

You can start locally, then move the same test model to a remote Agent or BenchCI Cloud.

Fastest path

If you already have a board connected to your machine:

pip install benchci
benchci login
benchci init --preset flash-uart
benchci doctor --bench bench.yaml
benchci run --bench bench.yaml --suite suite.yaml --artifact build/fw.elf

BenchCI will load your hardware configuration, flash the firmware, run the suite, and write results under benchci-results/.

If you prefer to start in the browser, the dashboard Config Builder can export starter bench.yaml and suite.yaml files for the same preset-based flows.

Core mental model

bench.yaml  -> describes the hardware
suite.yaml  -> describes the test logic
benchci run -> executes the suite on real hardware

Use benchci init --list-presets to see starter configs for UART smoke tests, flash + UART flows, GPIO reset/ready checks, power cycling, measurements, CAN, Modbus, and bounded protocol fuzzing. The generated files are intentionally editable YAML, not a hidden cloud configuration.

A bench is your physical setup: DUT, debugger, UART/CAN/Modbus adapters, GPIO, relays, power controllers, measurement instruments, and related resources.

A suite is what should happen: flash, reset, send commands, wait for logs, read registers, check GPIO, validate CAN frames, run bounded fuzzing, cycle power, take measurements, and assert metrics.

An Agent is the hardware-connected process that lets CI or remote users run tests without sitting next to the device.

BenchCI Cloud Mode adds workspace-aware scheduling, shared/private benches, dashboard visibility, and CI-friendly execution.

What BenchCI supports today

Communication:

  • UART

  • Modbus RTU

  • Modbus TCP

  • CAN

  • I2C (bus scan, register read/write, expect)

  • SPI (full-duplex transfer, expect)

  • bounded UART/CAN/Modbus protocol fuzzing

  • experimental allow-listed power, GPIO, and malformed-UART-byte fault injection

Flashing:

  • OpenOCD

  • STM32CubeProgrammer

  • SEGGER J-Link

  • esptool

Control:

  • Linux GPIO

  • remote GPIO through Agent

  • GPIO-backed power control

  • HTTP relay control

  • serial relay command maps

Power control is exposed through bench-level resources, so suites use generic steps such as power_cycle while bench.yaml defines the concrete GPIO, HTTP, or serial relay backend.

Measurements:

  • HTTP/lab-controller measurement resources

  • SCPI measurement resources over TCP/IP, serial/RS232, and USB/VISA-style instruments

  • SCPI power-supply current/voltage readback with conservative presets

  • I2C power-monitor measurements

  • script and serial measurement resources

  • recorded metrics and threshold assertions

Measurement follows the same bench-level resource pattern, so suites use generic measure steps while bench.yaml defines the concrete measurement backend.

Evidence and traceability:

  • evidence.json machine-readable run evidence

  • evidence.html human-readable evidence reports

  • input snapshots of bench.yaml and suite.yaml

  • firmware, suite, and bench config hashes

  • optional requirement, test case, risk, release, and tag metadata

  • structured failure explanations and suggested checks

  • artifact manifest.json with SHA256 hashes

  • captured measurements and metrics

  • fuzz campaign summaries, seeds, first failing cases, and JSONL case logs when fuzzing is used

  • configured or UART-verified DUT identity

  • LCOV summaries attached from unit tests or simulation

  • firmware handling mode, SHA256 verification, deletion state, and audit events

  • imported JUnit/CTRF results and authenticated external artifacts

  • experimental controlled fault-injection recovery evidence

    • optional release bundles with coverage matrices, review sign-off, and HTML/PDF reports

  • recent reliability history, flaky-test warnings, and advisory failure assessments

Execution:

  • local runs

  • direct Agent runs

  • backend-controlled Cloud Mode runs

Documentation

Getting Started

Getting started

Core Reference

Core reference

Advanced Assurance Workflows

Advanced assurance workflows