Quickstart

Run your first real hardware test with BenchCI.

This guide uses the simplest useful path:

firmware artifact
        ↓
benchci run
        ↓
flash board
        ↓
read UART output
        ↓
write structured results

By the end, you will have a local BenchCI run that flashes firmware, validates UART output, and stores logs under benchci-results/.

What you need

  • Python 3.11+

  • BenchCI installed

  • a board connected to your machine, for example an STM32 NUCLEO

  • a supported flashing tool, for example OpenOCD

  • firmware that prints expected UART output

  • a serial connection to the board

Install BenchCI first:

pip install benchci

Log in:

benchci login

Check your active account and workspace:

benchci whoami

Step 1 — Create starter YAML

Start with a preset instead of writing both YAML files from scratch:

benchci init --list-presets
benchci init --preset flash-uart

benchci init writes bench.yaml and suite.yaml, validates them, and prints the next commands to run. It refuses to overwrite existing files unless you pass --force.

Use --yes in scripts or CI bootstrap flows to skip interactive prompts and accept the preset defaults:

benchci init --preset flash-uart --yes --bench-output bench.yaml --suite-output suite.yaml

You can also create the same starter files from the dashboard Config Builder, then copy or download them into your repository.

Step 2 — Inspect your machine

Run doctor so you know which ports, USB devices, tools, and GPIO chips BenchCI can see:

benchci doctor

After creating bench.yaml, cross-check it against the current machine:

benchci doctor --bench bench.yaml

This is especially useful for finding the correct UART port, checking whether OpenOCD/J-Link/esptool is installed, and confirming GPIO device paths such as /dev/gpiochip0.

Step 3 — Review bench.yaml

bench.yaml describes the physical hardware.

Example:

version: "1"

bench:
  name: nucleo_uart_demo
  description: Simple single-node UART bench

defaults:
  node: dut
  timeouts:
    within_ms: 1000

nodes:
  dut:
    kind: mcu
    role: target

    flash:
      backend: openocd
      interface_cfg: interface/stlink.cfg
      target_cfg: target/stm32f4x.cfg

    reset:
      method: openocd

    transports:
      console:
        backend: uart
        port: /dev/ttyUSB0
        baud: 115200
        timeout_ms: 100

artifacts:
  root_dir: benchci-results
  per_node_dirs: true

This defines:

  • one node named dut

  • OpenOCD flashing

  • OpenOCD reset

  • one UART transport named console

  • artifact output under benchci-results/

Adjust these fields for your board:

  • target_cfg

  • UART port

  • UART baud

  • optional probe serials or extra flash arguments

Step 4 — Review suite.yaml

suite.yaml defines the test logic.

Example:

version: "1"

suite:
  name: firmware_smoke
  description: Flash firmware and validate boot logs

tests:
  - name: boot_ok
    steps:
      - expect_uart:
          node: dut
          transport: console
          contains: "[BOOT] OK"
          within_ms: 3000

  - name: ping
    steps:
      - send_uart:
          node: dut
          transport: console
          data: "PING\n"

      - expect_uart:
          node: dut
          transport: console
          contains: "PONG"
          within_ms: 1000

This suite checks that:

  • the firmware prints [BOOT] OK

  • the device responds to PING with PONG

Optional: add power control

After your first UART-only run works, you can make the suite closer to real CI by adding a bench-level power resource.

In bench.yaml:

resources:
  dut_power:
    kind: power_controller
    driver:
      type: gpio_power
      chip: /dev/gpiochip0
      outlets:
        main: 17
      active_high: true
      initial_state: false

In suite.yaml:

- power_cycle:
    resource: dut_power
    outlet: main
    off_ms: 1000
    on_settle_ms: 2000

This keeps the suite focused on intent while the bench file describes whether power is GPIO-backed, HTTP-backed, or driven by a serial relay command map.

Optional: add a measurement

Measurement resources let you record physical behavior such as current or voltage.

For example, a lab controller can expose a measured value over HTTP:

resources:
  sleep_current:
    kind: measurement
    driver:
      type: http_measurement
      quantity: current
      url: "http://192.168.1.60/measurements/sleep_current"
      value_field: value
      unit_field: unit

For a real SCPI-capable power supply or meter, use a SCPI measurement resource. TCP/IP example:

resources:
  supply_current:
    kind: measurement
    driver:
      type: scpi_power_supply_measurement
      address: tcp://127.0.0.1:5025
      quantity: current
      timeout_ms: 1000

Serial/RS232 example:

resources:
  supply_current:
    kind: measurement
    driver:
      type: scpi_power_supply_measurement
      preset: owon_sp
      address: serial:///dev/ttyUSB0
      quantity: current
      timeout_ms: 1000
      baudrate: 9600

USB/VISA-style example:

resources:
  supply_current:
    kind: measurement
    driver:
      type: scpi_measurement
      address: USB0::0x1234::0x5678::INSTR
      query: "MEAS:CURR?"
      quantity: current
      unit: A
      timeout_ms: 1000

Use benchci measure to debug one measurement resource before running a full suite:

benchci measure --bench bench.yaml supply_current

In suite.yaml:

- measure:
    resource: supply_current
    record_as: sleep_current_a
    unit: A
    expect_less_than: 0.150

The measured value can appear in results, evidence reports, CLI summaries, and dashboard run detail where supported.

Optional: add bounded protocol fuzzing

After your smoke/regression checks are stable, add a short fuzz step to exercise parser robustness on real hardware.

UART example:

- fuzz_uart:
    node: dut
    transport: console
    seed: 12648430
    iterations: 50
    max_duration_ms: 10000
    mode: ascii
    min_length: 0
    max_length: 64
    suffix: "\n"
    fail_contains: "ASSERT"

BenchCI records the seed, step index, first failing case, and a JSONL case log in logs/fuzz/, so failures can be replayed. See Protocol Fuzzing before adding fuzzing to CI gates.

Optional: add traceability

For a first run, a simple suite is enough. When you want evidence reports to show requirement or risk coverage, add optional traceability fields:

suite:
  name: firmware_smoke
  description: Flash firmware and validate boot logs
  requirement_ids:
    - REQ-BOOT-001
  risk_ids:
    - RISK-BOOT-001
  tags:
    - smoke

tests:
  - name: boot_ok
    test_case_id: TC-BOOT-001
    requirement_ids:
      - REQ-BOOT-001
    risk_ids:
      - RISK-BOOT-001
    tags:
      - boot
      - uart
    steps:
      - expect_uart:
          node: dut
          transport: console
          contains: "[BOOT] OK"
          within_ms: 3000

Traceability fields are optional. Use them when you want the run to support release or QA evidence workflows.

Step 5 — Validate configuration

Validate without touching hardware:

benchci validate --bench bench.yaml --suite suite.yaml

If validation fails, fix the config before running on the device.

Step 6 — Run locally

benchci run \
  --bench bench.yaml \
  --suite suite.yaml \
  --artifact build/fw.elf \
  --verbose

BenchCI will:

  1. load and validate bench.yaml

  2. load and validate suite.yaml

  3. flash the artifact unless --skip-flash is used

  4. start only the required transports and GPIO resources

  5. execute the suite steps

  6. write logs and structured results

Step 7 — Inspect results

BenchCI writes artifacts into benchci-results/ by default. Use --results-dir to choose a different result root for a run, or set artifacts.root_dir in bench.yaml for the local-run default.

Typical structure:

benchci-results/
└── 20260328-142200/
    ├── results.json
    ├── evidence.json
    ├── evidence.html
    ├── manifest.json
    ├── metadata.json
    ├── inputs/
    │   ├── bench.yaml
    │   └── suite.yaml
    └── logs/
        ├── nodes/
        │   └── dut/
        │       ├── flash.log
        │       └── transport-console.log
        └── fuzz/
            └── uart_protocol_fuzz-step1-uart.jsonl

results.json contains the high-level outcome.

Per-node logs contain flash, transport, GPIO, power, or protocol logs depending on the bench and suite. Fuzz case logs are stored under logs/fuzz/ when a suite uses fuzz_uart, fuzz_can, or fuzz_modbus.

Common first-run problems

UART port is wrong

Check available ports:

ls /dev/ttyUSB* /dev/ttyACM*

Update bench.yaml.

Permission denied on serial port

On Linux, add your user to the serial group used by your distribution, often dialout:

sudo usermod -aG dialout $USER

Then log out and back in.

Flash tool not found

Install the required flash tool, for example:

sudo apt-get install -y openocd

Boot message not found

Increase timeout or verify the firmware really prints the expected text:

within_ms: 5000

Next step: connect CI

Once local execution works, move to the end-to-end CI flow:

End-to-End Example

Evidence outputs

BenchCI writes both machine-readable and human-readable evidence:

  • results.json — execution summary, test results, structured failures, and per-test traceability

  • evidence.json — firmware hash, Git/CI metadata, bench/suite hashes, traceability, fuzz summaries, and artifact list

  • evidence.html — human-readable evidence report

  • metadata.json — supporting run metadata

  • inputs/bench.yaml and inputs/suite.yaml — snapshots of the exact inputs used

Open evidence.html after the run when you want a report that is easier to share with a teammate, QA reviewer, or release owner.