Quickstart

This guide shows a first working BenchCI setup using one node, UART output, and firmware flashing.

By the end, you will:

  • flash firmware

  • run an automated test suite

  • validate output over UART

  • collect structured results

Install BenchCI

pip install benchci

Create or access a workspace

BenchCI uses accounts and workspaces for Cloud Mode, dashboard access, and team visibility.

Open:

https://app.benchci.dev

Then log in from the CLI:

benchci login

How BenchCI works

BenchCI uses two configuration files:

  • bench.yaml describes the hardware bench

  • suite.yaml defines the test logic

You then run:

benchci run --bench bench.yaml --suite suite.yaml --artifact build/fw.elf

Execution modes

BenchCI supports multiple execution paths using the same bench and suite definitions.

Direct local execution

The CLI runs run_local(...) on the machine that is connected to the hardware.

benchci run --bench bench.yaml --suite suite.yaml --artifact build/fw.elf

Remote Agent execution

The CLI submits the run to a BenchCI Agent on another machine.

benchci run   --agent http://agent-host:8080   --bench bench.yaml   --suite suite.yaml   --artifact build/fw.elf

Registered-bench Agent execution

The Agent already knows the bench, so the CLI only sends the suite and artifact.

benchci run   --agent http://agent-host:8080   --bench-id my-bench   --suite suite.yaml   --artifact build/fw.elf

Cloud Mode execution

The CLI submits the run to the BenchCI backend, which schedules it on a cloud-connected Agent.

benchci run   --cloud   --bench-id my-cloud-bench   --suite suite.yaml   --artifact build/fw.elf

For a full mode-by-mode explanation, see execution_flow.md.

Prerequisites

You need:

  • Python 3.11+

  • BenchCI installed

  • a board connected to your machine

  • a supported flashing tool such as OpenOCD

  • firmware that prints expected UART output

Run diagnostics first:

benchci doctor

Step 1 — Create bench.yaml

Example:

version: "1"

bench:
  name: nucleo_uart_demo
  description: Simple single-node UART bench

defaults:
  node: dut
  timeouts:
    within_ms: 1000

nodes:
  dut:
    kind: mcu
    role: target

    flash:
      backend: openocd
      interface_cfg: interface/stlink.cfg
      target_cfg: target/stm32f4x.cfg

    reset:
      method: openocd

    transports:
      console:
        backend: uart
        port: /dev/ttyUSB0
        baud: 115200
        timeout_ms: 100

This bench defines:

  • one node named dut

  • OpenOCD flashing

  • OpenOCD reset

  • one UART transport named console

Step 2 — Create suite.yaml

Example:

version: "1"

suite:
  name: firmware_smoke

tests:
  - name: boot_ok
    steps:
      - expect_uart:
          node: dut
          transport: console
          contains: "[BOOT] OK"
          within_ms: 3000

  - name: ping
    steps:
      - send_uart:
          node: dut
          transport: console
          data: "PING\n"

      - expect_uart:
          node: dut
          transport: console
          contains: "PONG"
          within_ms: 1000

Step 3 — Validate the files

benchci validate --bench bench.yaml --suite suite.yaml

Step 4 — Run the suite locally

benchci run \
  --bench bench.yaml \
  --suite suite.yaml \
  --artifact build/fw.elf \
  --verbose

BenchCI will:

  1. load and validate the bench

  2. load and validate the suite

  3. flash firmware unless --skip-flash is used

  4. start required transports and GPIO

  5. execute the test steps

  6. write artifacts

Adding --verbose enables richer diagnostics, including:

  • detailed step logs

  • better failure context

  • additional transport and flash insights

  • more helpful debugging output when expectations fail

Step 5 — Inspect results

BenchCI writes artifacts into benchci-results/.

Typical structure:

benchci-results/
└── 20260328-142200/
    ├── results.json
    └── nodes/
        └── dut/
            ├── flash.log
            └── transport-console.log

results.json contains the high-level outcome. Per-node logs contain transport, flash, and GPIO details.

Remote quickstart

If your hardware is attached to another machine running BenchCI Agent:

benchci run \
  --agent http://192.168.1.50:8080 \
  --bench bench.yaml \
  --suite suite.yaml \
  --artifact build/fw.elf \
  --token "$BENCHCI_AGENT_TOKEN" \
  --verbose

If the Agent already has a registered bench:

benchci run \
  --agent http://192.168.1.50:8080 \
  --bench-id my-bench \
  --suite suite.yaml \
  --artifact build/fw.elf \
  --token "$BENCHCI_AGENT_TOKEN" \
  --verbose

In remote mode, the CLI downloads the final artifacts ZIP into benchci-results/.

Cloud quickstart

List benches visible to your active workspace:

benchci benches list

Run on a cloud bench:

benchci run \
  --cloud \
  --bench-id my-cloud-bench \
  --suite suite.yaml \
  --artifact build/fw.elf \
  --verbose

Inspect the run:

benchci runs list
benchci runs show <RUN_ID>
benchci runs events <RUN_ID>

You can also inspect the same run from:

https://app.benchci.dev

Troubleshooting

If the run fails:

  • inspect results.json

  • inspect the relevant transport log

  • inspect flash.log

  • confirm the serial/CAN/Modbus settings in bench.yaml

  • run benchci doctor --bench bench.yaml

If you are diagnosing a failure, prefer rerunning with --verbose.