GitLab CI Integration¶

Run real hardware tests from a GitLab pipeline.

This is the recommended production flow:

GitLab CI builds firmware
        ↓
BenchCI CLI schedules a cloud run
        ↓
Cloud-connected Agent flashes and tests real hardware
        ↓
Results, measurements, evidence, and traceability return to GitLab artifacts and the BenchCI dashboard

Use this when you want pipeline validation on real devices instead of compile-only firmware checks.

Run BenchCI hardware tests from a GitLab pipeline.

This guide shows the recommended GitLab CI flow for BenchCI:

build firmware in GitLab CI
upload or reference the firmware artifact
run BenchCI against real hardware
store BenchCI results as GitLab artifacts

Before using Cloud Mode¶

Cloud Mode requires at least one BenchCI Agent registered to your workspace and connected to a real hardware bench.

Before running CI, make sure:

the Agent machine is connected to the DUT, debugger, UART/CAN/Modbus adapters, GPIO, relays, or other required hardware
the Agent is registered to your BenchCI workspace
the bench appears in the dashboard
the bench is online/idle
you know the bench ID to use in CI

You can check visible benches with:

benchci benches list

Use that bench ID as:

BENCHCI_BENCH_ID=my-cloud-bench

Recommended setup¶

For most teams, use BenchCI Cloud Mode from GitLab CI.

GitLab CI job
    ↓
BenchCI CLI
    ↓
BenchCI API
    ↓
BenchCI Agent
    ↓
real hardware bench
    ↓
results + logs + measurements

This keeps your GitLab Runner separate from the hardware lab network.

Requirements¶

Before starting, make sure you have:

a GitLab project
a BenchCI account and workspace
a registered BenchCI bench
a working suite.yaml
firmware produced by your pipeline
BenchCI secrets stored as GitLab CI/CD variables

Step 1 — Add GitLab variables¶

In your GitLab project, open:

Settings → CI/CD → Variables

Add these variables:

BENCHCI_EMAIL=engineer@company.com
BENCHCI_PASSWORD=your-password
BENCHCI_BENCH_ID=my-cloud-bench

Optional stable CI machine identity:

BENCHCI_MACHINE_ID=my-repo-ci

Most shared GitLab runners should leave BENCHCI_MACHINE_ID unset. Set it only for a persistent runner or a non-parallel pipeline lane that should always count as the same CI runner. If another active CI session already uses the same value, login is blocked until that session logs out or expires.

Use masked/protected variables where appropriate.

Step 2 — Add .gitlab-ci.yml¶

Example pipeline:

stages:
  - build
  - hardware-test

build-firmware:
  stage: build
  image: ubuntu:24.04
  script:
    - apt-get update
    - apt-get install -y make gcc-arm-none-eabi binutils-arm-none-eabi
    - make
    - mkdir -p build
    - cp path/to/firmware.elf build/firmware.elf
  artifacts:
    paths:
      - build/firmware.elf

hardware-test:
  stage: hardware-test
  image: python:3.11
  needs:
    - job: build-firmware
      artifacts: true
  script:
    - pip install --upgrade benchci
    - benchci login --email "$BENCHCI_EMAIL" --password "$BENCHCI_PASSWORD"
    - benchci run --cloud --bench-id "$BENCHCI_BENCH_ID" --suite suite.yaml --artifact build/firmware.elf --verbose
  # The CLI automatically logs out the CI session after benchci run --cloud completes,
  # freeing the runner slot immediately regardless of run outcome.
  artifacts:
    when: always
    paths:
      - benchci-results/

Update these paths for your project:

path/to/firmware.elf
suite.yaml

No-upload firmware option¶

If your team does not want firmware bytes uploaded to BenchCI Cloud, publish the firmware to a private artifact store, create a short-lived signed URL, compute SHA256 in CI, and pass both values:

hardware-test:
  stage: hardware-test
  image: python:3.11
  script:
    - pip install --upgrade benchci
    - benchci login --email "$BENCHCI_EMAIL" --password "$BENCHCI_PASSWORD"
    - FIRMWARE_SHA256="$(sha256sum build/firmware.elf | awk '{print $1}')"
    - benchci run --cloud --bench-id "$BENCHCI_BENCH_ID" --suite suite.yaml --firmware-url "$SIGNED_FIRMWARE_URL" --firmware-sha256 "$FIRMWARE_SHA256" --verbose

Step 3 — Push and inspect the pipeline¶

After pushing, the expected flow is:

GitLab builds your firmware
GitLab passes the firmware artifact to the hardware-test job
BenchCI logs in
BenchCI schedules the run on the selected bench
the bench flashes firmware and executes the test suite
BenchCI downloads results into benchci-results/ by default
GitLab stores those results as job artifacts

Results¶

BenchCI writes results into:

benchci-results/

Typical contents include:

results.json
evidence.json
evidence.html
manifest.json
evidence.json
evidence.html
metadata.json
inputs/bench.yaml
inputs/suite.yaml
flash.log
transport-*.log
gpio.log
power.log

The exact logs depend on your bench and test suite.

Evidence in CI¶

When the workflow runs in CI, BenchCI can capture CI metadata such as provider, run ID, job ID, and job URL. Together with Git metadata and firmware hashes, this lets the evidence package connect a hardware result back to the exact CI build.

If your suite.yaml includes requirement_ids, test_case_id, risk_ids, release_id, or tags, those values are included in results.json, evidence.json, evidence.html, and the dashboard run detail.

Upload benchci-results/, or the directory passed to --results-dir, as a CI artifact so the evidence package is retained with the pipeline.

Direct Agent mode¶

Use Direct Agent mode when your GitLab Runner can reach the hardware machine directly over the network.

GitLab CI job
    ↓
BenchCI CLI
    ↓
BenchCI Agent on lab machine
    ↓
real hardware bench

Agent variables¶

Add:

BENCHCI_AGENT_URL=http://192.168.1.50:8080
BENCHCI_AGENT_TOKEN=secure-token

Pipeline example¶

stages:
  - hardware-test

hardware-test:
  stage: hardware-test
  image: python:3.11
  script:
    - pip install --upgrade benchci
    - benchci run --agent "$BENCHCI_AGENT_URL" --bench bench.yaml --suite suite.yaml --artifact build/firmware.elf --token "$BENCHCI_AGENT_TOKEN" --verbose
  artifacts:
    when: always
    paths:
      - benchci-results/

Registered bench Agent mode¶

If the Agent already has the bench registered, use --bench-id instead of uploading bench.yaml from CI.

hardware-test:
  stage: hardware-test
  image: python:3.11
  script:
    - pip install --upgrade benchci
    - benchci run --agent "$BENCHCI_AGENT_URL" --bench-id nucleo-uart --suite suite.yaml --artifact build/firmware.elf --token "$BENCHCI_AGENT_TOKEN" --verbose
  artifacts:
    when: always
    paths:
      - benchci-results/

This is usually better for stable shared lab infrastructure.

Troubleshooting¶

If the job fails:

confirm the GitLab variables are available to the job
confirm the bench ID is visible to your workspace
confirm the firmware artifact path exists
inspect benchci-results/
rerun with --verbose

For Direct Agent mode:

confirm the runner can reach the Agent URL
confirm the Agent token matches
confirm the hardware machine can flash and test locally first

Recommended preflight before enabling CI¶

Before enabling GitLab CI hardware runs, verify the bench on the hardware-connected machine:

benchci validate --bench bench.yaml --suite suite.yaml
benchci benches self-test --bench bench.yaml --open-hardware --log-dir bench-health
benchci run --bench bench.yaml --suite suite.yaml --artifact build/fw.elf --dry-run-plan
benchci run --bench bench.yaml --suite suite.yaml --artifact build/fw.elf

Then start the cloud Agent and confirm the bench is visible and healthy/degraded:

benchci benches list

Cloud scheduling intentionally avoids benches with failing or unknown health.