Running Tests with run_test.py

run_test.py is the hardware test runner shipped in every FBOSS package (/opt/fboss/bin/run_test.py). It discovers tests in a gtest binary, skips known-bad and unsupported ones for your ASIC and SDK, runs each test in its own process with cold boot and warm boot passes, manages the services the tests need (hardware agents, qsfp_service, FSDB), and writes a CSV of results. This page is the reference for its subcommands, options, and outputs. Which suites to run, and in what order, is covered in Test Suites.

Before You Start

  • Install the package in /opt/fboss (see Packaging and Running on a Switch).
  • run_test.py changes into the package root itself (from $FBOSS, or two directories above its own module) and sets up the FBOSS environment variables, so you can start it from anywhere; result files are written to the package root.
  • If the environment cannot be set up it prints FBOSS environment not set. Run `source /opt/fboss/bin/setup_fboss_env' and exits with status 0.
  • It points /tmp/platform_descriptors at $FBOSS_DATA/platform_descriptors (replacing whatever was there), or warns if FBOSS_DATA is not set or the directory is missing.
cd /opt/fboss
./bin/run_test.py <test-type> [options]
./bin/run_test.py <test-type> --help

Running run_test.py without a test type prints help and exits with status 1.

Test Types

Subcommand Runs Config option
sai_agent Agent hardware tests (multi_switch_agent_hw_test, or sai_agent_hw_test-* in mono mode) --config
sai SAI hardware tests (sai_test-*) --config
bcm Native Broadcom SDK tests --config
sai_agent_scale Agent scale tests --config
sai_invariant_agent Agent invariant tests --config
qsfp qsfp_hw_test --qsfp-config
link Link tests (needs cabled ports) --config, --qsfp-config
platform Platform services hardware tests none
led led_service_hw_test
benchmark SAI agent or QSFP benchmarks --config or --qsfp-config
fboss2_integration fboss2 CLI integration tests
services Start or stop services, no tests see below

Common Options

These are accepted by every test type (pass them after the subcommand):

Option Default Meaning
--config FILE Agent config to run with, usually ./share/hw_test_configs/<platform>.agent.materialized_JSON
--filter EXPR gtest filter, passed through as --gtest_filter. Supports * wildcards, : to separate patterns, and - for negative patterns: --filter=*Vlan*:*Port*:-*Mac*:*Intf*
--filter_file FILE Run the tests listed in a file, such as ./share/hw_sanity_tests/t0_agent_hw_tests.conf
--tier {0,1,2} Run the tests classified in a certification tier
--profile NAME Only include patterns tagged with this profile; requires --filter_file or --tier
--list_tests false List the tests that would run, without running them
--coldboot_only false Skip the warm boot pass
--num-warmboot-iterations N 1 Warm boot N times after the initial cold boot (soak); ignored with --coldboot_only
--skip-known-bad-tests KEY Skip tests listed as known bad or unsupported for this key
--known-bad-tests-file FILE, --unsupported-tests-file FILE per runner Override the lists used with --skip-known-bad-tests
--test-run-timeout SECONDS 1200 Timeout for each test run
--results-json FILE Also write structured per-test results as JSON
--fboss_logging LEVEL DBG4 FBOSS log level: INFO, ERR, DBG0 to DBG9
--sai_logging LEVEL WARN SAI log level: DEBUG, INFO, NOTICE, WARN, ERROR, CRITICAL
--sai_replayer_logging DIR Record a SAI replayer log into DIR
--sai_replayer_sdk_log_level LEVEL SDK log level for the replayed run (requires --sai_replayer_logging)
--extra-gflags FLAGS Extra flags appended to every test binary invocation; use = to attach a value starting with --
--mgmt-if IF eth0 Management interface
--fruid-path FILE /var/facebook/fboss/fruid.json fruid.json to use
--switch_id_for_testing N Switch ID on multi-NPU platforms
--simulator ASIC Run against a Broadcom simulator: th3, th4, th4_b0, th5, or j3
--run-on-reference-board false Adjust SAI settings for a reference board
--setup-for-coldboot SCRIPT, --setup-for-warmboot SCRIPT Run a script before the cold boot or warm boot run
--log-bundle off Capture the run's output and the systemd service logs into a per-run directory under /opt/fboss/logs/run_test/ and zip it

--filter and --filter_file are mutually exclusive: using both fails with Error: Only one of the --filter or --filter_file can be specified at any time and exit status 2. --profile without --filter_file or --tier also fails with status 2.

Agent Run Mode Options

For sai_agent, sai_agent_scale, sai_invariant_agent, link, and benchmark:

Option Default Meaning
--agent-run-mode {mono,multi_switch} multi_switch (mono for benchmark) multi_switch runs the software agent and hardware agent as separate processes and manages the hw_agent service; mono is deprecated
--num-npus {1,2} 1 Number of NPUs (hardware agents) in multi-switch mode
--platform_mapping_override_path FILE Use this platform mapping JSON instead of the built-in one

sai_agent Options

Option Default Meaning
--enable-production-features ASIC Only run tests whose production features the ASIC supports; features whose minSdkVersion is above the built SDK are skipped
--production-features FILE ./share/production_features/asic_production_features.materialized_JSON Production features file
--test-config-options LIST Comma-separated options (for example rbb) that satisfy features' requiredTestConfigOptions; the agent run mode is always included
--list-tests-for-features LIST Print tests whose feature tags are all in the list, for example DLB,ACL_COUNTER,SINGLE_ACL_TABLE

Available ASIC names are the keys of asicToFeatureNames in the production features file.

Option Meaning
--qsfp-config FILE QSFP config, as an absolute path
--bsp_platform_mapping_override_path FILE Override the BSP platform mapping
--platform_mapping_override_path FILE Override the ASIC platform mapping
--without-transceivers Use the tier rules for platforms without pluggable transceivers (for qsfp: and with external PHYs)
--disable-fsdb (link) Do not run FSDB for link tests
--fsdb-config FILE (link) FSDB config to use

platform Options

--type runs one platform test binary: platform_hw_test, data_corral_service_hw_test, fan_service_hw_test, fw_util_hw_test, sensor_service_hw_test, weutil_hw_test, or platform_manager_hw_test. Without it all of them run.

benchmark Options

Option Default Meaning
--sai / --qsfp --sai Benchmark family (mutually exclusive)
--agent-run-mode mono mono runs sai_all_benchmarks-sai_impl; multi_switch runs sai_multi_switch_all_benchmarks-sai_impl with --multi_switch --hw_agent_for_testing
--qsfp-config FILE Required with --qsfp
--force-5pim-fuji false QSFP benchmarks on fuji_5x16Q
--port-manager-mode false QSFP benchmarks in port manager mode
--filter Regex matched against discovered benchmark names
--filter_file One BENCHMARK() name per line (list them with the binary's --bm_list)

fboss2_integration Options

--num-npus {1,2} and --skip-coldboot (skip per-test cold boots after the initial one).

Known-Bad Keys

--skip-known-bad-tests takes a key that selects entries in the known-bad and unsupported lists:

Test type Key format Example
sai, sai_agent, scale, invariant vendor/coldboot-sdk/warmboot-sdk/asic (agent types may append /mode) brcm/8.2.0.0_odp/8.2.0.0_odp/tomahawk4, brcm/13.3.0.0_odp/13.3.0.0_odp/tomahawk5/multi_switch
SAI benchmark vendor/sdk/asic brcm/13.3.0.0_odp/tomahawk5
QSFP benchmark <platform>/<phy_sdk> montblanc/credo-0.7.2

Default list files (relative to /opt/fboss):

Type Known bad Unsupported
SAI share/hw_known_bad_tests/sai_known_bad_tests.materialized_JSON share/sai_hw_unsupported_tests/sai_hw_unsupported_tests.materialized_JSON
SAI agent share/hw_known_bad_tests/sai_agent_known_bad_tests.materialized_JSON share/sai_hw_unsupported_tests/sai_agent_hw_unsupported_tests.materialized_JSON
SAI agent scale share/hw_known_bad_tests/sai_agent_scale_known_bad_tests.materialized_JSON
QSFP share/qsfp_known_bad_tests/fboss_qsfp_known_bad_tests.materialized_JSON share/qsfp_unsupported_tests/fboss_qsfp_unsupported_tests.materialized_JSON
Link share/link_known_bad_tests/agent_ensemble_link_known_bad_tests.materialized_JSON share/link_known_bad_tests/agent_ensemble_link_unsupported_tests.materialized_JSON

Look inside these files to find the keys that exist for your ASIC.

Examples

# All agent hardware tests, multi_switch, skipping known bad
./bin/run_test.py sai_agent --config ./share/hw_test_configs/$CONFIG \
  --agent-run-mode multi_switch --skip-known-bad-tests "$KEY"

# One test, cold boot only
./bin/run_test.py sai --config ./share/hw_test_configs/$CONFIG \
  --coldboot_only --filter=HwVlanTest.VlanApplyConfig

# Warm boot soak: cold boot once, then 64 warm boots
./bin/run_test.py sai_agent --config ./share/hw_test_configs/$CONFIG \
  --agent-run-mode multi_switch --filter AgentEmptyTest.CheckInit --num-warmboot-iterations 64

# List tests without running them
./bin/run_test.py sai --config ./share/hw_test_configs/$CONFIG --list_tests

# QSFP tests with non-default platform mappings
./bin/run_test.py qsfp --qsfp-config ./share/qsfp_test_configs/$QSFP_CONFIG \
  --filter=EmptyHwTest.CheckInit \
  --bsp_platform_mapping_override_path /path/to/bsp_mapping.json \
  --platform_mapping_override_path /path/to/platform_mapping.json

# Link test
./bin/run_test.py link --config share/link_test_configs/fuji.materialized_JSON \
  --qsfp-config /opt/fboss/share/qsfp_test_configs/fuji.materialized_JSON \
  --filter=LinkTest.asicLinkFlap

# SAI replayer log at DEBUG
./bin/run_test.py sai --config ./share/hw_test_configs/$CONFIG \
  --sai_replayer_logging /tmp/logs --sai_replayer_sdk_log_level DEBUG

services Subcommand

services starts or stops FBOSS services without running tests:

./bin/run_test.py services start --agent-config $AGENT_CONFIG --qsfp-config $QSFP_CONFIG
./bin/run_test.py services start agent --agent-config $AGENT_CONFIG
./bin/run_test.py services start agent qsfp fsdb --agent-config $AGENT_CONFIG --qsfp-config $QSFP_CONFIG
./bin/run_test.py services stop --num-npus 1
Argument Meaning
start [agent] [qsfp] [fsdb] Services to start; default all
stop [agent] [qsfp] [fsdb] Services to stop; default all
--agent-config FILE Required when agent is selected; must exist (Agent config does not exist: <path> otherwise)
--qsfp-config FILE Required when qsfp is selected
--fsdb-config FILE FSDB config
--platform_mapping_override_path, --bsp_platform_mapping_override_path Mapping overrides
--num-npus {1,2} Number of hardware agents (switch indexes); default 1

Generated configs go under /opt/fboss/logs/run_test/generated_configs. Before starting a hardware agent, the runner stops any fboss_hw_agent@N, fboss_hw_agent_for_testing_N, or fboss_hw_agent_oss@N units for the same index.

Results and Exit Codes

  • Test results are written to hwtest_results_<YYYY_Mon_DD-HH_MM_SS_AM|PM>.csv in the package root, and the path is printed as Test output stored at: <file>.
  • Benchmark results go to benchmark_results_<timestamp>.csv with columns benchmark_binary_name, benchmark_test_name, benchmark_time_ps, test_status, cpu_time_usec, max_rss, cpu_rx_pps, cpu_tx_pps, threshold_status, threshold_details. Benchmarks without a configured threshold print a WILL ALWAYS PASS warning, and without --skip-known-bad-tests thresholds are not checked at all (NO_THRESHOLD).
Exit code Meaning
0 All selected tests passed or were skipped (also: environment not set, see above)
1 At least one test failed or timed out; or no test type given
2 Usage error (conflicting options)
75 (EX_TEMPFAIL) Setup failed before tests ran

Running a Test Binary Directly

You can bypass the runner for quick checks:

./bin/sai_agent_hw_test-sai_impl --gtest_filter=AgentEmptyTest.CheckInit \
  --config ./share/hw_test_configs/$CONFIG --fruid_filepath /var/facebook/fboss/fruid.json

You then have to manage warm boot, services, and known-bad filtering yourself.

See Also