Test Suites

FBOSS has hundreds of hardware tests, organized into suites by what they exercise and into tiers by priority. This is how Meta and switch vendors qualify a new platform, ASIC, SDK, or optic. This page explains each suite, the hardware it needs, the T0/T1/T2 tiers and their filter files, the scale-up profile, and a sensible order for onboarding. The mechanics of run_test.py itself are on Running Tests.

Variables Used Below

Variable	Meaning
`$CONFIG`	Agent test config for your platform, a file in `share/hw_test_configs/` (or `share/link_test_configs/` for link tests)
`$QSFP_CONFIG`	QSFP test config in `share/qsfp_test_configs/`
`$ASIC`	ASIC name for production-feature filtering, a key of `asicToFeatureNames` in `share/production_features/asic_production_features.materialized_JSON`
`$KEY`	Known-bad key, for example `brcm/8.2.0.0_odp/8.2.0.0_odp/tomahawk` (see Known-Bad Keys)

The Suites

Suite	`run_test.py` type	What it verifies	Hardware setup
Agent HW tests	`sai_agent`	FBOSS features end to end through the agent: L3 routing, ACLs, L2 learning, QoS, CoPP, ECMP, PFC, mirroring, ... The binary is the agent with gtest built in.	Standalone switch, no cables; data-plane tests inject packets into front-panel ports in loopback
SAI tests	`sai`	The `HwSwitch`/SAI layer directly (`sai_test-*`), for example `HwVlanTest.VlanApplyConfig`	Standalone switch
Agent scale tests	`sai_agent_scale`	Features at production scale: maximum ECMP groups and members, maximum ACL entries (`AgentEcmpScaleTests.cpp`, `AgentAclScaleTests.cpp`)	Standalone switch
Agent invariant tests	`sai_invariant_agent`	Core forwarding invariants after a production-like config: ACLs programmed, CoPP in effect, ECMP load balancing, DSCP-to-queue mapping, diag and Thrift handlers. Role variants exist (RSW, RTSW, FTSW, STSW), for example `ProdInvariantRtswTest.verifyInvariants`	Standalone switch, with a config matching the role
QSFP HW tests	`qsfp`	`qsfp_service` with real transceivers and PHYs	Switch with optics or cables plugged in
Link tests	`link`	End-to-end link bring-up across agent, `qsfp_service`, and FSDB: link up on all cabled ports, link flaps, LLDP and fabric reachability, L1 diagnostics, PRBS, warm boot	Ports cabled in a snake topology
Platform tests	`platform`	`platform_manager`, `sensor_service`, `fan_service`, `data_corral_service`, `fw_util`, `weutil`, and general platform checks	Switch
BSP tests	(own binary)	The vendor's BSP kernel modules, configured by `platform_stack/bsp_tests.json`	Switch
LED tests	`led`	`led_service`	Switch
Benchmarks	`benchmark`	Performance: route programming scale, init and exit time, RX/TX packet rates, transceiver refresh	Switch
CLI integration	`fboss2_integration`	`fboss2` commands against a running agent	Switch or the `fboss-sim` container

Test sources: agent HW tests in fboss/agent/test/agent_hw_tests/ and fboss/agent/hw/sai/hw_test/, invariant tests in fboss/agent/test/prod_invariant_tests/, link tests in fboss/agent/test/link_tests/.

Binaries per Run Mode

Suite	`multi_switch` (recommended)	`mono` (deprecated)
Agent HW	`multi_switch_agent_hw_test`	`sai_agent_hw_test-sai_impl`
Scale	`multi_switch_agent_scale_test`	`sai_agent_scale_test-sai_impl`
Invariant	`multi_switch_invariant_agent_test`	`sai_invariant_agent_test-sai_impl`
Benchmarks	`sai_multi_switch_all_benchmarks-sai_impl`	`sai_all_benchmarks-sai_impl`

run_test.py defaults to multi_switch for the agent suites and link tests, but to mono for benchmarks.

Tiers

Tier	Contents	When it must pass
T0	Simple but critical functionality that other tests depend on	Before New Platform Onboarding EVT exit
T1	More complex tests that verify overall functionality	Next
T2	Complex features and performance tuning	Not blocking early bring-up, but required for platform qualification before production

Tiers are a subset of all tests. After T0, T1, and T2, the goal is a 100% pass rate on everything (run without a filter file or tier) minus the known-bad and unsupported lists for your key.

Tier membership comes from two places in share/hw_sanity_tests/: filter files such as t0_agent_hw_tests.conf, and metadata files (sai_test_metadata.json, qsfp_test_metadata.json, link_test_metadata.json, bsp_test_metadata.json) whose rules give each test-name pattern a tier, required capabilities (TRANSCEIVER, EXTERNAL_PHY), and profiles. run_test.py --tier 0|1|2 uses the metadata.

T0 Commands

# Platform services (all platform tests are T0)
./bin/run_test.py platform

# Agent HW tests
./bin/run_test.py sai_agent --filter_file=./share/hw_sanity_tests/t0_agent_hw_tests.conf \
  --config ./share/hw_test_configs/$CONFIG --enable-production-features $ASIC --skip-known-bad-tests $KEY

# SAI tests
./bin/run_test.py sai --filter_file=./share/hw_sanity_tests/t0_sai_tests.conf \
  --config ./share/hw_test_configs/$CONFIG --skip-known-bad-tests $KEY

# QSFP HW tests (use t0_qsfp_hw_tests_without_transceivers.conf on platforms without transceivers)
./bin/run_test.py qsfp --filter_file=./share/hw_sanity_tests/t0_qsfp_hw_tests.conf \
  --qsfp-config ./share/qsfp_test_configs/$QSFP_CONFIG --skip-known-bad-tests $KEY

# Link tests (use t0_ensemble_link_tests_without_transceivers.conf without transceivers)
./bin/run_test.py link --agent-run-mode mono \
  --filter_file ./share/hw_sanity_tests/t0_ensemble_link_tests.conf \
  --config ./share/link_test_configs/$CONFIG \
  --qsfp-config /opt/fboss/share/qsfp_test_configs/$QSFP_CONFIG \
  --known-bad-tests-file ./share/link_known_bad_tests/agent_ensemble_link_known_bad_tests.materialized_JSON \
  --unsupported-tests-file ./share/link_known_bad_tests/agent_ensemble_link_unsupported_tests.materialized_JSON \
  --skip-known-bad-tests $KEY

All BSP tests are T0 as well.

T1 and T2

Agent HW and SAI tests use t1_agent_hw_tests.conf, t1_sai_tests.conf, t2_agent_hw_tests.conf, t2_sai_tests.conf with the same command shapes.
T1 QSFP and link tests have no filter file: their scope is everything not excluded by the known-bad and unsupported lists.
T1 includes the agent benchmarks (./bin/run_test.py benchmark --config ./share/hw_test_configs/$CONFIG --skip-known-bad-tests $KEY); T2 includes the scale benchmarks (--filter ".*Scale.*").

Filter File Format

A filter file lists one gtest pattern per line, with # comments, optionally followed by profile tags:

# Vlan
*Vlan*              t s
# L2 Learning (scale-out only: ESUN 6.2.a static MAC)
*MacLearning*       t
*MacSwLearning*     t

The tags are profiles: t for traditional (scale-out) and s for scale-up. --profile=s keeps only patterns tagged s, which is how the scale-up variants of each tier are run:

./bin/run_test.py sai_agent --filter_file=./share/hw_sanity_tests/t0_agent_hw_tests.conf \
  --profile=s --config ./share/hw_test_configs/$CONFIG \
  --enable-production-features $ASIC --skip-known-bad-tests $KEY

Note: The upstream scale-up example passes --enable-production-features without a value and an --asic $ASIC option. In the current runner, --enable-production-features takes the ASIC as its value and there is no --asic option, so use the form above.

Agent HW Test Batches

Independently of tiers, the agent HW test guide groups tests into three batches by complexity, which is a practical bring-up order:

Batch	Areas	Patterns
A: basic confidence before the first ping	VLANs, L2 learning, neighbor resolution, L3 routing, control plane, queuing, PRBS	`.Vlan.`, `.MacLearning.`, `.MacSwLearning.`, `Neighbor.`, `.L3.`, `.Copp.`, `.PacketSend.`, `.RxReason.`, `.PacketFlood.`, `.SendPacketToQueue.`, `.DscpQueueMapping.`, `.PortBandwidth.`, `.Prbs.`
B: before test teams run the full test plan	ACLs, PFC, QoS, AQM, ECMP, UDF, queue-per-host	`.Acl.`, `.DscpMarking.`, `.Pfc.`, `.Qos.`, `.Aqm.`, `.Ecmp.`, `.LoadBalancer.`, `.HwUdfTest.`, `.QueuePerHost.`
C: before production qualification	Trunks, sFlow and mirroring, PTP, MMU tuning, resource stats	`.Trunk.`, `.Sflow.`, `.Mirror.`, `.Ptp.`, `.IngressBuffer.`, `.MmuTuning.`, `.ResourceStats.`

Benchmarks

Benchmarks run through run_test.py benchmark. The runner lists benchmarks with the binary's --bm_list, drops known-bad ones, runs each benchmark in its own process for full setup and teardown isolation, parses the --json metrics, and compares them with per-platform thresholds:

SAI thresholds are looked up by benchmark name in share/hw_benchmark_tests/sai_bench.materialized_JSON, preferring entries for the current mode and falling back to the other mode.
QSFP thresholds come from share/hw_benchmark_tests/qsfp_bench.materialized_JSON, selecting entries whose test_config_regex matches the <platform>/<phy_sdk> key.
QSFP benchmarks (qsfp_hw_test_benchmark) initialize a WedgeManager in-process from --qsfp-config and force a cold boot per benchmark, so qsfp_service, FSDB, and the agent are not needed.

Benchmark binaries are only installed when you build with --benchmark-install (see run-getdeps.py Reference). Published results for many ASICs and SDKs are in the repo's docs/static/test_results/ CSV files.

Link Test Setup

Link tests need a snake topology: ports cabled to each other so that traffic can loop through every port. The procedure is:

Cable the topology described in the repo's QSFP and link test topology document.
Generate an agent config that matches the cabling (link test configs are in fboss/oss/link_test_configs/, and fboss/oss/fboss_link_test_topology/ holds topology data).
Generate the QSFP config.
Build qsfp_service and the link test binaries.
Run with run_test.py link, which starts qsfp_service and FSDB for you.

A Practical Onboarding Order

Platform tests and BSP tests (T0): the chassis, sensors, fans, and EEPROMs work.
sai_agent and sai T0, starting with AgentEmptyTest.CheckInit, then Batch A.
QSFP T0.
Link T0 once ports are cabled.
T1, then T2, across all suites, then the full suites minus known-bad.
Warm boot soak (--num-warmboot-iterations) and benchmarks.