Packaging and Running on a Switch

Once FBOSS is built, you copy it to a switch and run it from /opt/fboss. This page covers the packaging script, what the package contains, installing it on the switch, setting up the environment, the setup.py helper, the systemd services, and running binaries by hand. Running the test suites is covered separately in Running Tests.

Requirements on the Switch

FBOSS binaries built as described in Building FBOSS, against the SDK for the switch's ASIC (a fake SAI build only makes sense in a container).
A switch running CentOS 9. The packaged binaries are built in a CentOS Stream 9 container.
The ASIC vendor SDK's kernel modules built per the vendor's instructions and installed on the switch.
For the platform stack, a BSP (kernel modules for FPGAs, I2C, sensors) installed or installable by platform_manager. Meta's reference BSP kmods are at github.com/facebookincubator/fboss.bsp.kmods.

An alternative to all of this is the FBOSS lab distribution image, which installs a complete OS with FBOSS on a switch. See Lab Distro Image.

Step 1: Package the Build

Run this inside the build container:

rm -rf /var/FBOSS/tmp_bld_dir/fboss_bins*
cd /var/FBOSS/fboss

# Package as a directory under /var/FBOSS/tmp_bld_dir/
./fboss/oss/scripts/package-fboss.py --copy-root-libs --scratch-path /var/FBOSS/tmp_bld_dir/

# or package as a tarball /var/FBOSS/tmp_bld_dir/fboss_bins.tar.zst
./fboss/oss/scripts/package-fboss.py --copy-root-libs --scratch-path /var/FBOSS/tmp_bld_dir/ --compress

Option	Meaning
`--scratch-path PATH`	Required. The getdeps scratch path used for the build.
`--compress`	Also produce `fboss_bins.tar.zst` in the scratch path.
`--copy-root-libs`	Also copy needed libraries found in `/lib` and `/lib64`. Default: off.

The package directory is created with tempfile.mkdtemp(prefix="fboss_bins-1", ...), so its name is fboss_bins-1 followed by a random suffix (the $PKG_ID in the commands below). Clean old fboss_bins* directories first, as above, so you copy the right one.

What the Package Contains

Path in package	Contents	Source in the repo
`bin/`	All built binaries, `run_test.py` and its `fboss_test_runner` package, `setup.py`, `setup_fboss_env`, `npu_sdk_utils.py`	build output, `fboss/oss/scripts/run_scripts/`
`lib/`	Shared libraries the binaries need	resolved with `ldd` from the build tree
`share/hw_test_configs/`	Agent test configs (`<platform>.agent.materialized_JSON`)	`fboss/oss/hw_test_configs/`
`share/qsfp_test_configs/`	QSFP test configs	`fboss/oss/qsfp_test_configs/`
`share/link_test_configs/`	Link test configs	`fboss/oss/link_test_configs/`
`share/hw_sanity_tests/`	T0/T1/T2 filter files and test metadata	`fboss/oss/hw_sanity_tests/`
`share/hw_benchmark_tests/`	Benchmark thresholds	`fboss/oss/hw_benchmark_tests/`
`share/hw_known_bad_tests/`, `qsfp_known_bad_tests/`, `link_known_bad_tests/`	Known-bad test lists	`fboss/oss/*_known_bad_tests/`
`share/sai_hw_unsupported_tests/`, `qsfp_unsupported_tests/`	Unsupported test lists	`fboss/oss/*_unsupported_tests/`
`share/production_features/`	Per-ASIC production feature lists	`fboss/oss/production_features/`
`share/<asic>/`, `share/default_configs/`	Per-ASIC `bde.conf` and `fruid.json` (for `th`, `th3`, `th4`, `th5`, `th6`, `j3`, `r3`, `q4d`, ...), and per-platform default `agent.conf`, `qsfp.conf`, `led.conf`, `num_hw_agents`	`fboss/oss/scripts/run_configs/` (every subdirectory is copied)
`share/platform_descriptors/` and forwarding stack configs	Platform descriptors and generated configs	`fboss/configs/platforms/`
`share/bash-completion/completions/`	Bash completion for `fboss2`	`fboss/oss/scripts/fboss2_completion.bash`
`share/npu_sdk_metadata.json`	NPU SDK metadata, when the build produced it	build output

Step 2: Copy It to the Switch

From the host where the build container runs:

# Directory
scp -r /opt/app/FBOSS_DIR/tmp_bld_dir/fboss_bins-$PKG_ID root@$SWITCHNAME:/opt/

# or tarball
scp /opt/app/FBOSS_DIR/tmp_bld_dir/fboss_bins.tar.zst root@$SWITCHNAME:/opt/

Here /opt/app/FBOSS_DIR/tmp_bld_dir stands for the host side of the container's /var/FBOSS/tmp_bld_dir mount. On the switch:

# Directory
cd /opt
ln -s /opt/fboss_bins-$PKG_ID /opt/fboss

# or tarball
cd /opt
mkdir fboss && mv fboss_bins.tar.zst fboss/
cd fboss && tar -xvf fboss_bins.tar.zst

Either way you end up with /opt/fboss/ containing bin/, lib/, and share/.

Step 3: Set Up the Environment

cd /opt/fboss
source ./bin/setup_fboss_env

setup_fboss_env derives the FBOSS root from its own location (bin/..), so it works from any directory. It exports:

Variable	Value
`FBOSS`	`/opt/fboss`
`FBOSS_BIN`	`$FBOSS/bin`
`FBOSS_LIB`	`$FBOSS/lib`
`FBOSS_LIB64`	`$FBOSS/lib64`
`FBOSS_KMODS`	`$FBOSS/lib/modules`
`FBOSS_DATA`	`$FBOSS/share`

It prepends FBOSS_BIN to PATH and FBOSS_LIB64 and FBOSS_LIB to LD_LIBRARY_PATH. Sourcing it twice is harmless: it returns early if FBOSS already points at the same root.

Check Dependencies

Verify that every shared library a binary needs resolves:

ldd /opt/fboss/bin/$BINARY_NAME | grep "not found"

No output is good. A line like libre2.so.9 => not found means a library is missing. Install it with sudo dnf install ... if the switch has internet access, or copy it from the build's scratch path (<scratch-path>/installed/<missing_lib*>/) to /opt/fboss/lib/.

Step 4: Run `setup.py` (Broadcom Test Setups)

cd /opt/fboss
./bin/setup.py

setup.py prepares a switch for running Broadcom-based tests. It:

Runs lspci and recognizes two ASICs: a Broadcom BCM56960 (Tomahawk, treated as WEDGE100S+RSW) and a Broadcom b980 device (Tomahawk3, treated as MINIPACK+FSW).
Copies share/<th|th3>/fruid.json to /var/facebook/fboss/fruid.json, the matching share/bcm_configs/<PLATFORM>-bcm.conf to /etc/coop/bcm.conf, and bde.conf to /etc/modprobe.d/bde.conf, but only if those files do not already exist.
Symlinks linux-user-bde.ko and linux-kernel-bde.ko from $FBOSS/lib/modules into /lib/modules/$(uname -r), runs depmod -a if it created a link, and loads both modules with modprobe if lsmod does not show them.

With --reload it first deletes those files and links and unloads the modules, then sets everything up again.

Warning: setup.py uses the current working directory as the FBOSS root, so run it from /opt/fboss. On any other ASIC it does not set its source paths and fails. It also reads share/bcm_configs/, which package-fboss.py does not create; if /etc/coop/bcm.conf does not already exist, provide that directory (or the file) yourself. The onboarding guide says plainly that it "may not succeed for all platforms"; in that case just source ./bin/setup_fboss_env and provide fruid.json yourself.

fruid.json identifies the platform for the agent and qsfp_service. Its Information object contains Product Name (for example MONTBLANC), serial number, part numbers, and the Local MAC and Extended MAC Base/Extended MAC Address Size used to derive port MACs. Example files are in fboss/oss/scripts/run_configs/<asic>/fruid.json.

Step 5: Run the Services

Under systemd

On a production-style system each daemon is a systemd unit. The unit names used by the repo's scripts and the distro image include:

Unit	Binary
`fboss_sw_agent`	Software agent (split mode)
`fboss_hw_agent@N`	Hardware agent for switch index `N` (split mode)
`wedge_agent`	Monolithic agent
`qsfp_service`	Transceiver service
`fsdb`	State database
`platform_manager`, `sensor_service`, `fan_service`, `data_corral_service`, `led_service`	Platform services

Restart in dependency order when recovering a switch:

systemctl restart platform_manager
systemctl restart qsfp_service
systemctl restart fboss_hw_agent@0
systemctl restart fboss_sw_agent

The unit files for the distro image live in fboss-image/image_builder/templates/centos-09.0/root_files/, and the fboss-sim container installs the same ones.

Starting Services With `run_test.py`

run_test.py can start and stop the forwarding services for you without running tests, which is the quickest way to bring a lab switch up:

./bin/run_test.py services start --agent-config $AGENT_CONFIG --qsfp-config $QSFP_CONFIG
./bin/run_test.py services start agent --agent-config $AGENT_CONFIG
./bin/run_test.py services stop --num-npus 1

See Running Tests.

By Hand

Stop the service first, then run the binary with the flags you need. For example, a hardware agent with verbose logging:

systemctl stop fboss_hw_agent@0
/opt/fboss/bin/fboss_hw_agent-sai_impl --switchIndex=0 --config /etc/coop/agent.conf --minloglevel=0 --logging DBG5

Run a single SAI test directly against a packaged config:

./bin/sai_test-sai_impl-1.12.0 --config ./share/hw_test_configs/fuji.agent.materialized_JSON \
  --filter=HwVoqSwitchWithFabricPortsTest.init

(The -1.12.0 suffix in that example from the repo's guide reflects the SAI version of one particular build; use the name your build produced.)

Logs and State on the Switch

Location	Contents
`journalctl -u <unit>`	Service logs, including crash stack traces from systemd-coredump
`/var/facebook/logs/fboss/`	Per-service log files, `fboss_sw_agent_boot_history.log`, `*_snapshots.log`; rotated logs in `archive/`
`/var/facebook/fboss/`	Persistent state: `fruid.json`, `platform_name`, `running-agent.conf`, `crash/`, SDK dumps
`/dev/shm/fboss/`	Volatile state, including `warm_boot/`
`coredumpctl list`	Core dumps

Note: If you set up log rotation yourself, keep logrotate's copytruncate option. FBOSS services get their log file descriptor from systemd (StandardOutput=append:) and cannot reopen a renamed file, so rename-based rotation leaves the service writing into the rotated file and frees no space.