Troubleshooting
This page lists common problems with building, packaging, running, and testing FBOSS as symptom, cause, and fix, including several surprises found in the code and scripts. For the tools mentioned here, see Diagnostic Tools.
Building
The build dies with odd compiler errors or is killed
Cause: Out of memory. FBOSS translation units are large.
Fix: Lower parallelism with getdeps' --num-jobs N. Do not delete the scratch directory as a first step; it forces a full rebuild and rarely fixes the actual failure.
My code changes are not in the binaries
Cause: getdeps builds the FBOSS commit pinned in build/deps/github_hashes/facebook/fboss-rev.txt, not your working tree.
Fix: Add --src-dir . to the getdeps arguments. See Building FBOSS.
docker build of the FBOSS image fails
Cause: Usually a package or download failure.
Fix: Load the prebuilt image exported by the export-centos-docker-image.yml or export-debian-docker-image.yml GitHub Actions workflows (zstd -d ...; sudo docker load < ...). It contains the dependency sources, so later builds need no network.
Error: --npu-sai-sdk-version is required when using --npu-sai-impl=...
Fix: Always pass --npu-sai-sdk-version with --npu-sai-impl. Values are listed on run-getdeps.py Reference.
Error: SAI_VERSION is X but the SDK's SAI headers are Y
Cause: --npu-sai-version (or a SAI_VERSION already in your environment) disagrees with sai/inc/saiversion.h in the SDK.
Fix: Drop --npu-sai-version or make it match. If the value came from the environment and you use --preserve-env, unset it.
Failed to switch to Clang via update-alternatives
Cause: run-getdeps.py switches the container to Clang when it finds GCC active, and the switch failed (for example outside the FBOSS container).
Fix: Run inside the FBOSS build container, run the printed update-alternatives command yourself, or pass --use-gcc.
Warning: CMakeLists.txt not found, skipping clang-specific flags
Cause: run-getdeps.py reads Clang flags from CMakeLists.txt in the current directory.
Fix: Run it from the repository root (/var/FBOSS/fboss in the container).
The PHY build cannot find the PAI SDK
Cause: CMake reads the PAI SDK from the hard-coded path /var/FBOSS/pai_impl.
Fix: Pass --phy-pai-sdk-path or --phy-pai-sdk-tarball so the wrapper links it there, or populate that directory with lib/ (libepdm.a, libpai.a, libphymodepil.a) and include/ (sai/, pai_macsec/, epdm/).
Packaging and Start-Up
A binary fails with error while loading shared libraries
Cause: Missing library on the switch, or the environment is not set.
Fix: source /opt/fboss/bin/setup_fboss_env, then ldd /opt/fboss/bin/<binary> | grep "not found". Install missing libraries with dnf, or copy them from the build's <scratch-path>/installed/<lib>*/ into /opt/fboss/lib/. Package with --copy-root-libs to include libraries from /lib and /lib64.
I copied the wrong package directory
Cause: package-fboss.py names the package fboss_bins-1<random suffix> and old directories are not removed.
Fix: rm -rf <scratch>/fboss_bins* before packaging, as the packaging guide does.
setup.py fails
Cause: It only recognizes two Broadcom ASICs through lspci (BCM56960 and b980), uses the current directory as the FBOSS root, and expects share/bcm_configs/, which the packager does not create.
Fix: On other platforms skip it: source ./bin/setup_fboss_env and provide /var/facebook/fboss/fruid.json yourself (examples in share/<asic>/fruid.json). On supported ones, run it from /opt/fboss and provide /etc/coop/bcm.conf if needed.
The agent ignores my config, or settings vanished
Cause: The agent parses the config permissively. Unknown keys are ignored silently, and if the sw section parses as an empty SwitchConfig, the agent prints Not valid AgentConfig, fallback to parsing as SwitchConfig... and treats the whole file as a SwitchConfig, dropping defaultCommandLineArgs and platform.
Fix: Check stderr for that message and for Overriding default flag from config: ... lines. Validate the JSON and the key names against agent_config.thrift and switch_config.thrift. See Agent Configuration.
A flag in defaultCommandLineArgs has no effect
Cause: Those entries only set flag defaults; a flag given on the command line or in the systemd unit wins. Fix: Remove the explicit flag from the unit, or change it there.
The hardware agent cannot connect to the software agent (split mode in a container)
Cause: The hardware agent connects to ::1. With no non-loopback IPv6 address, getaddrinfo(..., AI_ADDRCONFIG) returns no IPv6 results, the Thrift servers bind to IPv4 only, and the connection is refused. Docker's default bridge disables IPv6 on eth0.
Fix: Run the container on an IPv6-enabled network. fboss-sim-docker-run.py creates one (fboss_sim_net_<user>).
fboss_sw_agent crashes in the simulator in TunManager
Cause: TUN interfaces are not supported under fake SAI (the simulator's setup notes a crash in getTableId).
Fix: Run with --tun_intf=false, as the fboss-sim configs and systemd drop-in do.
Platform services crash at start-up
Cause: The platform could not be identified (no usable dmidecode -s system-product-name), or there is no config for it.
Fix: Check /var/facebook/fboss/platform_name and dmidecode -s system-product-name. The name is cached in that file: delete it after changing what the BIOS reports. Use --config_file to point at a config explicitly. Note that the READMEs mention --config, but the flag is --config_file.
fan_service fails with a port conflict
Cause: Another fan_service (or something else) is listening on 5972.
Fix: Stop the other instance or start with --thrift_port.
Runtime
fboss2 says it cannot connect
Cause: The target service is down, on another port, or the CLI is pointed at the wrong host.
Fix: systemctl status fboss_sw_agent qsfp_service fsdb; use --agent-port, --qsfp-port, --fsdb-port, or -H. When the agent is down, use Diagnostic Tools.
A fboss2 set ... change was reverted
Cause: Runtime set changes are not persisted. Any config reload, session commit (even for an unrelated change), or agent restart re-applies agent.conf.
Fix: Make persistent changes with fboss2-dev config ... and config session commit. See Config vs set.
--filter on fboss2 fails with Each filter term must be of the form <key op value>.
Cause: Terms are split on single spaces and must have exactly three parts.
Fix: Write "name == eth1/1/1", not "name==eth1/1/1", and avoid values containing spaces.
The switch always cold boots
Cause: Warm boot state lives in /dev/shm/fboss/warm_boot/, which does not survive a reboot, and the can_warm_boot flag file must exist for the next start to warm boot.
Fix: Warm boot only applies to service restarts. Check /var/facebook/logs/fboss/fboss_sw_agent_boot_history.log for WARM_BOOT and COLD_BOOT entries.
Logs fill the disk even though rotation runs
Cause: Rename-based log rotation. Services write through file descriptors handed over by systemd (StandardOutput=append:) and cannot reopen files.
Fix: Use logrotate's copytruncate, as the distro image's /etc/logrotate.d/fboss does.
The agent crashed after a config change
Fix: Look at /var/facebook/fboss/crash/bad_update/old_state and new_state (the states before and after the failing update), journalctl -u fboss_sw_agent, and coredumpctl info fboss_sw_agent.
Testing
run_test.py exits immediately with status 1 and prints help
Cause: No test type was given. Pass one, such as sai_agent.
Error: Only one of the --filter or --filter_file can be specified at any time
Fix: Use one or the other. --profile likewise needs --filter_file or --tier.
Tests that should be skipped run anyway
Cause: --skip-known-bad-tests was not given, or the key does not match any entry.
Fix: Find the exact key in share/hw_known_bad_tests/*.materialized_JSON and share/sai_hw_unsupported_tests/*.materialized_JSON (format vendor/coldboot-sdk/warmboot-sdk/asic[/mode]).
Benchmarks always pass
Cause: Without --skip-known-bad-tests, threshold validation is skipped (NO_THRESHOLD); benchmarks without a threshold entry print WILL ALWAYS PASS.
Fix: Pass the platform key so thresholds are looked up.
Benchmarks ran in mono mode
Cause: benchmark defaults to --agent-run-mode mono, unlike the other agent test types.
Fix: Pass --agent-run-mode multi_switch.
The scale-up command from the docs fails to parse
Cause: The upstream example passes --enable-production-features with no value and an --asic option that does not exist.
Fix: Use --enable-production-features $ASIC. See Test Suites.
Getting Help
Use GitHub Issues or Discussions on meta/fboss. Report security problems through Meta's bug bounty program rather than a public issue, as CONTRIBUTING.md asks.