Lab Distro Image

The FBOSS lab distribution ("FBOSS Distro") is a complete, bootable OS image for a switch: a CentOS-based system with the FBOSS kernel, BSPs, the platform stack, the forwarding stack, and the SAI runtime already installed, whose services start automatically. It is the fastest way to get a lab switch running FBOSS. This page explains what the image contains, how its manifest describes a build, how to build it with the fboss-image tool, and how to provision switches.

What Runs on a Distro Switch

Stack	Services	CLIs
Platform stack	`platform_manager` (installs the BSP kernel modules and configures hardware), `fan_service`, `sensor_service`, `data_corral_service`	`weutil`, `fw_util`, sensor data client
Forwarding stack	FSDB, the agent (`fboss_sw_agent` plus one `fboss_hw_agent` per ASIC), `qsfp_service`; a BGP daemon is announced as coming	`fboss_bcm_shell`, `bcm_shell` (per the provisioning guide)

Services start in the order the hardware requires: platform first, then forwarding. FBOSS files land under /opt/fboss. The image's systemd units and root file system overlay come from fboss-image/image_builder/templates/centos-09.0/root_files/ (the same files the fboss-sim container uses), and the image ships a logrotate policy for /var/facebook/logs/fboss/ (100 MB per log, 20 compressed generations, run every 15 minutes).

The Manifest

An image build is described by a JSON manifest. Ready-made manifests live in fboss-image/manifests/, for example 6.11.1_brcm_xgs_13.3.0.0_kmod_6.5.32.json, 6.11.1_brcm_dnx_14.2.0.0_kmod_6.5.34.json, 6.11.1_cisco_25.5.4210-3.json, 6.11.1_fake.json (fake SAI), kernel_only.json, and per-platform ones such as minipack3.json.

{
  "distribution_formats": {
    "pxe": "FBOSS-k6.4.3.tar",
    "onie": "FBOSS-k6.4.3.bin",
    "usb": "FBOSS-k6.4.3.iso"
  },
  "kernel": {
    "download": "https://example.com/fboss-oss_kernel_v6.4.3.tar"
  },
  "other_dependencies": [
    {"download": "file:vendor_debug_tools/tools.rpm"}
  ],
  "fboss-platform-stack": {
    "execute": ["../fboss/oss/scripts/build_fboss_stack.sh", "platform"]
  },
  "bsps": [
    {"execute": "vendor_bsp/build.make"}
  ],
  "npu_sai": {
    "execute": "fboss_brcm_sai/build.sh"
  },
  "phy_sai": {
    "execute": ["../fboss/oss/scripts/build_phy_sai.sh"]
  },
  "fboss-forwarding-stack": {
    "execute": ["../fboss/oss/scripts/build_fboss_stack.sh", "forwarding"]
  },
  "image_build_hooks": {
    "after_pkgs_install": "additional_os_pkgs.json",
    "after_pkgs_execute": "additional_image_mods.json"
  }
}

Keys

Key	Meaning
`distribution_formats`	Output images to produce: `onie` (ONIE installer, `.bin`), `usb` (bootable ISO, `.iso`), `pxe` (network install tarball, `.tar`)
`kernel`	The kernel; its artifact is an uncompressed tarball of kernel RPMs in the standard Red Hat layout (see `fboss-image/kernel`)
`other_dependencies`	Extra RPMs, one per entry, installed into the image (runtime dependencies resolved by dnf); see `examples/thirdparty_build.sh`
`fboss-platform-stack`, `fboss-forwarding-stack`	Built by `fboss/oss/scripts/build_fboss_stack.sh`; each artifact is a tarball extracted under `/opt/fboss`. The platform stack is built with only the kernel installed, the forwarding stack also with the SAI
`bsps`	List of BSP builds; each artifact is a tarball of RPMs copied (not installed) into `/usr/local/share/local_rpm_repo`, from which `platform_manager` installs them at run time
`npu_sai`	The hardware agent's SAI; artifact contains `sai_build.env`, `include/`, `lib/libsai_impl.a`, and `sai-runtime.rpm` (installed into the image)
`phy_sai`	The QSFP service's PHY SAI; artifact contains `phy_sai_build.env`, `include/`, `lib/libphy_sai.a`, and `phy_sai-runtime.rpm`
`image_build_hooks`	`after_pkgs_install`: a JSON file `{"packages": [...]}` of extra CentOS packages; `after_pkgs_execute`: a JSON file `{"execute": [[cmd, args...], ...]}` of commands run after package installation

Each component is either download (a URL or file; file: paths are relative to the manifest, file:/ paths are absolute; archives may be compressed) or execute (a script, or an array of script and arguments, run in the script's directory inside a fresh build container, with dependencies installed or available in /deps; it must put its artifact in /output). Build scripts are responsible for their own caching. List components such as bsps are built independently and their artifacts collected.

sai_build.env carries the variables the FBOSS build needs, for example:

SAI_BRCM_IMPL=1
SAI_SDK_VERSION=SAI_VERSION_13_3_0_0_ODP
SAI_VERSION=1.16.1

fboss/oss/scripts/fake-sai-devel.tar.zstd is a special npu_sai artifact that switches the forwarding stack to the fake SAI build.

The fboss-image Tool

fboss-image (in fboss-image/distro_cli/) builds images and manages lab devices together with the Distro Infrastructure container. It needs Python 3.10 or newer and no external packages.

Building

./fboss-image build manifest.json              # all components and all image formats
./fboss-image build manifest.json kernel sai   # only rebuild these components

With no component names, every component is built and every format in distribution_formats is produced. With component names, only those components are rebuilt and no complete image is produced. The FBOSS_BUILDER_CACHE_EXPIRATION_HOURS environment variable (default 24) controls how long the builder Docker image is cached regardless of content changes.

Managing Devices

Devices are identified by the Ethernet MAC address of their management port, which lets the tool support PXE without taking over your DHCP server. The Distro Infrastructure container passively learns IP-to-MAC mappings; if it has none, the tool scans, which only works when your workstation is on the same L2 segment as the management port.

Command	Action
`fboss-image device <MAC> image-upstream <kernel-version> <sai>`	Fetch a reference image from Meta and serve it to the device
`fboss-image device <MAC> image <image-file>`	Serve a given image; PXE or ONIE installer is detected
`fboss-image device <MAC> reprovision`	Log in, wipe FBOSS to a factory-fresh state, and reboot to netboot
`fboss-image device <MAC> update <manifest> <component>`	Build a component, copy it to the device, and restart the affected services (not all components, for example not the kernel)
`fboss-image device <MAC> getip`	Print the device's IP
`fboss-image device <MAC> ssh`	Same as `ssh admin@$(fboss-image device <MAC> getip)`

Provisioning a Switch

Requirements: the BMC and eth0 get addresses by SLAAC on the management network.

Method	How
USB	Write the ISO to a USB drive with any imaging tool and boot from it
PXE, own DHCP server	Configure your lab DHCP server (Kea, dhcpd, ...) to serve the image to specific MACs
PXE, dedicated provisioning server	A separate DHCP server on the management L2 network that only answers PXE requests for target MACs
PXE, Distro container	The FBOSS Distro container serves PXE with dnsmasq; deploy images with `fboss-image`
ONIE	ONIE installer images (support in development)

PXE requests must not race with other DHCP servers for the eth0 address.

Custom Binaries on a Distro Switch

Run your controller or tool in a Docker container on the switch (the recommended approach), or
rebuild it in the FBOSS build environment (see Building FBOSS), copy it to the switch, and run it.