Diagnostic Tools

Besides fboss2, FBOSS ships a set of single-purpose utilities for transceivers, EEPROMs, firmware, health checks, and low-level ASIC access. Several of them work when the agent or qsfp_service is down, which is exactly when fboss2 cannot help. All of them are installed in /opt/fboss/bin/, which is not on the default PATH until you source setup_fboss_env (see Packaging and Running on a Switch).

Tool Purpose Needs a running service?
wedge_qsfp_util Read and control transceivers qsfp_service, unless --direct_i2c
weutil Read and decode FRU EEPROMs No
fw_util Read, verify, and program firmware (BIOS, FPGA, CPLD, ...) No
fixmyfboss Automated health checks No
rma-showtech Collect system information for support No
diag_shell_client SAI vendor diagnostic shell A hardware agent
sensor_service_client Query sensor_service sensor_service
rackmon Rack power monitoring service

wedge_qsfp_util

wedge_qsfp_util reads and controls transceivers. By default it goes through qsfp_service over Thrift (port 5910, SSL policy encrypted by default via --ssl_policy). With --direct_i2c it opens the transceiver bus itself using the BSP drivers, which is the way to debug optics when qsfp_service is down. Direct access bypasses qsfp_service, which normally owns the transceiver bus, so prefer it for the case where the service is down.

# Read transceiver info without qsfp_service
/opt/fboss/bin/wedge_qsfp_util eth1/1/1 --direct_i2c

# More detail, including thresholds
/opt/fboss/bin/wedge_qsfp_util eth1/1/1 --direct_i2c --verbose

# Read 128 bytes starting at register 0
/opt/fboss/bin/wedge_qsfp_util eth1/1/1 --direct_i2c --read_reg --offset 0 --length 128

# Hard reset a module
/opt/fboss/bin/wedge_qsfp_util eth1/1/1 --direct_i2c --qsfp_hard_reset

# One line per port: name, vendor, part number
/opt/fboss/bin/wedge_qsfp_util --direct_i2c --port_info_summary

The upstream debugging guide writes these flags with dashes (--direct-i2c, --read-reg); the flag names defined in fboss/util/wedge_qsfp_util.cpp use underscores.

Main Options

Option Default Meaning
--direct_i2c false Access transceivers over I2C instead of through qsfp_service
--platform Platform on which the tool runs
--verbose false Print more details
--list_commands false Print the list of commands
--read_reg, --write_reg false Read or write a register; use with --offset (0 to 255), --length (1 to 128, default 1) or --data (comma-separated bytes), and --page
--i2c_address 0x50 I2C address
--channel -1 Channel within the module, starting at 1
--tx_disable, --tx_enable Set or clear TX disable bits, optionally per --channel
--set_low_power, --clear_low_power Force or release low-power mode (--clear_low_power is needed for LR4 optics)
--set_40g, --set_100g, --app_sel Rate select for CWDM4 modules, application select for CMIS modules
--cdr_enable, --cdr_disable Set or clear CDR bits
--qsfp_hard_reset false Hard reset the module
--qsfp_reset false Reset modules, through qsfp_service unless --direct_i2c; type HARD_RESET, action RESET_THEN_CLEAR
--electrical_loopback, --optical_loopback, --host_output_loopback, --media_output_loopback, --clear_loopback Module loopback modes (the last two are CMIS only); --skip_check skips safety checks
--pause_remediation N 0 Stop qsfp_service from remediating modules for N seconds
--update_module_firmware Upgrade module firmware with --firmware_filename, --msa_password, --image_header_len
--update_bulk_module_fw Bulk firmware upgrade with --module_type, --fw_version, --port_range (for example 1,3,5-8), --dsp_image
--get_module_fw_info Module firmware info for a port range
--cdb_command Read the CDB block with --command_code and --cdb_payload
--vdm_info VDM monitoring info
--prbs_start, --prbs_stop, --prbs_stats Module line-side PRBS with --generator or --checker; --prbs_pattern default PRBS31Q
--batch_ops --batchfile FILE Bulk read/write operations, one OP(R/W) Offset Value DelayMS per line; requires --direct_i2c
--capabilities, --module_io_stats, --dump_tcvr_i2c_log Module capabilities, I/O stats, dump the I2C log to /dev/shm/fboss/qsfp_service
--open_timeout 30 Seconds to wait to open the bus

weutil

weutil reads FRU EEPROMs (chassis, SCM, PIMs, ...) and prints the decoded content: product name, serial number, MACs, and so on. It detects the platform and loads the matching EEPROM list from its built-in config.

/opt/fboss/bin/weutil                 # chassis EEPROM
/opt/fboss/bin/weutil --list          # list EEPROMs known for this platform
/opt/fboss/bin/weutil --all           # print every EEPROM
/opt/fboss/bin/weutil --eeprom SCM    # one EEPROM by name or device type
/opt/fboss/bin/weutil --json          # JSON output
/opt/fboss/bin/weutil --path /sys/bus/i2c/devices/<dev>/eeprom --offset 0
Option Default Meaning
--eeprom NAME chassis EEPROM name or device type
--path PATH Ignore the config and read this EEPROM file
--offset N 0 Offset for --path
--list false List the EEPROMs in the config
--all false Print all EEPROMs
--json false JSON output

--path and --eeprom are mutually exclusive (Please use either --path or --eeprom, not both!). The EEPROM formats are specified in the repo's Meta EEPROM v5 and v6 format documents.

fw_util

fw_util manages firmware of the field-programmable devices (FPDs): BIOS, FPGAs, CPLDs, and others, as defined per platform in its built-in config (or --config_file).

fw_util                                       # same as --fw_action=version --fw_target_name=all
fw_util --fw_action=list
fw_util --fw_action=version --fw_target_name=CPLD
fw_util --fw_action=program --fw_target_name=bios --fw_binary_file=/path/to/firmware.bin
fw_util --fw_action=verify --fw_target_name=FPGA --fw_binary_file=/path/to/firmware.bin
fw_util --fw_action=audit
Option Default Meaning
--fw_action version One of program, verify, read, version, list, audit
--fw_target_name all Firmware target; --fw_action=list shows the names for this platform
--fw_binary_file Image to program or verify against
--config_file Override the built-in platform config
--verify_sha1sum false Verify the image's SHA1 sum
--dry_run false Do everything except programming
--version Print the tool's build version

fw_util logs to /var/facebook/logs/fboss/fw_util.log. The code still contains a fallback for an old positional form (fw_util <target> <action> <file>) with a deprecation warning, but because --fw_action now defaults to version, use the named flags.

Warning: program rewrites device firmware. A wrong image can brick a FRU. Use --dry_run first.

fixmyfboss

fixmyfboss runs automated health checks on the local switch, or remotely over SSH.

/opt/fboss/bin/fixmyfboss --list-checks
/opt/fboss/bin/fixmyfboss --verbose
/opt/fboss/bin/fixmyfboss --checks "mac|pci" --json
Option Meaning
--list-checks Show the available checks and exit
--checks REGEX Only run checks whose name matches (case-insensitive)
--include-disruptive Also run checks that reboot or power cycle the switch
--hostname HOST Diagnose a remote switch over SSH
--transport auto|ssh|sush2 How to reach --hostname
--bmc-hostname HOST / --no-bmc BMC to check (default <hostname>-oob with --hostname), or skip BMC checks
--hw-tests-dir DIR Prebuilt platform hardware tests to copy to the switch and run (disruptive)
--bmc-platform NAME Platform for the BMC test runner
--json Print a JSON report
-v, --verbose / -d, --debug INFO or DBG logging

Checks listed in the upstream guide include MAC address correctness, PCI devices present, recent manual reboot, recent kernel panic, watchdog not stopped, and i801 SMBus timeouts.

rma-showtech

rma-showtech collects system information for troubleshooting or RMA. --details is required and takes a comma-separated list:

/opt/fboss/bin/rma-showtech --details all
/opt/fboss/bin/rma-showtech --details fboss,logs,sensor

Detail names listed in the upstream guide: all, fan, fanspinner, fboss, fwutil, gpio, host, i2c, i2cdump, logs, lspci, nvme, pem, port, powergood, psu, sensor, weutil. Some (for example fwutil, i2c, i2cdump, pem, psu) are disruptive; the --disruptive flag enables disruptive functions. --config_file overrides the platform's showtech config.

diag_shell_client

diag_shell_client connects to the vendor SAI diagnostic shell inside a hardware agent. It needs fboss_hw_agent@N to be running; the default port is 5931 (switch index 0).

/opt/fboss/bin/diag_shell_client
/opt/fboss/bin/diag_shell_client --host ::1 --port 5931

Commands inside the shell are ASIC-vendor specific. A similar fboss_pai_diag_shell_client connects to the PHY (PAI) diag shell through qsfp_service on port 5910.

When the CLI Is Down

  1. Check services: systemctl status 'fboss*' platform_manager qsfp_service fsdb.
  2. Read logs: journalctl -u fboss_hw_agent@0 -f, and files in /var/facebook/logs/fboss/.
  3. Look for crash dumps in /var/facebook/fboss/crash/ and coredumpctl list.
  4. Check optics with wedge_qsfp_util --direct_i2c and FRUs with weutil.
  5. Run fixmyfboss.

More scenarios are in Troubleshooting.

See Also