FSDB

FSDB (FBOSS State Database) is the switch's central publish/subscribe store. On-box services publish their state and statistics into it as Thrift trees; local and remote consumers subscribe to whole trees or to paths within them and receive updates as Thrift streams. This page covers the data model, the API, the configuration file, the server's flags, and how publishing is enabled in the agent and other services.

Model: State and Stats

FSDB keeps two separate trees:

Tree Nature Root struct (open source model) Publishers
State Event-driven: changes when something happens FsdbOperStateRoot with agent (AgentData: config, switchState, FSDB subscription states, agent info, internal PHY states), bgp (BgpData), qsfp_service Agent, BGP, qsfp_service
Stats Periodic metrics FsdbOperStatsRoot with agent (AgentStats), qsfp_service, sensor_service, te_srv6_agent Agent, qsfp_service, sensor_service

The open source model is in fboss/fsdb/if/oss/fsdb_model.thrift. Because the agent publishes its full SwitchState into the state tree, a subscriber to ["agent", "switchState"] sees exactly what the agent sees, without polling its Thrift API.

API

The service is FsdbService in fboss/fsdb/if/fsdb.thrift, on port 5908 by default. Publishers use Thrift sinks and subscribers use Thrift streams:

Method family Direction Payload
publishOperStatePath, publishOperStatsPath Publisher to FSDB Full OperState snapshots of a path
publishOperStateDelta, publishOperStatsDelta Publisher to FSDB OperDelta changes
subscribeOperStatePath, subscribeOperStatsPath FSDB to subscriber Snapshots of one path
subscribeOperStateDelta, subscribeOperStatsDelta FSDB to subscriber Deltas of one path
subscribeOperStatePathExtended, subscribeOperStateDeltaExtended, and stats variants FSDB to subscriber Several paths, including wildcards, in one subscription
publishState, publishStats, subscribeState, subscribeStats (and ...Extended) Both Newer generic API carrying PublisherMessage and SubscriberMessage; a comment in the IDL marks the older Oper* methods as to be phased out
getOperState, getOperStats, getOperStateExtended, getOperStatsExtended Request/response One-shot reads
getAllOperPublisherInfos, getAllOperSubscriberInfos, and per-ID variants Request/response Who is connected

C++ clients use the libraries in fboss/fsdb/client/ (there is also a cgo wrapper in fboss/fsdb/client/cgo/).

From the CLI:

fboss2 show fsdb publishers
fboss2 show fsdb subscribers
fboss2 show fsdb state
fboss2 stream fsdb stats

Configuration

FSDB reads /etc/coop/fsdb/current (--fsdb_config) when --readConfigFile is true (the default). The file is the JSON form of Config in fboss/fsdb/if/fsdb_config.thrift:

Key Meaning
defaultCommandLineArgs Flag defaults, applied like the agent's
trustedSubnets Subnets allowed to connect (DSF clusters)
publishers Per publisher ID: a list of paths, each with path ({"raw": [...]}), isExpected (default false), and isStats (default false), plus skipThriftStreamLivenessCheck (default false)
subscribers Per subscriber ID: trackReconnect (default true), allowExtendedSubscriptions (default false), numExpectedSubscriptions (default 1), expectedSubscriptionType (STATE, STATS, or ALL)
operPublishers_DEPRECATED Old form of publishers

An excerpt of the config used for link tests (fboss/oss/link_test_configs/fsdb.conf):

{
  "defaultCommandLineArgs": {
    "serveHeartbeats": "true",
    "statsSubscriptionHeartbeat_s": "5",
    "useIdPathsForSubs": "true"
  },
  "trustedSubnets": [],
  "publishers": {
    "agent": {
      "paths": [
        { "path": { "raw": ["agent"] }, "isExpected": true, "isStats": false },
        { "path": { "raw": ["agent"] }, "isExpected": true, "isStats": true }
      ],
      "skipThriftStreamLivenessCheck": false
    }
  }
}

Server Flags

Flag Default Meaning
--fsdbPort 5908 Thrift port (--migrated_fsdbPort 5958 is the planned replacement)
--fsdbPort_high_priority 5957 Port for high-priority connections (ToS 0xc0), used by the DSF subscriber
--fsdb_config /etc/coop/fsdb/current Config file
--readConfigFile true Whether to read the config file
--fsdb_timestamp_dir /dev/shm/fboss Directory for timestamp files
--streamExpire_ms 900000 (15 minutes) Delay before a connection to a silent publisher is expired
--stat_publish_interval_ms 1000 How often thread-local stats are published to fb303
--metricsTtl_s 43200 (12 hours) TTL for metrics
--ssl_policy required Thrift SSL policy: disabled, permitted, or required
--trustedSubnets empty Comma-separated DSF in-band subnets
--useIdPathsForSubs false Serve subscribers using ID paths only
--checkOperOwnership false Check that a publisher owns the paths it publishes
--enable_thrift_acceptor, --enable_tos_reflect false Connection acceptor and ToS reflection
--fsdb_queue_timeout_ms 100 Thrift server queue timeout
--fsdb_workers_join_timeout_s 4 Shutdown drain time; must be between 1 and 4, enforced by a flag validator
--memoryProfiling false Enable memory profiling

Note: --ssl_policy defaults to required; an unknown value is logged as Invalid Thrift server sslPolicy: <value>. Lab setups without certificates usually relax SSL on both sides. On the client side, the fboss-sim container starts fboss_sw_agent with --fsdb_client_ssl_preferred=false and --thrift_ssl_policy=disabled.

Turning Publishing On

Services do not publish by default. In the agent, enable it with flags (usually via defaultCommandLineArgs):

Agent flag Default Effect
--publish_state_to_fsdb false Publish switch state
--publish_stats_to_fsdb false Publish stats
--subscribe_to_state_from_fsdb false Subscribe to state from FSDB
--subscribe_to_stats_from_fsdb false Subscribe to stats from FSDB

sensor_service publishes sensor stats (served at --fsdb_statsStream_interval_seconds, default 5), and fan_service can subscribe to optics data with --subscribe_to_qsfp_data_from_fsdb.

Running and Testing

The fsdb binary is part of the fboss_forwarding_stack target. run_test.py services start fsdb starts it, and link tests start it automatically unless --disable-fsdb is given (with --fsdb-config to choose the config). FSDB's own tests and benchmarks live under fboss/fsdb/tests/ and fboss/fsdb/benchmarks/.

See Also