This document describes the configuration management system.
Terragraph implements a centralized configuration management architecture. The E2E controller serves as the point of truth for all node configurations, and is responsible for keeping all nodes' local copies in sync with its own view. All config changes are applied using the controller.
Node configuration is a JSON-serialized Thrift structure (
and is generally manipulated using folly::dynamic objects. A node's full
configuration is computed by applying several "override" layers on top of the
base configuration for its software version. All configuration files are stored
/data/cfg/ directory on disk for both the E2E controller and
ConfigApp accepts the following user operations on the node
|Get Node Config|
|Get Config Layers|
|Set Config Overrides|
|Get Config Metadata|
|Get Config Actions|
SET operations on the controller are validated using the config
metadata, which is a separate JSON file describing every configuration
GET operations are served locally except for getting
post-config actions, which sends out
GET_MINION_CONFIG_ACTIONS_REQ messages to
minions to request the actions that they would take for a given configuration
change; responses can then be polled from the controller. Additionally, the
controller handles the
EDIT_NODE command from
TopologyApp to migrate a
node's configuration upon a node name change.
The controller pushes the full node configuration to a minion's
SET_MINION_CONFIG_REQ command. The minion updates its copy of
the configuration on disk (
/data/cfg/node_config.json) and takes any required
post-config actions. These actions are determined by taking the
JSON difference between the current and new node configurations, then applying
all actions defined in the local metadata file.
Note that much of the functionality described above is implemented within helper classes:
ConfigHelper(e2e-controller) manages reading, modifying, and writing to all separate configuration layers.
NodeConfigWrapper(e2e-common) manages reading and writing the full node configuration.
ConfigMetadata(e2e-common) manages reading the config metadata file, validating configuration, and extracting post-config actions.
ConfigUtil(e2e-common) contains miscellaneous static utilities, such as version string parsing.
To keep config in sync, nodes send an MD5 hash of their local configuration to
the controller in their periodic status reports (
StatusApp. The controller's
ConfigApp periodically iterates over all
received status reports and looks for mismatches with its computed
configuration. For each mismatch, the controller will push a new node
configuration to the minion, as described above. There is a fixed interval
between consecutive pushes to the same node, as a protective measure in case the
new configuration causes a fatal error on the node; this delay is reset when the
controller receives a configuration change.
The automatic config sync can be disabled by "un-managing" the network or
specific nodes via a special boolean configuration field
sysParams.managedConfig. This may be needed temporarily for testing purposes.
ConfigApp contains an optional scheduling algorithm, similar
to that of
UpgradeApp, to roll out configuration changes that affect multiple
nodes in batches (e.g. when changing network overrides). This protects against
network-wide service disruption and network isolation, but will delay the
propagation of configuration changes.
To enable this staging algorithm, set the
on the controller; by default, all config updates to nodes are sent
When the controller finds a node with an unrecognized hardware board ID and version, it will disable config updates for that node because unknown hardware-specific configs may exist.
Additionally, if the
unknown_hw_queries_enabled flag is set (default on), the
controller will send
GET_MINION_BASE_CONFIG queries to these nodes
(rate-limited per node and board ID). In response, each minion will return a
MINION_BASE_CONFIG message containing its latest base configuration and
metadata. The controller merges these minion structures into its existing config
data (non-persistently), and can afterwards re-enable config updates for all
nodes with the same hardware board ID. Any conflicting metadata entries are
dropped, and unknown post-config actions are parsed but ignored.
Layered Configuration Model
Terragraph follows a layered configuration model, with a node's "full" configuration computed as the union of all layers in the following order:
- Base configuration - The default configuration, which is tied to a specific software version and is included as part of the image. The controller finds the closest match for a node's software version string, and falls back to the latest if no match was found.
- Firmware-specific base configuration - The default configuration tied to a specific firmware version, which is also included as part of the image. Values are applied on top of the initial base configuration layer.
- Hardware-specific base configuration - The default configuration tied to a
specific hardware type, which is also included as part of the image. Each
hardware type supplies configuration that changes with software versions.
Values are applied on top of the firmware base configuration layer. The
mapping between hardware types and hardware board IDs is defined in
- Automated node overrides - Contains any config parameters for specific nodes that were automatically set by the E2E controller.
- Network overrides - Contains any config parameters that should be uniformly overridden across the entire network. This takes precedence over the base configuration and automatic overrides.
- Node overrides - Contains any config parameters that should be overridden only on specific nodes (e.g. PoP nodes). This takes precedence over the network overrides.
The E2E controller manages and stores the separate config layers. Terragraph nodes have no knowledge of these layers, except the base configuration on the image. The nodes will copy the latest base version (via natural sort order) if the configuration file on disk is missing or corrupt.
A separate config metadata file (
/etc/e2e_config/config_metadata.json) is used
for per-config documentation, verification, and actions. Every configuration
field has a corresponding entry in this file.
Whereas node configuration is a JSON-serialized Thrift structure, the metadata
is freeform JSON that is recursively parsed into C++ structures defined in
Each root metadata entry is parsed into the
CfgParamMetadata structure, which
CfgRecursiveParam base type. A root entry is identified by the
presence of its required properties ("desc", "type", "action"). Depending on the
associated configuration field's data type, the root structure may or must also
contain a type-specific sub-structure (e.g. required for maps and objects). The
table below shows the association between all possible data types (defined in
the Thrift enum
thrift::CfgParamType), the name of their type-specific field,
and the C++ structure for the field:
Note that these data types constitute all JSON types except arrays. Arrays are not supported in the configuration model due to ambiguity in layering. Also note that map keys must be strings, in accordance with JSON specifications.
A post-config action is associated with each metadata entry, determining what
the minion should do when the configuration field is changed. These are defined
in the Thrift enum
|Reboot the node|
|Restart openr and pop_config|
|Restart pop_config and fib_nss|
|Reload nameservers listed in /etc/resolv.conf|
|Restart NTP service with updated config|
|Rewrite rsyslog.conf and restart rsyslogd|
|Restart sshd with updated CA keys file|
|Restart kea (dhcp)|
|Sync link metrics with openr LinkMonitor|
|Sync keys with openr KvStore|
|Rewrite firewall(ip6tables) rules|
|Update MCS-based link metric config for openr|
|Update glog's VLOG level|
|Reload firmware, usually by restarting e2e_minion|
|Dynamically change this firmware parameter|
|Dynamically change this firmware parameter at a specific BWGD index, or reload firmware if not possible|
|Set firmware stats config|
|Dynamically change link airtime allocation|
|Restart UDP ping server|
|Restart SSH daemon|
|Reassign channels across topology|
|Restart NET-SNMP and TG snmp agent daemons|
|Restart WebUI (HTTP server)|
|Reload tunnel configuration|
|Run vpp_chaperone to re-apply VPP config and restart monitor services|
|Redo POP config if required or do nothing|
|Restart stats_agent and fluent-bit services|
It is possible that multiple actions will be triggered for a single parameter,
e.g. if an object property defines an action different from the root entry. The
object property action does not simply replace the root entry action. Note
SET_FW_PARAMS and related actions are a special case and will only take
effect when set on an object property (not on a root entry).
The code handling each action is located in the minion's
ConfigApp::performNodeActions() method. The minion is responsible for applying
actions in a sensible order, e.g. rebooting or restarting minion last.
Each root metadata entry must include these fields:
- desc: A string description of the field
- action: The post-config action (
- type: The data type of the field (
Type-specific constraints can be defined using the previously-listed sub-structures:
- allowedRanges: An array of allowed ranges, where each range is a two-element integer array [min, max] (inclusive)
- allowedValues: An array of allowed integer values, separate from allowedRanges (e.g. for special values like -1)
- allowedRanges: An array of allowed ranges, where each range is a two-element floating-point array [min, max] (inclusive)
- allowedValues: An array of allowed floating-point values, separate from allowedRanges
- regexMatches: A regular expression string which the value must match
- intRanges: An array of allowed integer ranges (like above); the string value is parsed as an integer
- floatRanges: An array of allowed floating-point ranges (like above); the string value is parsed as a floating-point number
- allowedValues: An array of allowed string values, separate from regexMatches
- properties: A map of string property names to objects with the
- desc: A string description of the object property
- action: An additional post-config action (
- required: Whether defining this property is required (default false)
- type: The data type of the property value (
- intVal, floatVal, strVal, boolVal, objVal, mapVal
- properties: A map of string property names to objects with the following fields:
- type: The data type of the map values (enum, see above)
- intVal, floatVal, strVal, boolVal, objVal, mapVal
A configuration field can be marked as "deprecated" by setting the optional
"deprecated": true on any root or object-property metadata entry.
Deprecated configuration fields will still be initially read from disk, but only
GET operations during runtime.
A configuration field can be marked as "read-only" by setting the optional
"readOnly": true on any root or object-property metadata entry.
Read-only configuration fields will still be initially read from disk, but only
GET operations during runtime.
The config metadata parser supports a simple form of macro expansion. Macro
expansion blocks are denoted with a "copy-block" marker (
which is a normal JSON property that takes as a string value the absolute
location of the target block to copy (with all properties delimited by a
period). The target JSON value will be copied into the source location (i.e.
where copy-block was used). Any other properties defined within the original
object with copy-block will override those same properties in the copied
In the example below, the "mcs" block under "radioParamsBase" will be copied into "linkParamsBase", but with the action "SET_FW_PARAMS" instead of "RESTART_MINION":
"desc": "Link parameters",
"desc": "Firmware parameters for links",
"desc": "Radio parameters",
"desc": "Firmware parameters for radios",
"desc": "MCS used by transmitter (1-12: Static MCS, 35: Joint LA-TPC)",
"allowedRanges": [[1, 12]],
The current implementation handles circular- and self-references (throw error), as well as dependencies (parse with no order requirement).
The controller and aggregator are configured using separate configuration files
with a similar overall design as the node configuration, and are managed by the
E2EConfigWrapper class. The configuration model does not have any layers (or
"base" configuration), but the user operations on these configurations are
otherwise the same as for nodes.
These E2E configuration files are shown in the table below.
|Thrift Structure||Configuration File||Metadata File|
The following actions are supported in the E2E configuration files:
|Send SIGTERM to the process|
|Update glog's VLOG level|
|Reload associated gflag value|
|Update scan configuration|
The E2E configuration structures contain a special "flags" map, which defines
command-line flags to be passed into the E2E controller and NMS aggregator on
startup. The systemd service scripts will pass the flags, and utilize the
config_print_flags script which reads a configuration file and returns the
command-line flags as a formatted string.
Certain metadata properties are specific to E2E configuration, and are described in the sections below.
Descriptive tags can be set on any root or object-property metadata entry via
"tag" string property. Tags are purely informational.
Controller configuration is partially synced between peers under High
Availability mode (refer to High Availability for
further details). Configuration fields must be marked "syncable" by setting the
"sync": true on any root or object-property metadata entry.
In general, only network-level configuration is synced, and host-level
configuration is not synced.
Software Version Strings
Selection of the correct base configuration and hardware-specific base
configuration is determined by each node's software version string, contained
in the file
Optionally, "major" and "minor" version numbers are extracted from version strings that contain one of the following substrings:
[...] RELEASE_M<major> [...]
[...] RELEASE_M<major>_<minor> [...]
For example, the string below represents major version 60, minor version 7:
Facebook Terragraph Release RELEASE_M60_7-0-g5bec98eb9 michaelcallahan@devvm1112 2020-12-04T15:49:48
If major and minor versions are present, they are used to compare different
software versions. Otherwise, string-type comparisons are done on the full
version string. Order is important when determining the latest known
configuration, for example during node startup where the initial node
configuration file is generated if absent (via
There is an assumption that the "latest" base configuration file present on a
node's filesystem is the one it should use locally.
Adding New Configuration
The sections below describe the steps required to write new configuration.
Hardware-specific configuration is applied based on a hardware board ID, hardware type, and software version.
- The hardware board ID is defined by the value of
HW_BOARD_IDin the node info file (
/var/run/node_info). This file is generated by a startup script
gen_node_info_file.shwhich must be modified for new hardware platforms. See Service Scripts for more details.
- The hardware type is defined in
hw_types.json(described above), which is a mapping between board IDs and subdirectory names (within
./base_versions/hw_versions/). One hardware type can represent multiple board IDs (e.g. for related platforms with identical configuration).
- The software version is used to match both the base configuration and
hardware-specific configuration. If any hardware-specific configuration should
be applied, then a file should exist for each supported software version in
For interoperability, vendors must define a new, unique hardware board ID for each product.
New Configuration Fields
All configuration fields must have the following parts:
- Definition - Within the appropriate Thrift structure (i.e.
AggregatorConfig), as described above.
- Metadata - Within the corresponding metadata file (i.e.
- Default value - Within the base configuration JSON file. For hardware-specific fields, the default value should be placed in the hardware-specific base configuration file instead; in these cases, an empty value is usually added to the base configuration as well (for visibility).
For interoperability, vendors should follow these guidelines:
- Choose obviously non-conflicting key names for custom fields. For example, include the company name in every field name (for top-level structures).
- Pick indices for new Thrift fields within unique ranges to avoid serialization and/or merge conflicts. For example, start at 20000 (random number) instead of the next consecutive integer.
New Command-Line Flags
When adding new command-line flags to the E2E controller or NMS aggregator, it is recommended (but not required) to add an entry to corresponding metadata file. This provides better visibility (e.g. in the NMS), enables input validation, and allows post-config actions to be applied.
Flags for other services (e.g. E2E minion) are not supported here; consider using explicit configuration fields instead.
For configuration changes which are not backwards compatible (e.g.
moving/renaming fields), it is recommended to support automatic migration
between the old and new configuration versions to avoid the need for user
intervention. This migration procedure is implemented in the script
src/terragraph-e2e/lua/migrate_e2e_data.lua, which should be run automatically
during software upgrades (e.g. when using the NMS installer) and will modify
data files as needed. Follow instructions in this script to add additional
- folly::dynamic - Meta's C++ dynamically-typed value abstraction