MLB uses NetBox automation for network configuration backup
Major League Baseball used NetBox in tandem with its network automation project to streamline its configuration backup process and keep its infrastructure systems in sync.
When Major League Baseball runs its nightly configuration backup for the 1,800 network devices throughout its infrastructure, it needs to ensure it collects accurate and current data from various systems.
To implement data consistency between its infrastructure systems, the MLB network engineering team launched an initiative to improve its network reliability engineering. This meant better integration among its monitoring system, help desk, server system, discovery and assessment system, GitHub repository and more, said Jeremy Schulman, senior network automation engineer at MLB, during a recent virtual conference hosted by NetBox, an open source project that provides a network management web application. To get the best results, the MLB team needed to build more network automation into its infrastructure.
"For me, network automation is about the integration of systems together to create reliable outcomes," Schulman said. "If we build systems that aren't reliable, then people don't trust them. And, if people don't trust them, they don't want to use them."
In order for MLB's disparate systems to exchange data, the systems needed to be in sync with each other, Schulman said. This meant the team needed an inventory source of truth -- a record that would collect and track data across different infrastructure systems, storing information about device statuses, configurations and maintenance. This source of truth would enable network automation workflows, feeding scripts the data they need to run. That's when MLB turned to NetBox.
NetBox as a source of truth
NetBox would conduct two-way communication with the different systems in MLB's environment, Schulman said, and automatically update its inventory information using data from those systems. For example, NetBox could communicate with MLB's Simple Network Management Protocol monitoring system to collect and track device data.
If we build systems that aren't reliable, then people don't trust them. And, if people don't trust them, they don't want to use them.
Jeremy SchulmanSenior network automation engineer, MLB
The team's first use case for NetBox automation was to integrate it with MLB's ChatOps systems, which developers use to communicate with applications, projects and infrastructure via a central collaboration platform. The team set up NetBox as the inventory system for all of MLB's ballpark infrastructure, Schulman said.
"When our ChatOps system presented a series of dropdowns and widgets about which devices and interfaces to select from, it pulled them from NetBox," he said.
NetBox for configuration backups
Eventually, the MLB team extended its NetBox use case to automate its nightly configuration backups. NetBox detects and indicates which network devices are active and which ones are offline or decommissioned, separating the devices that should be backed up from those that should be left alone. One way in which NetBox identifies those devices is with tagging.
For example, NetBox compiles information from the entire MLB infrastructure, so its inventory includes non-network devices that the team might not want to back up, such as console servers, Schulman said. In response, the team tags those devices as no-backup.
MLB uses NetBox tagging to specify which devices shouldn't be backed up.
Schulman also created custom dynamic inventory scripts that pull information from NetBox and other MLB systems. The scripts communicate with NetBox to ask for device configuration information, while they gather broader information about the networking environment. Schulman said he originally set up the dynamic inventory script -- versus a more generally available plugin -- because MLB network engineers used Ansible Tower to run workflows and playbooks. As a result, the NetBox inventory needed to integrate into Ansible Tower so the network engineers could access it.
"If somebody wanted to run or write their own playbooks and not run them through Tower, they're going to be using the same exact code in both environments, and having that consistency is important to make sure the same set of features is available for inventory," he said.
The backup process
The MLB team runs Ansible playbooks to perform the automated configuration backups. The backup process uses an authorized jump server -- a server used to access devices in separate security zones -- with a cron job that is scheduled to run a Bash script, which then executes a series of playbooks, Schulman said. For 1,800 devices, the process can take up to an hour, he added.
An example of MLB's cron job script
MLB has a GitHub storage repository it uses to store current network configurations and backups. NetBox also enables features -- such as custom links -- that let teams more easily integrate into GitHub to better track and access network configurations. For example, Schulman inserted a custom link to MLB's GitHub environment, where users can go directly to device configurations and "reduce the friction of going between one system to another," he said.
