Using Automation to Verify Network State

A network source of truth (NSoT) is a record of the desired state of critical parts of the network that network engineers can compare with the operational state of the network. It is critical to not use the network itself to determine whether it is configured and operating as desired. For example, half of a redundant subsystem could fail and go unnoticed because the network and applications continue to function.

 

Organizations that have integrated their NSoT with network automation are more likely to have successful network automation experiences than those who haven’t (see this Cisco-sponsored Enterprise Management Associates).

 

A separate, automated system is necessary for collecting and analyzing network state information. The analysis compares the operational state with the desired state to verify network correctness. NSoT analysis requires automation; manual methods are subject to human error and ineffective at the required scale. You should note that network failure diagnostic processes will replicate the set of functions in an NSoT tool, so why not incorporate those processes into standard network operational practices?

 

What should you check? Start with the infrastructure that’s critical to the applications and processes that are core to your organization’s business. The system should verify the same data that you’d check to diagnose a failure. Are key devices and interfaces up and capable of passing traffic? Check key server connections and verify that redundant paths are functional. Are the routing and switching protocols functioning correctly and do they have the desired neighbors? The network connectivity could exist, but a configuration error could cause a routing neighbor change that deletes a desired redundant path.

 

This analysis is different than what we’ve typically done with network management systems (NMS) — i.e., collecting and analyzing device and interface statistics. You can think of network management as a micro view of network infrastructure, NSoT as an intermediate view, and digital experience monitoring (DEM) as a macro view. For example, the NMS isn’t likely to report the undesirable routing change mentioned above, and the DEM may not detect it, but the NSoT would identify it.

 

You can create an NSoT using manual processes, but the better way is to use automation to collect the raw data. Then use manual processes to identify the critical elements that need validation, and to verify that the detected relationships are correct. An automated mechanism could use NetConf, RESTConf, SNMP, and CLI collection methods.

 

NSoT systems may also use APIs to collect and correlate data from systems that are the source of record (SoR) for their respective data. For example, they might correlate data between an asset management system of device information and a circuit database containing device connection information. You would then need to augment and validate this information using manual processes as described above.

 

The process of validating the NSoT database requires knowledge of the network’s design and operation. You must be able to identify the critical devices, links, neighbors, and protocols. Using an automated system to help populate the NSoT reduces the time and effort that you have to spend in this validation stage.

 

You can find NSoT systems from a variety of companies as well as two open source projects. The exact functionality depends on the system, though most of them use a foundation of IP address allocation (NSOT, NetBox, FusionLayer). At this point these systems have limited extension into neighbor relationships and protocol validation. Companies like Network to Code provide consulting, training, and software development expertise to help organizations adopt NSoT systems and customize them for the organization’s needs.

 

The NSoT is a valuable addition to a network automation system for its use in performing pre-change, post-change, and periodic network validation.

 

Pre-change use verifies that the network is functioning as desired before implementing a change; this prevents deployment of a configuration or OS update change in a network that is not operating as desired.

 

Post-change mode validates the change to the desired network state is achieved after a change, allowing the change process to revert a failed implementation.

 

Periodic execution can detect other changes that cause the network to deviate from the desired state. These might be a link failure outside of configuration or OS changes.

 

Here are some simple examples that demonstrate the ways in which an NSoT can highlight potential network problems. If done correctly, the post-change validation can easily be incorporated into the periodic network validation process.

 

 

Network automation’s integration with network state validation is a two-way street. In one scenario automation supports the collection of network state data to initially populate the NSoT. In another use case, NSoT uses automation to collect network state data to validate against the stored state data. In the reverse use case, NSoT’s standardization across the network facilitates the use of automation, particularly as described in the EMA whitepaper mentioned above.

 

Whatever the use case, automation and the network source of truth support each other.

---

For more network insight from Terry, see him at Enterprise Connect 2021, Sept. 27-29, in Orlando, Fla. To save $200 off your registration, use the code NJAL200 at checkout.