In talking with a customer recently about network management, an interesting question came up: "Is the network management industry headed toward instrumentation on the network devices and exporting the data to analysis systems?"
I've seen some early indications that this is starting to happen. For example, I'm aware of several products that use agents residing on network devices to gather and quickly send real-time data to an external analysis system. So the question stayed with me, and I began to wonder about analyzing such data to improve network monitoring and control. It finally sank into my head that the ideal terminology is "data driven network." While a quick Internet search turned up several papers and presentations on the topic, I found no products. I expect that to change.
The Basic Idea
The idea behind data-driven networking is that the analysis of large volumes of real-time network data can help optimize network forwarding mechanisms. Stated another way, data-driven networking is the application of big data analysis to raw network data, with the results then used to optimize network performance. Is the subject that new? Not really. Most of the papers about data-driven networking published in 2016:
Network Instrumentation
The traditional approach to network monitoring and management, Simple Network Management Protocol (SNMP), uses a simple mechanism to collect data from network devices. Unfortunately, the simple mechanism makes the network device's SNMP data structures more complex. The combination of simple mechanism and complex data structures creates a rather slow system for collecting much information about a network device. It's too slow to use for real-time control of network forwarding and performance.
Newer application technology, known as the Advanced Message Queuing Protocol or message-oriented middleware (typified by RabbitMQ and ZeroMQ) implement a message bus in which data sources publish data streams and data consumers subscribe to data streams. These are often known as publish-subscribe, or Pub-Sub, systems. The MapReduce and Hadoop systems also provide mechanisms for processing large volumes of data, often referenced as big data. Technologies like these create a new data processing model, and we're starting to see network analysis products use these systems.
One of the first systems to use data collection agents and big data technology is Cisco's Tetration Analytics. It uses collectors in certain Cisco products and end systems (i.e., servers) to send information about all network traffic to a central processing system. Once in the Tetration Analytics system, the data is analyzed for a variety of purposes, including security, network anomalies, and application performance. The system collects all packets, so it's more thorough than sampled flow data. The volume of data is large (1-2% of the full network traffic) so the system uses big data techniques for its analysis. (For details on Tetration, Cisco provided the Network Field Day delegates with a presentation at Cisco Live Europe 2017 and an update on Tetration at NFD16.)
Continuing that trend are Arista Networks, Apstra, and Veriflow Systems. (Each of these companies also presented at NFD16.) While these companies use different mechanisms, they all rely on remote data collectors to collect a richer set of data than is possible using SNMP. Each company's product collects different details and performs different analysis. In my opinion, the common theme is that they're all implementing some form of a data-driven network. The richness of the collected data, coupled with innovative analysis that provides greater visibility and control of the network, is very interesting. SNMP doesn't seem as interesting any more.
Does a data-driven network imply that SNMP is about to be replaced? I don't think so. There is still room for basic network visibility and alerting. The above products are doing much more than the basics. Replicating what can be done with existing products that use SNMP doesn't make sense for them.
The combination of basic network management (which must be properly configured) and more advanced analysis is an exciting prospect, especially since most of these products can work with existing networks. Depending on the product, you may not need to change topologies or wait for a hardware refresh to take advantage of a data-driven network.
That's very interesting.
