Network Flow: A look into the meta-information
Most network monitoring tools already provide many interesting insights into a network. How extensive this is depends primarily on the information that the respective software receives from the various network components. Since most monitoring solutions work with SNMP, they depend on the data provided by the SNMP agent. This provides – if the protocol is implemented correctly – a very good insight into the network utilisation, e.g. the bandwidth on the different network interfaces, the status and/or the utilisation of the devices in the system.
Administrators who require a more in-depth analysis of their network should therefore use monitoring software that offers more advanced functions, such as examining and monitoring of the network flow. Network flow is meta-information derived from the data traffic. A flow data record thus provides information about who is talking to whom in the network, and over which protocols. It provides information about who is using which port, and how much bandwidth each application is using on each port. It thus enables more thorough root cause analysis, helping to identify bottlenecks in the network more quickly.
Network flow also allows packet inspection. The network flow data sets are very tightly consolidated, but such packet insights require access to the raw packets – which can only be obtained via a mirror port or a Network Tap. However, these can usually also be connected as a data source for a network flow analysis.
Advantages of a network flow analysis
Get a deep insight into your network traffic
Network flow is meta-information derived from the data traffic. Monitoring the flow data enables the administrator to know who is doing what, when and where in the network. So it is possible to detect possible bottlenecks. It also helps to recognize anomalies in the network.
Know when something is wrong
With the integration it is possible to get an overview of all alerts from ntop, such as when the DNS is misused, invalid certificates are used, or encryption via TLS does not work correctly.
Identify the top talkers and top listeners
The inspection of network flow can show the top talkers and top Listeners in the network, as well as breaking down network usage by source, destination address, protocol or application.
Get detailed information of your devices
Network flow data also provides more detailed information on network's hosts such as traffic, packets, ports, peer or applications.
Top talker and top listener in your network
With the help of the network flow it is possible to obtain detailed user data from the used switches and routers – provided these support a network flow protocol. In this way the monitoring solution in operation can list, for example, top talkers and top listeners in the network, as well as breaking down network usage by source, destination address, protocol or application. In addition the administrator also receives further information on what is happening in the network.
The flow data is provided via various protocols, such as NetFlow or sFlow. Cisco originally developed NetFlow with the intention of simplifying the creation of access control lists. However, the data collected has proven to be very useful for additional analysis, so the network manufacturer has developed the protocol further.
Although other analysis techniques from other network manufacturers are now available, such as J-flow from Juniper Networks, NetStream from Huawei or Rflow from Ericsson, Cisco's technique is the most common. With IPFIX, the IETF (Internet Engineering Task Force) has also introduced an industry standard for the export of flow data records based on NetFlow version 9. Many producers now support this protocol with their devices.
With flow examination, the administrator is able to know who is doing what, when and where in the network, and how the data traffic flows in their infrastructure. It also enables the administrator to detect any anomalies that may indicate malware or other undesirable network events.
The data also helps with capacity planning in the network. Finally, an inspection of network flow can be used to determine how much network bandwidth an application requires. On the one hand this helps to assess the impact of certain applications on the network. For example, peaks can be quickly identified and connection bottlenecks in the network optimised.
On the other hand, the information is also useful for the implementation of a Quality of Service (QoS) in a network environment. QoS concerns the prioritisation of data traffic. This means that the data packets of a business-critical application which depends on low latency – for instance VoIP (Voice over IP) – are given ‘priority’ over data packets from a less latent application, such as emails.
How network flow works
A flow packet contains various types of meta-information, such as the IP address of the sender and the receiver, the source and destination ports, Layer 3 protocol types, the classification of the service, and the router or switch interface. The protocol groups all packets with the same contents into a flow and then summarises the packets and bytes together at the end.
Network components that support a network flow protocol create and send such a flow packet via a push method. A collector then gathers these data records. Depending on which protocols it supports, this collector converts these data into standardised values, thus preparing them for examination. The collector then forwards the data to a central instance, which then displays the processed data graphically, making it available in the form of a GUI for deeper insights.
Network flow monitoring with Checkmk
With Checkmk 2.0 the monitoring solution is extended by the possibility of network flow monitoring. For this purpose tribe29, the company behind Checkmk, integrates the ntop network flow monitoring solution from the company of the same name into tribe29’s own software. With the aim of providing a uniform look and feel, from Checkmk 2.0 onwards the network flow data from the ntop servers should be easily accessible via Checkmk monitoring – without the user having to leave the software interface. Watch our presentation about the ntop integration in Checkmk.
Netflow monitoring with ntop
The ntop architecture
The network flow monitoring of ntop is based on two components: the collector called nprobe, and the ntopng analysis and visualisation console.
In nprobe, traffic data is collected from switches and routers on the network by collecting their NetFlow, sFlow or IPFIX records. The collector then prepares this data for analysis and sends it to ntopng, which provides a web-based, graphical overview of the data.
By visualising the data, a user can gain the insight into his network infrastructure that they need. In addition to flow data, nprobe can also be fed with ‘raw data’, i.e. data packets not aggregated into a flow from a mirror port or network tap, to perform a deep packet inspection, for example.
Visualisation of the network traffic
ntop provides the network administrator with insights into their own network that go far beyond the information obtained via SNMP. This includes:
- Network flow analysis, such as top talkers, etc.
- Deep performance monitoring, such as delay, round-trip times, etc.
- Threat detection support, by quickly detecting threads such as DDoS attacks.
With ntop it is possible to view the top talkers of a port, check where the data is going, who or what is using the most bandwidth of a port, and which is the most frequent destination address of a port. The administrator can also view the data traffic in real time and, for example, examine the most active interfaces and top applications. Furthermore, it is also possible to analyse historical traffic data, for example to identify anomalies or trends.
The network flow monitoring in ntop also provides administrators with more detailed information on the network’s hosts. Various filters in the dashboard allow you to view numerous details for a host, such as traffic, packets, ports, peers or applications. In this way it is possible, for example, to use the host information provided by Checkmk to enrich additional details from ntop’s flow monitoring.
For this purpose, each host overview in Checkmk will contain an ntopng icon for viewing the additional flow data from ntop. This is made possible by the hardware and software inventory in Checkmk, which can search for interfaces such as ntop’s host view.
As already mentioned, ntop also provides the function of a own Deep Package Inspection (nDPI). Deep Package Inspection allows ntop to break down which application protocols are used by a specific IP address over a specific port. In this way it is possible to trace which applications a user is using, such as Microsoft Office 365, Citrix, SSH or Checkmk. According to its own description, ntop supports over 250 different application protocols. Since more and more network traffic is also encrypted via SSL, nDPI can support encrypted connections and check for their encryption certificate using a decoder for SSL certificates from clients and servers. In this way, application protocols for Citrix Online or Apple iCloud can also be identified which otherwise remain undiscovered in encrypted data traffic.
Alerts: Detecting when something is wrong
With network flow monitoring it is of course not only possible to analyse network traffic. It is also possible to get an overview of all alerts in ntop. The alert dashboard in Checkmk provides a table of active alerts, such as when thresholds have been exceeded. Another table provides an overview of past alerts that are no longer active (past alerts). The third available table of the dashboard lists the flow alerts. In this category, ntop reports anomalous or suspicious data flows. These alerts do not appear in the active or past alerts table. In this way, the administrator gets an overview of all the problem areas in his network.