Data Analysis in Terabit Ethernet Traffic

Quite every laptop nowadays is capable of handling Gigabit traffic. But doing a network analysis will hit the boundaries of CPU load quite quickly. Now, with 10GbE lines as the usual speed of carrier's and company's backbone, traffic monitoring and analysis became more and more painful. Even the biggest and most expensive analysis appliances on the market are barely capable of a real time traffic monitoring for more than 8Gbit/s.

That's were a couple of vendors showed up and created devices which can handle multiple 10GbE lines at the same time. They call them “Active Distributed Traffic Capture Systems” or “Intelligent Data Access Networking Switches” – in short “Data Access Systems”. The primary use is for the aggregation and distribution of traffic. But all of the Data Access Systems are also capable of filtering traffic with the help of FPGA or CPLD techniques. So a carrier, bank or lawful interception organization can aggregate the data from many physical lines into one Data Access System, enter some filters with the help of a browser GUI, and distribute the resulting traffic to the analysis machines. It's easy to monitor 100 lines of 10GbE traffic.

For competitive reasons, those vendors started to invent new features for a better or easier analysis of the data on the analysis devices. These include ingress port tagging, time stamping with nanosecond accuracy, slicing of packets and recalculation of checksums in realtime, blanking bits in packets, or even layer 7 filtering for e-mail and instant messenger addresses with full flow capturing.

The interesting part for the usage is to create an infrastructure where even without data retention and a long term analysis specific users or just their communication with possible ”interesting“ data for intelligence agencies can be triggered and captured in real time. So, the process of the analysis can be quickened to quite no time. It's safe to say, that the flagship appliance by a vendor has been designed by request of US intelligence agencies.

Of course, those devices have to be managed by administrators. For the ease of usage every vendor moved from a CLI based configuration interface to a shiny web GUI – with a couple of flaws. It is easy to break into the system or read out the configuration without access.

This lecture will discuss the possibilities of today's data analysis with the help of these Data Access Systems. An overview of the features will help to understand that data analysis devices are not anymore the limiting factor in deep packet inspection of a huge amount of traffic. Examples will show what already has been set up and what is possible by companies and organizations – and which traffic they might monitor yet.

During the last three years the speaker installed those appliances from different vendors at customers across Europe, gained deep knowledge of their usage, established a strong contact to the technicians and chief officers both at the vendors and customers side, and found out a lot about the hardware and software by reverse engineering.