One of our customers operates a large computer network. In order to maintain its healthy infrastructure, it is necessary to properly monitor all network activity, analyze traffic flows, and predict eventual problems.
However, even having been condensed by standardized statistical techniques, the flow of the data is still too big and raw to store and analyze without preliminary processing.
We were assigned the task to design a scalable distributed system for aggregating, enriching, and analysing large streams of network data.