Florian Skopik - Smart Log Data Analytics: Techniques for Advanced Security Analysis

This Springer imprint is published by the registered company Springer Nature Switzerland AG

The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Prudent event monitoring and logging are the only means that allow system operators and security teams to truly understand how complex systems are utilized. Log data are essential to detect intrusion attempts in real time or forensically work through previous incidents to create a vital understanding of what has happened in the past.

Today, almost every organization already logs data to some extent, and although it means a considerable effort to establish a secure and robust logging infrastructure as well as the governing management policies and processes, basic and raw logging is comparatively simple in contrast to log analysis. The latter is an art of its own, which not many organizations know how to master. Log data are extremely diverse and processing them is unfortunately quite complex. There is no standard that dictates the granularity, structure, and level of details that log events provide. There is no agreement what logs comprise and how they are formatted.

Facing these facts, it is astonishing that not much literature that concerns logging in computer networks exists. And although there are at least some great books out there, it is not enough. On the one side some existing literature did not age well (certain topics are simply outdated after several years as technologies evolve and newer concepts such as bring your own device, cloud computing, and IoT hit the market), and on the other side some relevant topics are simply not sufficiently covered yet, especially when it comes to complexand sometimes drylog data analytics.

We take Dr. Chuvakins (et al.) book Logging and Log Management from 2013 as a starting point. This is a great book that covers all the essential basics from a technical and management point of view, such as what log data actually are, how to collect log data, and how to perform simple analysis, and also explains filtering, normalization, and correlation as well as reporting of findings. It further elaborates on available tools and helps the practitioner to adopt state-of-the-art logging technologies quickly. However, while it provides a profound and important basis for everyone who is in charge of setting up a logging infrastructure, this book does not go far enough for certain audiences. The authors essentially stop there, where our book starts. We assume, the reader of our book knows the basics and has already collected experience with logging technologies. We further assume, the reader spent some serious thoughts on what to log, how to log and why to logand that common challenges regarding the collection of log data have been solved, including time synchronization, access control for log agents, log buffering/rotation, and consistency assurance. For all these topics, technical (and vendor-specific) documentation exists.

We pick up the reader at this point, where they ask the question what to do with the collected logs beyond simple outlier detection and static rule-based evaluations. Here, we enter new territory and provide insights into latest research results and promising approaches. We provide an outlook on what kind of log analysis is actually possible with the appropriate algorithms and provide the accompanying open-source software solution AMiner to try out cutting-edge research methods from this book on own data!

In the last decade, numerous people supported this project. We would like to specifically thank Roman Fiedler as one of the founders of the AMiner project, Wolfgang Hotwagner for the invaluable infrastructure and implementation support, Georg Hld for his contributions to the advanced detectors, and Ernst Leierzopf for software quality improvements.

This work has been financially supported by the Austrian Research Promotion Agency FFG and the European Unions FP7 and H2020 programs in course of several research projects from 2011 to 2021.