Ted Dunning - Practical machine learning: a new look at anomaly detection

Beijing Cambridge Farnham Kln Sebastopol Tokyo

Everyone loves a mystery, and at the heart of it, thats what anomaly detection isspotting the unusual, catching the fraud, discovering the strange activity. Anomaly detection has a wide range of useful applications, from banking security to natural sciences to medicine to marketing. Anomaly detection carried out by a machine-learning program is actually a form of artificial intelligence. With the ever-increasing volume of data and the new types of data, such as sensor data from an increasingly large variety of objects that needs to be considered, its no surprise that there also is a growing interest in being able to handle more decisions automatically via machine-learning applications. But in the case of anomaly detection, at least some of the appeal is the excitement of the chase itself.

Figure 1-1. Finding anomalies is the detective work of machine learning.

When are anomaly-detection methods a good choice? Unlike fictional detective stories, in anomaly detection, you may not have a clear suspect to search for, and you may not even know what the crime is. In fact, one way to think about when to turn to anomaly detection is this: Anomaly detection is about finding what you dont know to look for .

You are searching for anomalies, but you dont know what their characteristics will be. If you did, you could use a different form of machine learning, called classification, or you would just write specific rules to find the anomalies. But thats not generally where you start.

Classification is a form of supervised learning where you have examples of each kind of thing you are looking for. You apply a learning algorithm to these examples to build a model that can use features of new data to classify them into categories that represent each kind of data of interest. When you have examples of normal and some number of abnormal situations, classifers can help you mark new situations as normal or abnormal. Even when you know about some kinds of anomalies, it is always good to keep an eye out for new kinds that you dont know about. That is where anomaly detection is applied.

So you use the unsupervised-learning approach of anomaly detection when you dont know exactly what you are looking for. Anomaly detection is a discovery process to help you figure out what is going on and what you need to look for. The anomaly-detection program must discover interesting patterns or connections in the data itself, and the detector does this by first identifying the most important aspect of anomaly detection: finding what is normal . Once your model does that, your machine-learning program can then spot outliers, in other words, data that falls outside of what is normal.

Anomalies are defined not by their own characteristics, but in contrast to what is normal. You may not know what the anomalies will look like, but you can build a system to detect them in contrast to what youve discovered and defined as being a normal pattern. Note that normal in this context includes all of the anomalies that you already know about and have accounted for using a classifier. The outliers are only those events that dont match what you already know. Consider this way to think about the problem: anomaly in this context just means different than expectedit does not refer to desirable or undesirable. You may know of certain types of events that are somewhat unusual and require attention, perhaps certain failures in a system. If these occur sufficiently often to be well characterized, you can use a classifier to catalog them as problems of a particular type. Thats a somewhat different goal than true anomaly detection where you are looking for events that are rare relative to what is expected and that often are surprising, or at least undefined ahead of time.

Together, anomaly detection and classification make for a useful pair when it comes to finding a solution to real-world problems. Anomaly detection is used firstin a discovery phaseto help you figure out what is going on and what you need to look for. You could use the anomaly-detection model to spot outliers, then set up an efficient classification model to assign new examples to the categories youve already identified. You then update the anomaly detector to consider these new examples as normal and repeat the process. This idea is shown in as one way to use anomaly detection.

Figure 1-2. Use anomaly detection when you dont know what to look for. Sometimes this discovery process makes a useful preliminary stage to define the categories of interest for a classifier.

Anomaly detection, like classification, is not new, but recently there has been an increased interest in using it. Fortunately, there also are new approaches to carrying it out effectively in practical settings; much more accurate and sophisticated methods are now available. Some of the biggest changes have to do with being able to handle anomaly detection at huge scale, in real time. We will describe some approaches that can help, especially when using a realtime distributed file system. We will focus particularly on approaches that have demonstrated, practical, and simple implementations.

The move from specialized academic research to methods that are useful for practical machine learning is happening in response to more than just an increase in the volume of available datathere is also a great increase in new types of data. For example, many new forms of sensors are being deployed. Smart meters monitor energy usage in businesses and residential settings, reporting back every few minutes. This information can be used individually or looked at as a group from a particular geographical location.

Figure 1-3. This wall of smart meters reports a granular view of energy usage for a utility company. Sensor data is becoming a huge source of valuable information that can be analyzed through machine learning techniques such as anomaly detection.

Industrial equipment such as drilling rigs and manufacturing tools use sensors to report on a wide range of parameters. The advances in medical device sensors are astounding. Radio-frequency identification (RFID) tags are also commonplace on merchandise in retail stores, in warehouses, or even on your cat. Data provided by these sensors and other sources range from simple identification signals to complex measurements of temperature, pressure, vibrations, and more.

How can reporting from all these interconnected objects be used? Collectively, these objects begin to make up the Internet of Things (IoT). Relationships between objects and people, between objects and other objects, conditions in the present, and histories of their condition over time can be monitored and stored for future analysis, but doing so is quite a challenge. However, the rewards are also potentially enormous. Thats where machine learning and anomaly detection can provide a huge benefit.