Christoph Molnar - Introduction To Conformal Prediction With Python : A Short Guide For Quantifying Uncertainty Of Machine Learning Models

1. 11.4 What happens if I choose a large \alpha for conformal classification?

A Short Guide for Quantifying Uncertainty of Machine Learning Models

Christoph Molnar

A prerequisite for trust in machine learning is uncertainty quantification. Without it, an accurate prediction and a wild guess look the same.

Yet many machine learning models come without uncertainty quantification. And while there are many approaches to uncertainty from Bayesian posteriors to bootstrapping we have no guarantees that these approaches will perform well on new data.

At first glance conformal prediction seems like yet another contender. But conformal prediction can work in combination with any other uncertainty approach and has many advantages that make it stand out:

• Guaranteed coverage: Prediction regions generated by conformal prediction come with coverage guarantees of the true outcome
• Easy to use: Conformal prediction approaches can be implemented from scratch with just a few lines of code
• Model-agnostic: Conformal prediction works with any machine learning model
• Distribution-free: Conformal prediction makes no distributional assumptions
• No retraining required: Conformal prediction can be used without retraining the model
• Broad application: conformal prediction works for classification, regression, time series forecasting, and many other tasks

Sound good? Then this is the right book for you to learn about this versatile, easy-to-use yet powerful tool for taming the uncertainty of your models.

This book:

• Teaches the intuition behind conformal prediction
• Demonstrates how conformal prediction works for classification and regression
• Shows how to apply conformal prediction using Python
• Enables you to quickly learn new conformal algorithms

With the knowledge in this book, youll be ready to quantify the uncertainty of any model.

My first encounter with conformal prediction was years ago, when I read a paper on feature importance. I wasnt looking for uncertainty quantification. Nevertheless, I tried to understand conformal prediction but was quickly discouraged because I didnt immediately understand the concept. I moved on.

About 4 years later, conformal prediction kept popping up on my Twitter and elsewhere. I tried to ignore it, mostly successfully, but at some point I became interested in understanding what conformal prediction was. So I dug deeper and found a method that I actually find intuitive.

My favorite way to learn is to teach, so I decided to do a deep dive in the form of an email course. For 5 weeks, my newsletter Mindful Modeler became a classroom for conformal prediction. I didnt know how this experiment would turn out. But it quickly became clear that many people were eager to learn about conformal prediction. The course was a success. So I decided to build on that and turn everything I learned about conformal prediction into a book. You hold the results in your hand (or in your RAM).

I love turning academic knowledge into practical advice. Conformal prediction is in a sweet spot: Theres an explosion of academic interest and conformal prediction holds great promise for practical data science. The math behind conformal prediction isnt easy. Thats one reason why I gave it a pass for a few years. But it was a pleasant surprise to find that from an application perspective, conformal prediction is simple. Solid theory, easy to use, broad applicability conformal prediction is ready. But it still lives mostly in the academic sphere.

With this book, I hope to strengthen the knowledge transfer from academia to practice and bring conformal prediction to the streets.

This book is for data scientists, statisticians, machine learners and all other modelers who want to learn how to quantify uncertainty with conformal prediction. Even if you already use uncertainty quantification in one way or another, conformal prediction is a valuable addition to your toolbox.

Prerequisites:

• You should know the basics of machine learning
• Practical experience with modeling is helpful
• If you want to follow the code examples, you should know the basics of Python or at least another programming language
• This includes knowing how to install Python and Python libraries

The book is not an academic introduction to the topic, but a very practical one. So instead of lots of theory and math, there will be intuitive explanations and hands-on examples.

In this chapter, youll learn

• Why and when we need uncertainty quantification
• What conformal prediction is

Machine learning models make predictions and to fully trust them, we need to know how certain those predictions really are.

Uncertainty quantification is essential in many situations:

• When we use model predictions to make decisions
• When we want to design robust systems that can handle unexpected situations
• When we have automated a task with machine learning and need an indicator of when to intervene
• When we want to communicate the uncertainty associated with our predictions to stakeholders

The importance of quantifying uncertainty depends on the application for which machine learning is being used. Here are some use cases:

• Uncertainty quantification can improve fraud detection in insurance claims by providing context to case workers evaluating potentially fraudulent claims. This is especially important when a machine learning model used to detect fraud is uncertain in its predictions. In such cases, the case workers can use the uncertainty estimates to prioritize their review of the claim and intervene if necessary.
• Uncertainty quantification can be used to improve the user experience in a banking app. While the classification of financial transactions into rent, groceries, and so on can be largely automated through machine learning, there will always be transactions that are difficult to classify. Uncertainty quantification can identify tricky transactions and prompt the user to classify them.
• Demand forecasting using machine learning can be improved by using uncertainty quantification, which can provide additional context on the confidence in the prediction. This is especially important in situations where the demand must meet a certain threshold in order to justify production. By understanding the uncertainty of the forecast, an organization can make more informed decisions about whether to proceed with production.

But where does this uncertainty come from?

A prediction is the result of measuring and collecting data, cleaning the data, and training a model. Uncertainty can creep into the pipeline at every step of this long journey:

• The model is trained on a random sample of data, making the model itself a random variable. If you were to train the model on a different sample from the same distribution, you would get a slightly different model.
• Some models are even trained in a non-deterministic way. Think of random weight initialization in neural networks or sampling mechanisms in random forests. If you train a model with non-deterministic training twice on the same data, you will get slightly different models.