Christoph Molnar - Modeling Mindsets: The Many Cultures Of Learning From Data

The Many Cultures of Learning From Data

Christoph Molnar

Is a linear regression model statistical modeling or machine learning? This is a question that I have heard more than once, but it starts from the false premise that models somehow belong to one approach or the other. Rather, we should ask: Whats the mindset of the modeler?

Differences in mindsets can be subtle yet substantial. It takes years to absorb a modeling mindset because, usually, the focus is on the methods and math rather than the mindset. Sometimes modelers arent even aware of their modeling limitations due to their mindset. I studied statistics, which gave me an almost pure frequentist mindset. Because of that narrow lens on modeling, I often hit a wall in my modeling projects, ranging from a lack of causal reasoning to crappy predictive models. I couldnt solve these problems by doubling down on my current mindset. Instead, I made the most progress when I embraced new modeling mindsets. Modeling Mindsets is the book I wish I had read earlier to save myself time and headaches.

The inspiration for Modeling Mindsets was the article Statistical Modeling: The Two Cultures by the statistician Leo Breiman. His article was the first to show me that modeling is not only about math and methods but its about the mindset through which you see the world. Leo Breimans paper is over 21 years old and hasnt lost any of its relevance even today. Modeling Mindsets builds on the same principle of making mindsets explicit and embraces the multiplicity of mindsets. I kept Modeling Mindsets short, (mostly) math-free and poured over a decade of modeling experiences into it. My hope is that this book will still be relevant 21 years from now and a great investment for you.

This book is for everyone who builds models from data: data scientists, statisticians, machine learners, and quantitative researchers.

To get the most out of this book:

• You should already have experience with modeling and working with data.
• You should feel comfortable with at least one of the mindsets in this book.

Dont read this book if:

• You are completely new to working with data and models.
• You cling to the mindset you already know and arent open to other mindsets.

You will get the most out of Modeling Mindsets if you keep an open mind. You have to challenge the rigid assumptions of the mindset that feels natural to you.

Every day, people use data for prediction, automation, science, and making decisions. For example:

• Identifying patients prone to drug side effects.
• Finding out how climate change affects bee hives.
• Predicting which products will be out-of-stock.

Each data point has details to contribute:

• Patient with ID 124 got acne.
• Bee colony 27 shrank during the drought in 2018.
• On that one Tuesday, the flour was sold out.

Data alone arent enough to solve the above tasks. Data are noisy and high-dimensional most of the information will be irrelevant to the task. No matter how long a human analyzes the data, its difficult to gain insights just by sheer human willpower.

Therefore, people rely on models to interpret and use data. A model simplifies and represents an aspect of the world. Models learned from data the focus of this book glue together the raw data and the world. With a model, the modeler can make predictions, learn about the world, test theories, make decisions and communicate results to others.

There is no philosophical consensus on the definition of a model. For our purpose, well go with this definition: a mathematical model thatconsists of variables and functions.

Variables represent aspects of the data and the model:

• A numerical variable can contain the blood pressure of a patient.
• A 3-dimensional variable may represent the colors of a pixel.
• Variables can also represent abstract aspects like happiness.
• A cluster variable represents data grouping.

Variables have different names in different mindsets: Random variables, covariates, predictors, latent variables, features, targets, and outcomes. The names can reveal the role of a variable in a model: For example, the target or dependent variable is the variable the modeler wants to predict.

Functions relate the variables to each other (.

• A linear regression model expresses one variable as a weighted sum of the other variables.
• In deep Q-learning a form of reinforcement learning the value of an action (variable) is a function of the state of the environment (variables).
• A clustering model can be a function that takes, as input, the variables of the data point and returns its cluster (variable).
• The joint distribution of two variables describes the co-occurrence of certain variable values in terms of probability.
• A causal model represents the causal relationship between variables.

Functions range from simple, like Y=5X , to complex, like a deep neural network with millions of parameters.

FIGURE 3.1: A mathematical model sets variables (dots) into relation (lines) using parameterized functions.

So far, we have discussed variables and functions, the ingredients of a model, but not how they are connected to data. Modelers use data to find the best function to relate the variables. Depending on the mindset of the modeler, the process of adapting a models function is called estimation, training, fitting or learning. In this process, the model function is optimized using data:

• In a linear regression model, the coefficients (also called weights) are optimized to minimize the squared difference between the weighted sum and the target variable.
• K-means clustering cycles between assigning data points to cluster centers and optimizing the centers to minimize distances.
• A decision tree is grown by finding split points in variables using data.

3.2 Models Are Embedded In Mindsets

The interpretation and use of the model cant be derived from the model itself. Two mathematically identical models might be used in different ways by different modelers. The use of a model depends on the mindset.

A model is a mathematical construct that doesnt contain the purpose of the model. The purpose of the model how to use and interpret it depends on the modeling mindset. To derive knowledge about the world from the model, modelers need to make further assumptions.

Consider a linear regression model that predicts regional rice yield as a function of rainfall, temperature, and fertilizer use. Its a model, but interpretation needs a mindset:

• Can the modeler interpret the effect of fertilizer as causal to rice yield? Yes, if based on a causal model.
• Is the model good enough for predicting rice yields? Depends if the modeler had a supervised learning mindset and has evaluated the generalization error properly.
• Is the effect of fertilizer on yield significant? This requires a frequentist mindset.

3.3 A Mindset Is A Perspective Of The World

A modeling mindset provides the framework for modeling the world with data (Figure ). Modeling mindsets are like different lenses. All lenses show us the world, but with a different focus. Some lenses magnify things that are close, and other lenses detect things that are far away. Also, some glasses are tinted so you can see in bright environments.