Nicolas Privault - Understanding Markov chains: Examples and Applications

Advisory Board

M. A. J. Chaplain, University of St. Andrews

A. MacIntyre, Queen Mary University of London

S. Scott, Kings College London

N. Snashall, University of Leicester

E. Sli, University of Oxford

M. R. Tehranchi, University of Cambridge

J. F. Toland, University of Bath

More information about this series at http://www.springer.com/series/3423

Additional material to this book can be downloaded from http://extras.springer.com .

This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd.

The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore

Stochastic and Markovian modeling are of importance to many areas of science including physics, biology, engineering, as well as economics, finance, and social sciences. This text is an undergraduate-level introduction to the Markovian modeling of time-dependent randomness in discrete and continuous time, mostly on discrete state spaces, with an emphasis on the understanding of concepts by examples and elementary derivations. This second edition includes a revision of the main course content of the first edition, with additional illustrations and applications. In particular, the exercise sections have been considerable expanded and now contain 138 exercises and 11 longer problems.

The book is mostly self-contained except for its main prerequisites, which consist in a knowledge of basic probabilistic concepts. This includes random variables, discrete distributions (essentially binomial, geometric, and Poisson), continuous distributions (Gaussian and gamma), and their probability density functions, expectation, independence, and conditional probabilities, some of which are recalled in the first chapter. Such basic topics can be regarded as belonging to the field of static probability, i.e., probability without time dependence, as opposed to the contents of this text which is dealing with random evolution over time.

Our treatment of time-dependent randomness revolves around the important technique of first-step analysis for random walks, branching processes, and more generally for Markov chains in discrete and continuous time, with application to the computation of ruin probabilities and mean hitting times. In addition to the treatment of Markov chains, a brief introduction to martingales is given in discrete time. This provides a different way to recover the computations of ruin probabilities and mean hitting times which have been presented in the Markovian framework. Spatial Poisson processes on abstract spaces are also considered without any time ordering.

There already exist many textbooks on stochastic processes and Markov chains, including [BN96, in75, Dur99, GS01, JS01, KT81, Med10, Nor98, Ros96, Ste01]. In comparison with the existing literature, which is sometimes dealing with structural properties of stochastic processes via a more compact and abstract treatment, the present book tends to emphasize elementary and explicit calculations instead of quicker arguments that may shorten the path to the solution, while being sometimes difficult to reproduce by undergraduate students.

Some of the exercises have been influenced by [in75, JS01, KT81, Med10, Ros96] and other references, while a number of them are original, and their solutions have been derived independently. The problems , which are longer than the exercises, are based on various topics of application. This second edition only contains the answers to selected exercises, and the remaining solutions can be downloaded in a solution manual available from the publishers Web site, together with Python and R codes. This text is also illustrated by 41 figures.

Some theorems whose proofs are technical, as in Chaps. , have been quoted from [BN96, KT81]. The contents of this book have benefited from numerous questions, comments, and suggestions from undergraduate students in Stochastic Processes at the Nanyang Technological University (NTU) in Singapore.

A stochastic process is a mathematical tool used for the modeling of time-dependent random phenomena. Here, the term stochastic means random and process refers to the time-evolving status of a given system. Stochastic processes have applications to multiple fields and can be useful anytime one recognizes the role of randomness and unpredictability of events that can occur at random times in, e.g., physical, biological, or financial system.

For example, in applications to physics one can mention phase transitions, atomic emission phenomena, etc. In biology, the time behavior of live beings is often subject to randomness, at least when the observer has only access to partial information. This latter point is of importance, as it links the notion of randomness to the concept of information: What appears random to an observer may not be random to another observer equipped with more information. Think, for example, of the observation of the apparent random behavior of cars turning at a crossroad versus the point of view of car drivers, each of whom are acting according to their own decisions. In finance, the importance of modeling time-dependent random phenomena is quite clear, as no one can make definite predictions for the future moves of risky assets. The concrete outcome of random modeling lies in the computation of