E. T. Jaynes - Probability Theory. The Logic of Science

PROBABILITY THEORY

THE LOGIC OF SCIENCE

This book goes beyond the conventional mathematics of probability theory, viewing the subject in a wider context. Now results are discussed, along with the application of probability theory to a wide variety of problems in physics, mathematics, economics, chemistry and biology. It contains many exercises and problems, and is suitable for use as a textbook on graduate level courses involving data analysis.

The material is aimed at readers who are already familiar with applied mathematics at an advanced undergraduate level or higher. The book is not restricted to one particular discipline but rather will be of interest to scientists working in any area where inference from incomplete information is necessary.

Edwin T Jaynes obtained his Ph.D in physics from Princeton University in 1950. He then went to Stanford University, where he stayed until1960. He wrote extensively during this time and his papers reformulating statistical mechanics as a problem in inference were published while he was at Stanford.

In 1960 Jaynes moved to the University of Washington, St Louis where he became the Wayman Crow Professor of Physics. He retired in 1992, and died on April 30 1998.

PROBABILITY THEORY

THE LOGIC OF SCIENCE

E. T. Jaynes

edited by G. Larry Bretthorst

CAMBRIDGE UNIVERSITY PRESS
Cambridge, New York, Melbourne, Madrid, CapeTown, Singapore, So Paulo, Delhi, Mexico City

Cambridge University Press
The Edinburgh Building, Cambridge CB2 8RU, UK

Published in the United States of Americaby Cambridge University Press, New York

www.cambridge.org
Information on this title: www.cambridge.org

E. T. Jaynes 2003

This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press.

First published 2003
Reprinted with corrections 2004
10th printing 2012

Printed and bound by MPG Books Group, UK

A catalogue record for this publication is available from the British Library

Library of Congress Cataloguing in Publication data

Jaynes, E. T. (Edwin T.)
Probability theory: the logic of science / by E.T. Jaynes; edited by G. Larry Bretthorst.
p. cm.
Includes bibliographical references and index.
ISBN 0 521 59271 2
1. Probabilities. I. Bretthorst, G. Larry. II. Title.
QA273. J36 2003
519.2 dc21 2002071486

ISBN 978-0-521-59271-0 Hardback

Cambridge University Press has no responsibility for the persistence or accuracy of URLs for external or third-party internet websites referred to in this publication, and does not guarantee that any content on such websites is, or will remain, accurate or appropriate. Information regarding prices, travel timetables and other factual information given in this work are correct at the time of first printing but Cambridge Universtiy Press does not guarantee the accuracy of such information thereafter.

Dedicated to the memory of
Sir Harold Jeffreys, who
saw the truth and preserved it.

Editors foreword

E. T. Jaynes died April 30, 1998. Before his death he asked me to finish and publish his book on probability theory. I struggled with this for some time, because there is no doubt in my mind that Jaynes wanted this book finished. Unfortunately, most of the later chapters, Jaynes intended volume 2 on applications, were either missing or incomplete, and some of the early chapters also had missing pieces. I could have written these latter chapters and filled in the missing pieces, but if I did so, the work would no longer be Jaynes; rather, it would be a JaynesBretthorst hybrid with no way to tell which material came from which author. In the end, I decided the missing chapters would have to stay missing the work would remain Jaynes.

There were a number of missing pieces of varying length that Jaynes had marked by inserting the phrase MUCH MORE COMING. I could have left these comments in the text, but they were ugly and they made the book look very incomplete. Jaynes intended this book to serve as both a reference and a text book. Consequently, there are question boxes (Exercises) scattered throughout most chapters. In the end, I decided to replace the much more coming comments by introducing Editors Exercises. If you answer these questions, you will have filled in the missing material.

Jaynes wanted to include a series of computer programs that implemented some of the calculations in the book. I had originally intended to include these programs. But, as time went on, it became increasingly obvious that many of the programs were not available, and the ones that were were written in a particularly obscure form of basic (it was the programs that were obscure, not the BASIC). Consequently, I removed the references to these programs and, where necessary, inserted a few sentences to direct people to the necessary software tools to implement the calculations.

Numerous references were missing and had to be supplied. Usually the information available, a last name and date, was sufficient to find one or more probable references. When there were several good candidates, and I was unable to determine which Jaynes intended, I included multiple references and modified the citation. Sometimes the information was so vague that no good candidates were available. Fortunately, I was able to remove the citation with no detrimental effect. To enable readers to distinguish between cited works and other published sources, Jaynes original annotated bibliography has been split into two sections: a Reference list and a Bibliography.

Finally, while I am the most obvious person who has worked on getting this book into publication, I am not the only person to do so. Some of Jaynes closest friends have assisted me in completing this work. These include Tom Grandy, Ray Smith, Tom Loredo, Myron Tribus and John Skilling, and I would like to thank them for their assistance. I would also like to thank Joe Ackerman for allowing me to take the time necessary to get this work published.

G. Larry Bretthorst

Preface

The following material is addressed to readers who are already familiar with applied mathematics, at the advanced undergraduate level or preferably higher, and with some field, such as physics, chemistry, biology, geology, medicine, economics, sociology, engineering, operations research, etc., where inference is needed. A previous acquaintance with probability and statistics is not necessary; indeed, a certain amount of innocence in this area may be desirable, because there will be less to unlearn.

We are concerned with probability theory and all of its conventional mathematics, but now viewed in a wider context than that of the standard textbooks. Every chapter after the first has new (i.e. not previously published) results that we think will be found interesting and useful. Many of our applications lie outside the scope of conventional probability theory as currently taught. But we think that the results will speak for themselves, and that something like the theory expounded here will become the conventional probability theory of the future.

History

The present form of this work is the result of an evolutionary growth over many years. My interest in probability theory was stimulated first by reading the work of Harold Jeffreys (1939) and realizing that his viewpoint makes all the problems of theoretical physics appear in a very different light. But then, in quick succession, discovery of the work of R. T. Cox (1946), Shannon (1948) and Plya (1954) opened up new worlds of thought, whose exploration has occupied my mind for some 40 years. In this much larger and permanent world of rational thinking in general, the current problems of theoretical physics appeared as only details of temporary interest.