Sergios Theodoridis - Machine Learning: A Bayesian and Optimization Perspective

Second edition

Sergios Theodoridis

Department of Informatics and Telecommunications, National and Kapodistrian University of Athens, Athens, Greece

Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen, China

1. Tables in Chapter 3
2. Tables in Chapter 6

1. Figures in Chapter 1
2. Figures in Chapter 2
3. Figures in Chapter 3
4. Figures in Chapter 4
5. Figures in Chapter 5
6. Figures in Chapter 6
7. Figures in Chapter 7
8. Figures in Chapter 8
9. Figures in Chapter 9
10. Figures in Chapter 10
11. Figures in Chapter 11
12. Figures in Chapter 12
13. Figures in Chapter 13
14. Figures in Chapter 14
15. Figures in Chapter 15
16. Figures in Chapter 16
17. Figures in Chapter 17
18. Figures in Chapter 18
19. Figures in Chapter 19

Academic Press is an imprint of Elsevier

125 London Wall, London EC2Y 5AS, United Kingdom

525 B Street, Suite 1650, San Diego, CA 92101, United States

50 Hampshire Street, 5th Floor, Cambridge, MA 02139, United States

The Boulevard, Langford Lane, Kidlington, Oxford OX5 1GB, United Kingdom

Copyright 2020 Elsevier Ltd. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Details on how to seek permission, further information about the Publisher's permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions.

This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein).

Notices

Knowledge and best practice in this field are constantly changing. As new research and experience broaden our understanding, changes in research methods, professional practices, or medical treatment may become necessary.

Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information, methods, compounds, or experiments described herein. In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility.

To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liability for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein.

Library of Congress Cataloging-in-Publication Data

A catalog record for this book is available from the Library of Congress

British Library Cataloguing-in-Publication Data

A catalogue record for this book is available from the British Library

ISBN: 978-0-12-818803-3

For information on all Academic Press publications visit our website at https://www.elsevier.com/books-and-journals

Publisher: Mara Conner

Acquisitions Editor: Tim Pitts

Editorial Project Manager: Charlotte Rowley

Production Project Manager: Paul Prasad Chandramohan

Designer: Greg Harris

Typeset by VTeX

oo`

For Everything

All These Years

Sergios Theodoridis is professor of machine learning and signal processing with the National and Kapodistrian University of Athens, Athens, Greece and with the Chinese University of Hong Kong, Shenzhen, China. He has received a number of prestigious awards, including the 2014 IEEE Signal Processing Magazine Best Paper Award, the 2009 IEEE Computational Intelligence Society Transactions on Neural Networks Outstanding Paper Award, the 2017 European Association for Signal Processing (EURASIP) Athanasios Papoulis Award, the 2014 IEEE Signal Processing Society Education Award, and the 2014 EURASIP Meritorious Service Award. He has served as president of EURASIP and vice president for the IEEE Signal Processing Society. He is a Fellow of EURASIP and a Life Fellow of IEEE. He is the coauthor of the book Pattern Recognition, 4th edition, Academic Press, 2009 and of the book Introduction to Pattern Recognition: A MATLAB Approach, Academic Press, 2010.

Machine learning is a name that is gaining popularity as an umbrella and evolution for methods that have been studied and developed for many decades in different scientific communities and under different names, such as statistical learning, statistical signal processing, pattern recognition, adaptive signal processing, image processing and analysis, system identification and control, data mining and information retrieval, computer vision, and computational learning. The name machine learning indicates what all these disciplines have in common, that is, to learn from data, and then make predictions. What one tries to learn from data is their underlying structure and regularities, via the development of a model, which can then be used to provide predictions.

To this end, a number of diverse approaches have been developed, ranging from optimization of cost functions, whose goal is to optimize the deviation between what one observes from data and what the model predicts, to probabilistic models that attempt to model the statistical properties of the observed data.

The goal of this book is to approach the machine learning discipline in a unifying context, by presenting major paths and approaches that have been followed over the years, without giving preference to a specific one. It is the author's belief that all of them are valuable to the newcomer who wants to learn the secrets of this topic, from the applications as well as from the pedagogic point of view. As the title of the book indicates, the emphasis is on the processing and analysis front of machine learning and not on topics concerning the theory of learning itself and related performance bounds. In other words, the focus is on methods and algorithms closer to the application level.

The book is the outgrowth of more than three decades of the author's experience in research and teaching various related courses. The book is written in such a way that individual (or pairs of) chapters are as self-contained as possible. So, one can select and combine chapters according to the focus he/she wants to give to the course he/she teaches, or to the topics he/she wants to grasp in a first reading. Some guidelines on how one can use the book for different courses are provided in the introductory chapter.