Marco Gori Ph.D. - Machine Learning: A Constraint-Based Approach

First edition

Marco Gori

Universit di Siena

1. Tables in Chapter 2
2. Tables in Chapter 3
3. Tables in Chapter 5
4. Tables in Chapter 6

1. Figures in Chapter 1
2. Figures in Chapter 2
3. Figures in Chapter 3
4. Figures in Chapter 4
5. Figures in Chapter 5
6. Figures in Chapter 6

Morgan Kaufmann is an imprint of Elsevier

50 Hampshire Street, 5th Floor, Cambridge, MA 02139, United States

Copyright 2018 Elsevier Ltd. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Details on how to seek permission, further information about the Publisher's permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions.

This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein).

Notices

Knowledge and best practice in this field are constantly changing. As new research and experience broaden our understanding, changes in research methods, professional practices, or medical treatment may become necessary.

Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information, methods, compounds, or experiments described herein. In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility.

To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liability for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein.

Library of Congress Cataloging-in-Publication Data

A catalog record for this book is available from the Library of Congress

British Library Cataloguing-in-Publication Data

A catalogue record for this book is available from the British Library

ISBN: 978-0-08-100659-7

For information on all Morgan Kaufmann publications visit our website at https://www.elsevier.com/books-and-journals

Publisher: Katey Birtcher

Acquisition Editor: Steve Merken

Editorial Project Manager: Peter Jardim

Production Project Manager: Punithavathy Govindaradjane

Designer: Miles Hitchen

Typeset by VTeX

To the Memory of My Father who provided me with enough examples to appreciate the importance of hard work to achieve goals and to disclose the beauty of knowledge.

Machine Learning projects our ultimate desire to understand the essence of human intelligence onto the space of technology. As such, while it cannot be fully understood in the restricted field of computer science, it is not necessarily the search of clever emulations of human cognition. While digging into the secrets of neuroscience might stimulate refreshing ideas on computational processes behind intelligence, most of nowadays advances in machine learning rely on models mostly rooted in mathematics and on corresponding computer implementation. Notwithstanding brain science will likely continue the path towards the intriguing connections with artificial computational schemes, one might reasonably conjecture that the basis for the emergence of cognition should not necessarily be searched in the astonishing complexity of biological solutions, but mostly in higher level computational laws. The biological solutions for supporting different forms of cognition are in fact cryptically interwound with the parallel need of supporting other fundamental life functions, like metabolism, growth, body weight regulation, and stress response. However, most human-like intelligent processes might emerge regardless of this complex environment. One might reasonably suspect that those processes be the outcome of information-based laws of cognition, that hold regardless of biology. There is clear evidence of such an invariance in specific cognitive tasks, but the challenge of artificial intelligence is daily enriching the range of those tasks. While no one is surprised anymore to see the computer power in math and logic operations, the layman is not very well aware of the outcome of challenges on games, yet. They are in fact commonly regarded as a distinctive sign of intelligence, and it is striking to realize that games are already mostly dominated by computer programs! Sam Loyd's 15 puzzle and the Rubik's cube are nice examples of successes of computer programs in classic puzzles. Chess, and more recently, Go clearly indicate that machines undermines the long last reign of human intelligence. However, many cognitive skills in language, vision, and motor control, that likely rely strongly on learning, are still very hard to achieve.

Machine learning and information-based laws of cognition.

This book drives the reader into the fascinating field of machine learning by offering a unified view of the discipline that relies on modeling the environment as an appropriate collection of constraints that the agent is expected to satisfy. Nearly every task which has been faced in machine learning can be modeled under this mathematical framework. Linear and threshold linear machines, neural networks, and kernel machines are mostly regarded as adaptive models that need to softly-satisfy a set of point-wise constraints corresponding to the training set. The classic risk, in both the functional and empirical forms, can be regarded as a penalty function to be minimized in a soft-constrained system. Unsupervised learning can be given a similar formulation, where the penalty function somewhat offers an interpretation of the data probability distribution. Information-based indexes can be used to extract unsupervised features, and they can clearly be thought of as a way of enforcing soft-constraints. An intelligent agent, however, can strongly benefit also from the acquisition of abstract granules of knowledge given in some logic formalism. While artificial intelligence has achieved a remarkable degree of maturity in the topic of knowledge representation and automated reasoning, the foundational theories that are mostly rooted in logic lead to models that cannot be tightly integrated with machine learning. While regarding symbolic knowledge bases as a collection of constraints, this book draws a path towards a deep integration with machine learning that relies on the idea of adopting multivalued logic formalisms, like in fuzzy systems. A special attention is reserved to deep learning, which nicely fits the constrained-based approach followed in this book. Some recent foundational achievements on representational issues and learning, joined with appropriate exploitation of parallel computation, have been creating a fantastic catalyst for the growth of high tech companies in related fields all around the world. In the book I do my best to jointly disclose the power of deep learning and its interpretation in the framework of constrained environments, while warning from uncritical blessing. In so doing, I hope to stimulate the reader to conquer the appropriate background to be ready to quickly grasp also future innovations.