Rostami Mohammad - Transfer Learning Through Embedding Spaces

First edition published 2021

by CRC Press

6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487-2742

and by CRC Press

2 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN

2021 Mohammad Rostami

CRC Press is an imprint of Taylor & Francis Group, LLC

The right of Mohammad Rostamito be identified as author of this work has been asserted by him in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988.

Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers.

For permission to photocopy or use material electronically from this work, access

Trademark notice: Product or corporate names may be trademarks or registered trademarks and are used only for identification and explanation without intent to infringe.

Library of Congress Cataloging-in-Publication Data

ISBN: 9780367699055 (hbk)

ISBN: 9780367703868 (pbk)

ISBN: 9781003146032 (ebk)

Typeset in Nimbus font

by KnowledgeWorks Global Ltd.

To my wife and my parents.

1. ].

The unprecedented processing demand, posed by the explosion of big data, challenges researchers to design efficient and adaptive machine learning algorithms that do not require persistent retraining and avoid learning redundant information. Inspired from learning techniques of intelligent biological agents, identifying transferable knowledge across learning problems has been a significant research focus to improve machine learning algorithms. In this book, we explore the challenges of knowledge transfer through embedding spaces that capture and store hierarchical knowledge.

In the first part of the book, we focus on the problem of cross-domain knowledge transfer. We first study the problem of zero-shot image classification, where the goal is to identify images from unseen classes using semantic descriptions of these classes. We then study feasibility of using this idea to match data distributions of two visual domains in a shared cross-domain embedding space. In the second part of the book, we investigate the problem of cross-task knowledge transfer. Here, the goal is to identify relations and similarities of multiple machine learning tasks to improve performance across the tasks. We first study the problem of zero-shot learning in a lifelong machine learning setting, where the goal is to learn tasks with no data using high-level task descriptions. The idea is to relate high-level task descriptors to the optimal task parameters through an embedding space. We then study the problem of catastrophic forgetting within continual learning setting of deep neural networks, where the idea is to enforce the tasks to share the same distribution in an embedding space. We further demonstrate that we can address the challenges of domain adaptation in the continual learning setting through matching distributions in an embedding space. Finally, we consider the problem of cross-agent knowledge transfer in the third part of thesis book. We demonstrate that multiple lifelong machine learning agents can collaborate to increase individual performances by sharing learned knowledge using an embedding space in a fully distributed learning setting.

We demonstrate that despite major differences, problems within the above learning scenarios can be tackled through learning an intermediate embedding space. Implementation of some of the algorithms presented in this book is publicly available at https://github.com/mrostami1366 .

I am grateful to many people who helped me to make this work possible. This book is primarily based on my PhD dissertation []. I apologize in advance to those whom I have not mentioned in this part. I would like to express my appreciation to my PhD supervisors, my MASc supervisors, my dissertation committee members, and my research collaborators who helped me during all the PhD studies years. Without their help and support, this work would not be possible.

Although this book is a result of my studies at the University of Pennsylvania and later at the University of Southern California, I think prior training for many years before graduate studies significantly contributed to development of this book. In particular, I would like to thank my professors at Sharif University of Technology who helped me to gain the background knowledge I needed for graduate school. I also thank my teachers at Shahid Beheshti elementary and high schools in my hometown, Zanjan, whose teachings have influenced my abilities forever.

Finally, my family members have played a crucial role in fulfillment of this book. I would like to thank my parents for their endless love and support. My upbringing played a primary role in pursuing higher education. I also thank my brothers who performed duties that were also partially on me in my absence. I thank my in-law family, who supported me and my wife to pursue higher education. Last but certainly not the least, my deep gratitude goes to my dear wife and best friend who patiently helped me to go through years of graduate studies while we were thousands of miles away from our homeland.

The emergence of data-driven industries, high-performance computing technologies, the Internet of Things (IoT), and crowdsourcing platform has led to the unprecedented deployment of Machine Learning (ML) algorithms and techniques in a broad range of applications. These applications include problems within computer vision, natural language processing, robotics, complex network science, and decision-making, among many others. Despite the diversity of these applications, the common goal for practical purposes is to develop algorithms and techniques that can mimic humans and replace or augment them in tasks at which humans are slow, inefficient, or inaccurate.

Some of the current ML algorithms have reached human-level performance, readily are used in practice, and their impact on the economy can be observed. In the consumer market, we can see that commercial personal assistant robots are available to purchase, the first commercial drone delivery service kicked off recently, autonomous vehicles are being tested at the moment, and even some ML-based medical diagnosis algorithms are as good as trained specialists. Progress in ML and Artificial Intelligence (AI) is not limited to handful of examples. In the stock market, we can see that nine of the ten largest companies by market capital are companies that heavily invest in and use ML []. All the above demonstrates that the importance of AI and ML has gone far beyond just a research area and many unexplored aspects need to be addressed by people outside the AI academic community.