Shroff - The intelligent web: search, smart algorithms, and big data

THE INTELLIGENT WEB

Search, Smart Algorithms, and Big Data

GAUTAM SHROFF

Great Clarendon Street, Oxford, OX2 6 DP ,
United Kingdom

Oxford University Press is a department of the University of Oxford. It furthers the Universitys objective of excellence in research, scholarship, and education by publishing worldwide. Oxford is a registered trade mark of Oxford University Press in the UK and in certain other countries

Gautam Shroff 2013

The moral rights of the author have been asserted

First Edition published in 2013

Impression: 1

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without the prior permission in writing of Oxford University Press, or as expressly permitted by law, by licence or under terms agreed with the appropriate reprographics rights organization. Enquiries concerning reproduction outside the scope of the above should be sent to the Rights Department, Oxford University Press, at the address above

You must not circulate this work in any other form and you must impose this same condition on any acquirer

Published in the United States of America by Oxford University Press 198 Madison Avenue, New York, NY 10016, United States of America

British Library Cataloguing in Publication Data

Data available

Library of Congress Control Number: 2013938816

ISBN 9780199646715

Printed in Italy by
L.E.G.O. S.p.A.-Lavis TN

Links to third party websites are provided by Oxford in good faith and for information only. Oxford disclaims any responsibility for the materials contained in any third party website referenced in this work.

To my late father,
who I suspect would have enjoyed this book the most

Many people have contributed to my thinking and encouraged me while writing this book. But there are a few to whom I owe special thanks. First, to V. S. Subrahamanian, for reviewing the chapters as they came along and supporting my endeavour with encouraging words. I am also especially grateful to Patrick Winston and Pentti Kanerva for sparing the time to speak with me and share their thoughts on the evolution and future of AI.

Equally important has been the support of my family. My wife Brinda, daughter Selena, and son Ahanmany thanks for tolerating my preoccupation on numerous weekends and evenings that kept me away from you. I must also thank my mother for enthusiastically reading many of the chapters, which gave me some confidence that they were accessible to someone not at all familiar with computing.

Last but not least I would like to thank my editor Latha Menon, for her careful and exhaustive reviews, and for shepherding this book through the publication process.

I grew up reading and being deeply influenced by the popular science books of George Gamow on physics and mathematics. This book is my attempt at explaining a few important and exciting advances in computer science and artificial intelligence (AI) in a manner accessible to all. The incredible growth of the internet in recent years, along with the vast volumes of big data it holds, has also resulted in a rather significant confluence of ideas from diverse fields of computing and AI. This new science of web intelligence, arising from the marriage of many AI techniques applied together on big data, is the stage on which I hope to entertain and elucidate, in the spirit of Gamow, and to the best of my abilities.

***

The computer science community around the world recently celebrated the centenary of the birth of the British scientist Alan Turing, widely regarded as the father of computer science. During his rather brief life Turing made fundamental contributions in mathematics as well as some in biology, alongside crucial practical feats such as breaking secret German codes during the Second World War.

Turing was the first to examine very closely the meaning of what it means to compute, and thereby lay the foundations of computer science. Additionally, he was also the first to ask whether the capacity of intelligent thought could, in principle, be achieved by a machine that computed. Thus, he is also regarded as the father of the field of enquiry now known as artificial intelligence.

with, I propose to consider the question, Can machines think? He then goes on to describe the famous Turing Test, which he referred to as the imitation game, as a way to think about the problem of machines thinking. According to the Turing Test, if a computer can converse with any of us humans in so convincing a manner as to fool us into believing that it, too, is a human, then we should consider that machine to be intelligent and able to think.

Recently, in February 2011, IBMs Watson computer managed to beat champion human players in the popular TV show Jeopardy!. Watson was able to answer fairly complex queries such as Which New Yorker who fought at the Battle of Gettysburg was once considered the inventor of baseball?. Figuring out that the answer is actually Abner Doubleday, and not Alexander Cartwright who actually wrote the rules of the game, certainly requires non-trivial natural language processing as well as probabilistic reasoning; Watson got it right, as well as many similar fairly difficult questions.

During this widely viewed Jeopardy! contest, Watsons place on stage was occupied by a computer panel while the human participants were visible in flesh and blood. However, imagine if instead the human participants were also hidden behind similar panels, and communicated via the same mechanized voice as Watson. Would we be able to tell them apart from the machine? Has the Turing Test then been passed, at least in this particular case?

There are more recent examples of apparently successful displays of artificial intelligence: in 2007 Takeo Kanade, the well-known Japanese expert in computer vision, spoke about his early research in face recognition, another task normally associated with humans and at best a few higher-animals: it was with pride that I tested the program on 1000 faces, a rare case at the time when testing with 10 images was considered a large-scale experiment.millions contained amongst the billions of images uploaded by users around the world.

Language is another arena where similar progress is visible for all to see and experience. In 1965 a committee of the US National Academy of Sciences concluded its review of the progress in automated translation between human natural languages with, there is no immediate or predicable prospect of useful machine translation. Today, web users around the world use Googles translation technology on a daily basis; even if the results are far from perfect, they are certainly good enough to be very useful.

Progress in spoken language, i.e., the ability to recognize speech, is also not far behind: Apples Siri feature on the iPhone 4S brings usable and fairly powerful speech recognition to millions of cellphone users worldwide.

As succinctly put by one of the stalwarts of AI, Patrick Winston: AI is becoming more important while it becomes more inconspicuous, as AI technologies are becoming an integral part of mainstream computing.

***

What, if anything, has changed in the past decade that might have contributed to such significant progress in many traditionally hard problems of artificial intelligence, be they machine translation, face recognition, natural language understanding, or speech recognition, all of which have been the focus of researchers for decades?