Doug Turnbull - Relevant Search: With applications for Solr and Elasticsearch

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact

Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com

2016 by Manning Publications Co. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps.

Recognizing the importance of preserving what has been written, it is Mannings policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine.

Manning Publications Co.20 Baldwin RoadPO Box 761Shelter Island, NY 11964

Development editor: Marina MichaelsTechnical development editor: Aaron ColcordCopy editor: Sharon WilkeyProofreader: Elizabeth MartinTechnical proofreader: Valentin CrettazTypesetter: Dennis DalinnikCover designer: Marija Tudor

ISBN: 9781617292774

Printed in the United States of America

1 2 3 4 5 6 7 8 9 10 EBM 21 20 19 18 17 16

Over the last decade, search has become ubiquitousthe keyword search box has evolved to become the de facto UI for exploring data and for navigating most websites and applications. At the same time, delivering a truly relevant search experience has been elusive, if not a critical blind spot for most organizations.

Powerful open source technologies have arisen to deliver fast, feature-rich search (Apache Lucene) in a distributed, highly scalable way with little-to-no coding required (Apache Solr and later Elasticsearch). This has provided the necessary infrastructure for almost any developer to build a generally relevant real-time search engine for the big data era. As more of the hard search infrastructure problems have been solved and their solutions commoditized, the competitive differentiators have moved away from providing fast, scalable search and more toward delivering the most relevant matches for a users information need. In other words, delivering generally relevant results is no longer sufficientGoogle and other top search engines have now trained users to expect search applications to almost read their minds. This book is about how to move more aggressively in that direction of understanding user intent.

Doug Turnbull and John Berryman are two highly experienced search and relevancy experts whom Ive known for years, typically running into each other at search conferences where weve all presented. I fondly recall times spent with them discussing ideas to solve some of the worlds hardest problems in search relevancy, recommendations, and personalization. No one is more excited than I to see their unique expertise codified in this bookone of the best and most engaging technical books Ive ever read.

Relevancy tuning is a hard problemits usually misunderstood, and its often not immediately obvious when something is wrong. It usually requires seeing many bad examples to identify problematic patterns, and its often challenging to know what better results would look like without actually seeing them show up. Unfortunately, its often not until well after a search system is deployed into production that organizations begin to realize the gap between out-of-the-box relevancy defaults and true domain-driven, personalized matching.

Not only that, but the skillsets needed to think about relevancy (domain expertise, feature engineering, machine learning, ontologies, user testing, natural language processing) are very different from those needed to build and maintain scalable infrastructure (distributed systems, data structures, performance and concurrency, hardware utilization, network calls and communication). The role of a relevance engineer is almost entirely lacking in many organizations, leaving so much potential untapped for building a search experience that truly delights users and significantly moves a company forward.

The spectrum of personalization between manually entered keyword searches and completely automated recommendations is also rich with opportunities to deliver relevant matches crafted for each specific users needs. The authors do a great job of explaining some of the more nuanced ways that search features/signals can be modeled to take full advantage of this spectrum. With the techniques in this book, you will be well-equipped to take on the role of a relevance engineer and solve many of the most challenging problems inherent in creating a truly personalized, relevant search experience.

T REY G RAINGER

A UTHOR , S OLR IN A CTION

S ENIOR V ICE P RESIDENT OF E NGINEERING AT L UCIDWORKS

John and I met while working together as consultants for OpenSource Connections (OSC) solving tough search problems for clients. Sometimes we triaged performance (make it go faster!). Other times we helped build out a search application. All of these projects had simple-to-measure success metrics. Did it go faster? Is the application complete?

Search relevance, though, doesnt play by these rules. And users, raised in the age of Google, wont tolerate good enough search. They want damn smart search. They want search to prioritize criteria they care about, not what the search engine often idiotically guesses relevant.

Like moths attracted to a flame, we both felt drawn to this hard problem. And just like said moths, we often found ourselves burned. Through these painful lessons, we persevered and grew, succeeding at tasks we initially considered too difficult.

During this time, we also found our voices on OSCs blog. We realized that little was being written about search relevance problems. We developed ideas such as testdriven relevancy. We documented our headaches, our problems, and our triumphs. Together we experimented with machine learning approaches, like latent semantic analysis. We dove into Lucenes guts and explored techniques for building custom search components to solve problems. We began exploring information retrieval research. As we learned more techniques to solve hard problems, we continued to write about them.

Still, blogs have their limits. John and I always hoped to express our ideas more systematically in book form. Luckily, we experienced one of those funny chains of events that often lead to opportunity knocking. I presented on Python concurrency at a local tech meet-up along with Andrew Montalenti. Since Andrew was giving this talk at PyCon, Manning called Andrew to discuss writing a book on Python concurrency. Andrew said he wasnt interested in writing a book, but perhaps his copresenter Doug would be.