Radu Gheorghe - Elasticsearch in Action

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact

Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com

2016 by Manning Publications Co. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps.

Recognizing the importance of preserving what has been written, it is Mannings policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine.

Manning Publications Co.20 Baldwin RoadPO Box 761Shelter Island, NY 11964

Development editor: Susan ConantTechnical development editor: David PombalCopyeditor: Linda RecktenwaldProofreader: Melody DolabTechnical proofreader: Valentin CrettazTypesetter: Dennis DalinnikCover designer: Marija Tudor

ISBN: 9781617291623

Printed in the United States of America

1 2 3 4 5 6 7 8 9 10 EBM 20 19 18 17 16 15

While writing this book, my objective was to provide you the information I needed when I started using Elasticsearch: what its main features are and how they work under the hood. To give you a better overview of this objective, let me tell you a more detailed story of how this book came to life.

I first met Elasticsearch in 2011 while working on a project for centralizing logs. My colleague Mihai Sandu showed me Graylog, which used Elasticsearch for log search, and setting everything up was extremely easy. Two servers could handle all our logging needs at the time, but we expected the data volume to grow hundreds of times in about one year. And it did. On top of that, we had more and more complex analysis requirements, so we quickly found out that tuning and scaling the setup required a deep understanding of Elasticsearch and its features.

There was no book to teach us that, so we had to learn the hard way: lots of experiments, lots of questions and answers to the mailing list. The upside was that I got to know a lot of nice people that posted there regularly. This is how I came to work at Sematext, where I could concentrate on Elasticsearch full-time, and this is why Manning asked me if I would be interested in writing about Elasticsearch.

Of course I was. They warned me it was hard work, but told me that Lee Hinman was also interested, so we joined forces. With two authors, we thought it was going to be easy, especially as Lee and I really clicked and provided useful feedback to one another. Little did we know that its much easier to present features in the early chapters than to combine those features into best practices for various use cases in later chapters. Then, with feedback from our reviewers, we found that its even more work to fit everything together, so our pace became slower and slower. Thats when Roy Russo joined us and helped with that final push.

After two and a half years of early mornings, late nights, and weekends, I can finally say were done. It was a tough experience, but a rich one as well. I would surely have loved to have this book in my hands four years ago, and I hope youll enjoy it, too.

R ADU G HEORGHE

Many people provided their invaluable support to make this book possible:

• Susan Conant, our development editor at Manning, who supported us in so many ways: by providing valuable feedback on draft chapters, helping to plan book and individual chapter structures, giving encouragement, advising us on upcoming steps, helping us overcome bumps in the road, and so on
• Jettro Coenradie, our technical editor, who helped us review big chunks of the manuscript before it went to production and again helped with the final steps before the book went to press
• Valentin Crettaz, who helped with his thorough technical proofread
• Our Manning Early Access Program (MEAP) readers who posted so many helpful comments in the Author Online forum
• The reviewers from the development process who provided such good feedback that I cant even begin to imagine how the book would look without them: Achim Friedland, Alan McCann, Artur Nowak, Bhaskar Karambelkar, Daniel Beck, Gabriel Katenbaumn, Gianluca Rhigetto, Igor Motov, Jeelani Shaik, Joe Gallo, Konstantin Yakushev, Koray Gcl, Michael Schleichardt, Paul Stadig, Ray Lugo Jr., Sen Xu, and Tanguy Leroux

R ADU G HEORGHE

Id like to express my thanks in chronological order. To my colleagues from Avira: Mihai Sandu, Mihai Efrim, Martin Ahrens, Matthias Ollig and many others, for supporting me in learning about Elasticsearch and tolerating my not-always-successful experiments. To my colleagues from Sematext: Otis Gospodneti, who supported me in learning and interacting with the community, and Rafa Ku (aka Master Rafa) for his invaluable tips and tricks. Finally, Id like to thank my family for supporting me in so many ways that I can barely scratch the surface here: my parents, Nicoleta and Mihai Gheorghe, and my in-laws, Madalina and Adrian Radu, for providing good food, quiet spaces, and the all-important moral support. My wife Alexandra, for being a real hero: she somehow managed to write her own stuff and still take care of everything in order for me to write. Last but not least, my son Andrei, now 6, for his understanding and his creative solutions on spending time together, like working on his own book next to me.

L EE H INMAN

First and foremost Id like to give my sincerest thanks to my wife Delilah for encouraging me in this endeavor and for being my adventuring partner. You have given me so much support in this and so many other parts of my life. Thank you for continuing to encourage me throughout the birth of our daughter, Vera Ovelia. Id also like to thank all of the people who have contributed to Elasticsearch. Without you, open source software would not be possible. Im honored to contribute to such a wide-reaching and powerful piece of software.

R OY R USSO

I would like to thank my daughters Olivia and Isabella, my son Jacob, and my wife Roberta, for standing beside me throughout my career and acting as a source of inspiration and motivation. You guys make the impossible possible with your support, love, and understanding.

Since it came out in 2010, Elasticsearch has become increasingly popular. Its being used in a variety of setups, from product searchwhich is the traditional use case for a search engineto real-time analytics of social media, application logs, and other flowing data. The strong points of Elasticsearch have always been its distributed modelwhich makes it scale out easily and efficientlyas well as its rich analytics functionality. All of this was built on top of the already established Apache Lucene search engine library. Lucene has evolved during this time as well, making it possible to process the same amount of data with less CPU, memory, and disk space.