Pete Warden - Big Data Glossary

Copyright 2011

OReilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (.

Nutshell Handbook, the Nutshell Handbook logo, and the OReilly logo are registered trademarks of OReilly Media, Inc. Big Data Glossary , the image of an elephant seal, and related trade dress are trademarks of OReilly Media, Inc.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and OReilly Media, Inc., was aware of a trademark claim, the designations have been printed in caps or initial caps.

While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.

Techniques originally developed by website developers coping with scaling issues are increasingly being applied to other domains.

Google has proven that research techniques from computer science can be effective at solving problems and creating value in many real-world situations. Thats led to increased interest in cross-pollination and investment in academic research from commercial organizations.

Now that machines with a decent amount of processing power can be hired for just a few cents an hour, many more people can afford to do large-scale data processing. They cant afford the traditional high prices of professional data software, though, so theyve turned to open source alternatives.

These trends have led to a Cambrian explosion of new tools, which means that when youre planning a new data project, you have a lot to choose from. This guide aims to help you make those choices by describing each tool from the perspective of a developer looking to use it in an application. Wherever possible, this will be from my firsthand experiences or from those of colleagues who have used the systems in production environments. Ive made a deliberate choice to include my own opinions and impressions, so you should see this guide as a starting point for exploring the tools, not the final word. Ill do my best to explain what I like about each service, but your tastes and requirements may well be quite different.

Since the goal is to help experienced engineers navigate the new data landscape, this guide only covers tools that have been created or risen to prominence in the last few years. For example, Postgres is not covered because its been widely used for over a decade, but its Greenplum derivative is newer and less well-known, so it is included.