Benjamin Bengfort - Applied Text Analysis with Python: Enabling Language Aware Data Products with Machine Learning

by Benjamin Bengfort , Tony Ojeda , and Rebecca Bilbro

Copyright 2016 Benjamin Bengfort, Tony Ojeda, Rebecca Bilbro. All rights reserved.

Printed in the United States of America.

Published by OReilly Media, Inc. , 1005 Gravenstein Highway North, Sebastopol, CA 95472.

OReilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles ( http://oreilly.com/safari ). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com .

• Editor: Nicole Tache
• Production Editor: FILL IN PRODUCTION EDITOR
• Copyeditor: FILL IN COPYEDITOR
• Proofreader: FILL IN PROOFREADER
• Indexer: FILL IN INDEXER
• Interior Designer: David Futato
• Cover Designer: Karen Montgomery
• Illustrator: Rebecca Demarest

• January -4712: First Edition

See http://oreilly.com/catalog/errata.csp?isbn=9781491962978 for release details.

The OReilly logo is a registered trademark of OReilly Media, Inc. Applied Text Analysis with Python, the cover image, and related trade dress are trademarks of OReilly Media, Inc.

While the publisher and the author(s) have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author(s) disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-491-96297-8

[FILL IN]

As we explored the architecture of language in the previous chapter, we began to see that it is possible to model natural language in spite of its complexity and flexibility. And yet, the best language models are often highly constrained and application-specific. Why is it that models trained in a specific field or domain of the language would perform better than ones trained on general language? Consider that the term bank is very likely to be an institution that produces fiscal and monetary tools in an economics, financial, or political domain, whereas in an aviation or vehicular domain it is more likely to be a form of motion that results in the change of direction of an aircraft. By fitting models in a narrower context, the prediction space is smaller and more specific, and therefore better able to handle the flexible aspects of language.

The bulk of our work in the subsequent chapters will be in feature extraction and knowledge engineering - where well be concerned with the identification of unique vocabulary words, sets of synonyms, interrelationships between entities, and semantic contexts. However, all of these techniques will revolve around a central text dataset: the corpus.

Corpora are collections of related documents that contain natural language. A corpus can be large or small, though generally they consist of hundreds of gigabytes of data inside of thousands of documents. For instance, considering that the average email inbox is 2GB, a moderately sized company of 200 employees would have around a half-terabyte email corpus. Documents contained by a corpus can also vary in size, from tweets to books. Corpora can be annotated, meaning that the text or documents are labeled with the correct responses for supervised learning algorithms, or unannotated, making them candidates for topic modeling and document clustering.