James Pustejovsky - Natural Language Annotation for Machine Learning: A guide to corpus-building for applications

Copyright 2012 James Pustejovsky and Amber Stubbs

More specifically, this book details the multi-stage process for building your own annotated natural language dataset (known as a corpus) in order to train machine learning (ML) algorithms for language-based data and knowledge discovery. The overall goal of this book is to show readers how to create their own corpus, starting with selecting an annotation task, creating the annotation specification, designing the guidelines, creating a "gold standard" corpus, and then beginning the actual data creation with the annotation process.

Because the annotation process is not linear, multiple iterations can be required for defining the tasks, annotations, and evaluations, in order to achieve the best results for a particular goal. The process can be summed up in terms of the MATTER Annotation Development Process cycle: Model, Annotate Train, Test, Evaluate, Revise. This books guides the reader through the cycle, and provides case studies for four different annotation tasks. These tasks are examined in detail to provide context for the reader and help provide a foundation for their own machine learning goals.

Additionally, this book provides lightweight, user-friendly software that can be used for annotating texts and adjudicating the annotations. While there are a variety of annotation tools available to the community, the Multi-purpose Annotation Environment (MAE), adopted in this book (and available to readers as a free download), was specifically designed to be easy to set up and get running, so readers will not be distracted from their goal with confusing documentation. MAE is paired with the Multi-document Adjudication Interface (MAI), a tool that allows for quick comparison of annotated documents.

This book is ideal for anyone interested in using computers to explore aspects of the information content conveyed by natural language. It is not necessary to have a programming or linguistics background to use this book, although a basic understanding of a scripting language like Python can make the MATTER cycle easier to follow. If you dont have any Python experience, we highly recommend the OReilly book Natural Language Processing with Python by Steven Bird, Ewan Klein, and Edward Loper, which provides an excellent introduction both to Python and to aspects of NLP that are not addressed in this book.