Terence Parr - The Definitive ANTLR 4 Reference, 2nd Edition

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and The Pragmatic Programmers, LLC was aware of a trademark claim, the designations have been printed in initial capital letters or in all capitals. The Pragmatic Starter Kit, The Pragmatic Programmer, Pragmatic Programming, Pragmatic Bookshelf and the linking g device are trademarks of The Pragmatic Programmers, LLC.

Every precaution was taken in the preparation of this book. However, the publisher assumes no responsibility for errors or omissions, or for damages that may result from the use of information (including program listings) contained herein.

Our Pragmatic courses, workshops, and other products can help you and your team create better software and have more fun. For more information, as well as the latest Pragmatic titles, please visit us at http://pragprog.com.

Its been roughly 25 years since I started working on ANTLR. In that time, many people havehelped shape the tool syntax and functionality, for which Im most grateful. Mostimportantly for ANTLR version 4, SamHarwell was mycoauthor. He helped write the software but also made critical contributions to theAdaptive LL(*) grammar analysis algorithm. Sam is also building the ANTLRWorks2grammar IDE.

The following people provided technical reviews: Oliver Ziegermann, Sam Rose, Kyle Ferrio, Maik Schmidt, Colin Yates, Ian Dees, Tim Ottinger, Kevin Gisi, Charley Stran, Jerry Kuch, Aaron Kalair, Michael Bevilacqua-Linn, Javier Collado, Stephen Wolff, and Bernard Kaiflin. I also appreciate those people who reported errors in beta versions of the book and v4 software. Kim Shrier and Graham Wideman deserve special attention because they provided such detailed reviews. Grahams technical reviews were so elaborate, voluminous, and extensive that I wasnt sure whether to shake his hand vigorously or go buy a handgun.

Finally, Id like to thank Pragmatic Bookshelf editor Susannah Davidson Pfalzer, who has stuck with me through three books! Her suggestions and careful editing really improved this book.

ANTLR v4 is a powerful parser generator that you can use to read, process, execute, ortranslate structured text or binary files. Its widely used in academia and industry tobuild all sorts of languages, tools, and frameworks. Twitter search uses ANTLR for queryparsing, with more than 2 billion queries a day. The languages for Hive and Pig and the datawarehouse and analysis systems for Hadoop all use ANTLR. LexMachina uses ANTLR forinformation extraction from legal texts. Oracle uses ANTLR within the SQL Developer IDE andits migration tools. The NetBeans IDE parses C++ with ANTLR. The HQL language in the Hibernateobject-relational mapping framework is built with ANTLR.

Aside from these big-name, high-profile projects, you can build all sorts of useful tools such as configuration file readers, legacy code converters, wiki markup renderers, and JSON parsers. Ive built little tools for creating object-relational database mappings, describing 3D visualizations, and injecting profiling code into Java source code, and Ive even done a simple DNA pattern matching example for a lecture.

From a formal language description called a grammar , ANTLR generates a parser for thatlanguage that can automatically build parse trees, which are data structures representinghow a grammar matches the input. ANTLR also automatically generates tree walkers that youcan use to visit the nodes of those trees to execute application-specific code.

This book is both a reference for ANTLR v4 and a guide to using it to solve language recognition problems. Youre going to learn how to do the following:

• Identify grammar patterns in language samples and reference manuals in order to build yourown grammars.

• Build grammars for simple languages like JSON all the way up to complex programming languages like R. Youll also solve some tricky recognition problems from Python and XML.

• Implement language applications based upon those grammars by walking the automatically generated parse trees.

• Customize recognition error handling and error reporting for specific application domains.