Sundeep Agarwal - GNU AWK

Sundeep Agarwal

When it comes to command line text processing, from an abstract point of view, there are three major pillars grep for filtering, sed for substitution and awk for field processing. These tools have some overlapping features too, for example, all three of them have extensive filtering capabilities.

Unlike grep and sed, awk is a full blown programming language. However, this book intends to showcase awk one-liners that can be composed from the command line instead of writing a program file.

This book heavily leans on examples to present options and features of awk one by one. It is recommended that you manually type each example and experiment with them. Understanding both the nature of sample input string and the output produced is essential. As an analogy, consider learning to drive a bike or a car no matter how much you read about them or listen to explanations, you need to practice a lot and infer your own conclusions. Should you feel that copy-paste is ideal for you, code snippets are available chapter wise on GitHub.

• Prior experience working with command line and bash shell, should know concepts like file redirection, command pipeline and so on
• Familiarity with programming concepts like variables, printing, functions, control structures, arrays, etc
• Knowing basics of grep and sed will help in understanding similar features of awk

If you are new to the world of command line, check out my curated resources on Linux CLI and Shell scripting before starting this book.

My Command Line Text Processing repository includes a chapter on GNU awk which has been edited and restructured to create this book.

• The examples presented here have been tested on GNU bash shell with GNU awk 5.1.0 and includes features not available in earlier versions.
• Code snippets shown are copy pasted from bash shell and modified for presentation purposes. Some commands are preceded by comments to provide context and explanations. Blank lines have been added to improve readability, only real time is shown for speed comparisons, output is skipped for commands like wget and so on.
• Unless otherwise noted, all examples and explanations are meant for ASCII characters.
• awk would mean GNU awk, grep would mean GNU grep and so on unless otherwise specified.
• External links are provided for further reading throughout the book. Not necessary to immediately visit them. They have been chosen with care and would help, especially during re-reads.
• The learn_gnuawk repo has all the code snippets and files used in examples, exercises and other details related to the book. If you are not familiar with git command, click the Code button on the webpage to get the files.

• GNU awk documentation manual and examples
• stackoverflow and unix.stackexchange for getting answers to pertinent questions on bash, awk and other commands
• tex.stackexchange for help on pandoc and tex related questions
• LibreOffice Draw cover image
• pngquant and svgcleaner for optimizing images
• softwareengineering.stackexchange and skolakoda for programming quotes
• Warning and Info icons by Amada44 under public domain
• arifmahmudrana for spotting an ambiguous explanation

Special thanks to all my friends and online acquaintances for their help, support and encouragement, especially during these difficult times.

I would highly appreciate if you'd let me know how you felt about this book, it would help to improve this book as well as my future attempts. Also, please do let me know if you spot any error or typo.

Issue Manager: https://github.com/learnbyexample/learn_gnuawk/issues

E-mail:

Twitter: https://twitter.com/learn_byexample

Sundeep Agarwal is a freelance trainer, author and mentor. His previous experience includes working as a Design Engineer at Analog Devices for more than 5 years. You can find his other works, primarily focused on Linux command line, text processing, scripting languages and curated lists, at https://github.com/learnbyexample. He has also been a technical reviewer for Command Line Fundamentals book and video course published by Packt.

List of books: https://learnbyexample.github.io/books/

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

Code snippets are available under MIT License

Resources mentioned in Acknowledgements section are available under original licenses.

1.3

See Version_changes.md to track changes across book versions.

The command name awk is derived from its developers Alfred V. Aho, Peter J. Weinberger, and Brian W. Kernighan. Over the years, it has been adapted and modified by various other developers. See gawk manual: History for more details. This chapter will show how to install or upgrade awk followed by details related to documentation.

If you are on a Unix like system, you are most likely to already have some version of awk installed. This book is primarily for GNU awk. As there are syntax and feature differences between various implementations, please make sure to follow along with what is presented here. GNU awk is part of text creation and manipulation commands provided by GNU. To install newer or particular version, visit gnu: software gawk. Check release notes for an overview of changes between versions.

See also gawk manual: Installation for advanced options and instructions to install awk on other platforms.

It is always a good idea to know where to find the documentation. From command line, you can use man awk for a short manual and info awk for full documentation. The online GNU awk manual has a better reading interface and provides the most complete documentation, examples and information about other awk versions, POSIX standard, etc.

Here's a snippet from man awk:

For a quick overview of all the available options, use awk --help from the command line.

This chapter will give an overview of awk syntax and some examples to show what kind of problems you could solve using awk. These features will be covered in depth in later chapters, but don't go skipping this chapter.

awk provides filtering capabilities like those supported by grep and sed plus some nifty features of its own. And similar to many command line utilities, awk can accept input from both