Sundeep Agarwal - Command line text processing with GNU Coreutils

Sundeep Agarwal

You might be already aware of popular coreutils commands like head, tail, tr, sort, etc. This book will teach you more than twenty of such specialized text processing tools provided by the GNU coreutils package.

My Command Line Text Processing repo includes chapters on some of these coreutils commands. Those chapters have been significantly edited for this book and new chapters have been added to cover more commands.

Prior experience working with command line and bash shell, should know concepts like file redirection, command pipeline and so on.

If you are new to the world of command line, check out my curated resources on Linux CLI and Shell scripting before starting this book.

• The examples presented here have been tested on GNU bash shell and version 8.30 of the GNU coreutils package.
• Code snippets shown are copy pasted from bash shell and modified for presentation purposes. Some commands are preceded by comments to provide context and explanations. Blank lines have been added to improve readability, only real time is shown for speed comparisons and so on.
• Unless otherwise noted, all examples and explanations are meant for ASCII characters.
• External links are provided throughout the book for you to explore certain topics in more depth.
• The cli_text_processing_coreutils repo has all the code snippets, example files and other details related to the book. If you are not familiar with git command, click the Code button on the webpage to get the files.

• /r/commandline/, /r/linux4noobs/ and /r/linux/ helpful forums
• stackoverflow and unix.stackexchange for getting answers on pertinent questions related to cli tools
• tex.stackexchange for help on pandoc and tex related questions
• canva cover image
• Warning and Info icons by Amada44 under public domain
• pngquant and svgcleaner for optimizing images

I would highly appreciate if you'd let me know how you felt about this book, it would help to improve this book as well as my future attempts. Also, please do let me know if you spot any error or typo.

Issue Manager: https://github.com/learnbyexample/cli_text_processing_coreutils/issues

E-mail:

Twitter: https://twitter.com/learn_byexample

Sundeep Agarwal is a freelance trainer, author and mentor. His previous experience includes working as a Design Engineer at Analog Devices for more than 5 years. You can find his other works, primarily focused on Linux command line, text processing, scripting languages and curated lists, at https://github.com/learnbyexample. He has also been a technical reviewer for Command Line Fundamentals book and video course published by Packt.

List of books: https://learnbyexample.github.io/books/

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

Code snippets are available under MIT License

Resources mentioned in Acknowledgements section above are available under original licenses.

1.0

See Version_changes.md to track changes across book versions.

I've been using Linux since 2007, but it took me ten more years to really explore coreutils for my Command Line Text Processing repository.

Any beginner learning Linux command line tools would come across cat within the first week. Sooner or later, they'll come to know popular text processing tools like grep, head, tail, tr, sort, etc. If you were like me, you'd come across sed and awk, shudder at their complexity and prefer to use a scripting language like Perl and text editors like Vim instead (don't worry, I've already corrected that mistake).

Knowing power tools like grep, sed and awk can help solve most of your text processing needs. So, why would you want to learn text processing tools from the coreutils package? The biggest motivation would be faster execution since these tools are optimized for the use cases they solve. And there's always the advantage of not having to write code (and test that solution) if there's an existing tool to solve the problem.

This book will teach you more than twenty of such specialized text processing tools provided by the GNU coreutils package. Plenty of examples are provided to make it easier to understand a particular tool and its various features.

Writing a book always has a few pleasant surprises for me. For this one, it was discovering a sort option for calendar months, regular expression based features of tac and nl commands, etc.

On a GNU/Linux based OS, you are most likely to already have GNU coreutils installed. This book covers version 8.30 of the coreutils package. To install a newer/particular version, see Coreutils download section for links and details.

If you are not using a Linux distribution, you may be able to access coreutils using these options:

• WSL
• brew

It is always a good idea to know where to find the documentation. From the command line, you can use the man and info commands for brief manual and full documentation respectively. I prefer using the online GNU coreutils manual which feels much easier to use and navigate.

See also:

• Release notes overview of changes and bug fixes between versions
• Bug list
• Common options
• FAQ
• Gotchas

cat derives its name from concatenation and provides other nifty options too.

tac helps you to reverse the input line wise, usually used for further text processing.

Yeah, cat can be used to write contents to a file by typing them from the terminal itself. If you invoke cat without providing file arguments or stdin from a pipe, it will wait for you to type the content. After you are done typing all the text you want to save, press Enter key and then Ctrl+d key combination. If you don't want the last line to have a newline character, press Ctrl+d twice instead of Enter and Ctrl+d. See also unix.stackexchange: difference between Ctrl+c and Ctrl+d.

In the above example, the output of cat is redirected to a file named greeting.txt. If you don't redirect the stdout, each line will be echoed as you type. You can check the contents of the file you just created by using cat again.

Here Documents is another popular way to create such files. Especially in shell scripts, since pressing Ctrl+d interactively won't be possible. Here's an example:

The termination string is enclosed in single quotes to prevent parameter expansion, command substitution, etc. You can also use