Sundeep Agarwal - GNU SED

Sundeep Agarwal

You are likely to be familiar with the "Find and Replace" dialog box from a text editor, word processor, IDE, etc to search for something and replace it with something else. sed is a command line tool that is similar, but much more versatile and feature-rich. Some of the GUI applications may also support regular expressions, a feature which helps to precisely define a matching criteria. You could consider regular expressions as a mini-programming language in itself, designed to solve various text processing needs.

The book heavily leans on examples to present options and features of sed one by one. Regular expressions will also be discussed in detail. However, commands to manipulate data buffers and multiline techniques will be discussed only briefly and some commands are skipped entirely.

It is recommended that you manually type each example and experiment with them. Understanding both the nature of sample input string and the output produced is essential. As an analogy, consider learning to drive a bike or a car no matter how much you read about them or listen to explanations, you need to practice a lot and infer your own conclusions. Should you feel that copy-paste is ideal for you, code snippets are available chapter wise on GitHub.

Prior experience working with command line and bash shell, should know concepts like file redirection, command pipeline and so on. Knowing basics of grep will also help in understanding filtering features of sed.

If you are new to the world of command line, check out my curated resources on Linux CLI and Shell scripting before starting this book.

My Command Line Text Processing repository includes a chapter on GNU sed which has been edited and expanded to create this book.

• The examples presented here have been tested on GNU bash shell with GNU sed 4.8 and may include features not available in earlier versions.
• Code snippets shown are copy pasted from bash shell and modified for presentation purposes. Some commands are preceded by comments to provide context and explanations. Blank lines to improve readability, only real time shown for speed comparisons, output skipped for commands like wget and so on.
• Unless otherwise noted, all examples and explanations are meant for ASCII characters only.
• sed would mean GNU sed, grep would mean GNU grep and so on unless otherwise specified.
• External links are provided for further reading throughout the book. Not necessary to immediately visit them. They have been chosen with care and would help, especially during rereads.
• The learn_gnused repo has all the code snippets and files used in examples, exercises and other details related to the book. If you are not familiar with git command, click the Code button on the webpage to get the files.

• GNU sed documentation manual and examples
• stackoverflow and unix.stackexchange for getting answers to pertinent questions on bash, sed and other commands
• tex.stackexchange for help on pandoc and tex related questions
• LibreOffice Draw cover image
• pngquant and svgcleaner for optimizing images
• softwareengineering.stackexchange and skolakoda for programming quotes
• Warning and Info icons by Amada44 under public domain
• arifmahmudrana for spotting an ambiguous explanation

Special thanks to all my friends and online acquaintances for their help, support and encouragement, especially during difficult times.

I would highly appreciate if you'd let me know how you felt about this book, it would help to improve this book as well as my future attempts. Also, please do let me know if you spot any error or typo.

Issue Manager: https://github.com/learnbyexample/learn_gnused/issues

E-mail:

Twitter: https://twitter.com/learn_byexample

Sundeep Agarwal is a freelance trainer, author and mentor. His previous experience includes working as a Design Engineer at Analog Devices for more than 5 years. You can find his other works, primarily focused on Linux command line, text processing, scripting languages and curated lists, at https://github.com/learnbyexample. He has also been a technical reviewer for Command Line Fundamentals book and video course published by Packt.

List of books: https://learnbyexample.github.io/books/

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

Code snippets are available under MIT License

Resources mentioned in Acknowledgements section above are available under original licenses.

1.5

See Version_changes.md to track changes across book versions.

The command name sed is derived from stream editor. Here, stream refers to data being passed via shell pipes. Thus, the command's primary functionality is to act as a text editor for stdin data with stdout as the output target. Over the years, functionality was added to edit file input and save the changes back to the same file.

This chapter will cover how to install/upgrade sed followed by details related to documentation. Then, you'll get an introduction to the substitute command, which is the most commonly used sed feature. The chapters to follow will add more details to the substitute command, discuss other commands and command line options. Cheatsheet, summary and exercises are also included at the end of these chapters.

If you are on a Unix like system, you are most likely to already have some version of sed installed. This book is primarily for GNU sed. As there are syntax and feature differences between various implementations, please make sure to follow along with what is presented here. GNU sed is part of text creation and manipulation commands provided by GNU and comes by default on GNU/Linux. To install newer or particular version, visit gnu: software. Check release notes for an overview of changes between versions. See also bug list.

If you are not using a Linux distribution, you may be able to access GNU sed using below options:

• git-bash
• WSL
• brew

It is always a good idea to know where to find the documentation. From command line, you can use man sed for a short manual and info sed for full documentation. I prefer using the online GNU sed manual which feels much easier to use and navigate.

For a quick overview of all the available options, use sed --help from the command line. Most of them will be explained in the coming chapters.

sed has various commands to manipulate text input. substitute command is most commonly used, which will be briefly discussed in this chapter. It is used to replace matching text with something else. The syntax is