Sundeep Agarwal - Python re(gex)?

Sundeep Agarwal

Scripting and automation tasks often need to extract particular portions of text from input data or modify them from one format to another. This book will help you learn Regular Expressions, a mini-programming language for all sorts of text processing needs.

The book heavily leans on examples to present features of regular expressions one by one. It is recommended that you manually type each example and experiment with them. Understanding both the nature of sample input string and the output produced is essential. As an analogy, consider learning to drive a bike or a car no matter how much you read about them or listen to explanations, you need to practice a lot and infer your own conclusions. Should you feel that copy-paste is ideal for you, code snippets are available chapter wise on GitHub.

Prior experience working with Python, should know concepts like string formats, string methods, list comprehension and so on.

If you have prior experience with a programming language, but new to Python, see my list of Python curated resources before starting this book.

• The examples presented here have been tested with Python version 3.8.3 and includes features not available in earlier versions.
• Code snippets shown are copy pasted from Python REPL shell and modified for presentation purposes. Some commands are preceded by comments to provide context and explanations. Blank lines have been added to improve readability. Error messages are shortened. import statements are skipped after initial use. And so on.
• Unless otherwise noted, all examples and explanations are meant for ASCII characters.
• External links are provided for further reading throughout the book. Not necessary to immediately visit them. They have been chosen with care and would help, especially during re-reads.
• The py_regular_expressions repo has all the code snippets and files used in examples and exercises and other details related to the book. If you are not familiar with git command, click the Clone button on the webpage to get the files.

• Python documentation manuals and tutorials
• /r/learnpython/ and /r/regex/ helpful forums for beginners and experienced programmers alike
• stackoverflow for getting answers to pertinent questions on Python and regular expressions
• tex.stackexchange for help on pandoc and tex related questions
• Cover image: draw.io, tree icon by Gopi Doraisamy under Creative Commons Attribution 3.0 Unported and wand icon by roundicons.com
• Warning and Info icons by Amada44 under public domain
• David Cortesi for helpful feedback on both the technical content and grammar issues
• Kye for spotting a typo
• Hugh's email exchanges helped me significantly to improve the presentation of concepts and exercises
• Christopher Patti for reviewing the book, providing feedback and brightening the day with kind words
• Users 73tada, DrBobHope, nlomb and others for feedback in this reddit thread

Special thanks to Al Sweigart, for introducing me to Python with his awesome automatetheboringstuff book and video course.

I would highly appreciate if you'd let me know how you felt about this book, it would help to improve this book as well as my future attempts. Also, please do let me know if you spot any error or typo.

Issue Manager: https://github.com/learnbyexample/py_regular_expressions/issues

Goodreads: https://www.goodreads.com/book/show/47142552-python-re-gex

E-mail:

Twitter: https://twitter.com/learn_byexample

Sundeep Agarwal is a freelance trainer, author and mentor. His previous experience includes working as a Design Engineer at Analog Devices for more than 5 years. You can find his other works, primarily focused on Linux command line, text processing, scripting languages and curated lists, at https://github.com/learnbyexample. He has also been a technical reviewer for Command Line Fundamentals book and video course published by Packt.

List of books: https://learnbyexample.github.io/books/

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

Code snippets are available under MIT License

Resources mentioned in Acknowledgements section above are available under original licenses.

3.1
See Version_changes.md to track changes across book versions.

Regular Expressions is a versatile tool for text processing. You'll find them included as part of standard library of most programming languages that are used for scripting purposes. If not, you can usually find a third-party library. Syntax and features of regular expressions vary from language to language. Python's syntax is similar to that of Perl language, but there are significant feature differences.

The str class comes loaded with variety of methods to deal with text. So, what's so special about regular expressions and why would you need it? For learning and understanding purposes, one can view regular expressions as a mini programming language in itself, specialized for text processing. Parts of a regular expression can be saved for future use, analogous to variables and functions. There are ways to perform AND, OR, NOT conditionals. Operations similar to range function, string repetition operator and so on.

Here's some common use cases.

• Sanitizing a string to ensure that it satisfies a known set of rules. For example, to check if a given string matches password rules.
• Filtering or extracting portions on an abstract level like alphabets, numbers, punctuation and so on.
• Qualified string replacement. For example, at the start or the end of a string, only whole words, based on surrounding text, etc.

You are likely to be familiar with graphical search and replace tool, like the screenshot shown below from LibreOffice Writer. Match case, Whole words only, Replace and Replace All are some of the basic features supported by regular expressions.

Another real world use case is password validation. The screenshot below is from GitHub sign up page. Performing multiple checks like string length and the type of characters allowed is another core feature of regular expressions.

Here's some articles on regular expressions to know about its history and the type of problems it is suited for.

• The true power of regular expressions it also includes a nice explanation of what regular means in this context
• softwareengineering: Is it a must for every programmer to learn regular expressions?
• softwareengineering: When you should NOT use Regular Expressions?
• codinghorror: Now You Have Two Problems
• wikipedia: Regular expression this article includes discussion on regular expressions as a formal language as well as details on various implementations