Kaiwan N Billimoria - Linux Kernel Debugging

Copyright 2022 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Early Access Publication: Linux Kernel Debugging

Early Access Production Reference: B17445

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK

ISBN: 978-1-80107-503-9

Copyright 2021 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Welcome to Packt Early Access. Were giving you an exclusive preview of this book before it goes on sale. It can take many months to write a book, but our authors have cutting-edge information to share with you today. Early Access gives you an insight into the latest developments by making chapter drafts available. The chapters may be a little rough around the edges right now, but our authors will update them over time. Youll be notified when a new version is ready.

This title is in development, with more chapters still to be written, which means you have the opportunity to have your say about the content. We want to publish books that provide useful information to you and other customers, so well send questionnaires out to you regularly. All feedback is helpful, so please be open about your thoughts and opinions. Our editors will work their magic on the text of the book, so wed like your input on the technical elements and your experience as a reader. Well also provide frequent updates on how our authors have changed their chapters based on your feedback.

You can dip in and out of this book or follow along from start to finish; Early Access is designed to be flexible. We hope you enjoy getting to know more about the process of writing a Packt book. Join the exploration of new topics by contributing your ideas and see them come to life in print.

1. A General Introduction to Debugging Software
2. Approaches to Kernel Debugging
3. Debug via Instrumentation printk and friends
4. Debug via Instrumentation Kprobes
5. Debugging Kernel Memory Issues Part 1
6. Debugging Kernel Memory Issues Part 2
7. Oops! Interpreting the kernel bug diagnostic
8. Lock Debugging
9. Tracing the Kernel Flow
10. Kernel Panic, Hangcheck, and Watchdogs
11. Using KGDB
12. Other Approaches to Kernel Debugging

Hello there! I welcome you on this, our journey, on learning how to go about debugging a really sophisticated, large, and complex piece of software thats proven absolutely critical to both enterprise business as well as tiny embedded systems and everything in between the Linux kernel, and, to some extent, learn to debug the Linux user space ecosystem as well.

Let's begin this very first chapter, and our journey on kernel (and user mode) debugging, by first understanding a little more on what a bug really is, and the origins and myths of the term debugging. Next, a glimpse at some actual real-world software bugs will (hopefully) provide the required inspiration and motivation (to firstly avoid bugs and then to find and fix your bugs, of course). You will be guided on how to setup an appropriate workspace to actually work on a custom kernel and debug issues, including setting up a full-fledged debug kernel. Well wrap up with some useful tips on debugging.

In this chapter were going to cover the following main topics:

• Software debugging what it is, origins, and myths
• Software bugs a few actual cases
• Setting up the workspace
• Debugging a few tips

You will require a modern and powerful desktop or laptop. We tend to use Ubuntu 20.04 LTS as the primary platform for this book. Ubuntu desktop specifies the recommended minimum system requirements (https://help.ubuntu.com/community/Installation/SystemRequirements) for the installation and usage of the distribution; do refer it to verify that your system (even a guest) is up to it.

The complete source code for this book is freely available on GitHub at https://github.com/PacktPublishing/Linux-Kernel-Debugging. You can work on it by cloning the Git tree using the following command:

The source code is organized chapter-wise. Each chapter is represented as a directory in the repository for example, ch1/ has the source code for this chapter. A detailed description on installing a viable system is covered in the Setting up the workspace section.

In the context of the software practitioner, a bug is a defect, an error, within the code. A key, and often large, part of our job as software developers is to hunt them down and fix them, so that, as far as is humanely possible, the software is defect-free and runs precisely as designed.

Of course, to fix a bug, you first have to find it. Indeed, with non-trivial bugs, its often the case that you arent even aware there is a bug (or several) until some event occurs to expose it! Shouldnt we have a disciplined approach to finding bugs before shipping the product or project? Of course we do its the Quality Assurance (QA) process, more commonly known as testing. Though glossed over at times, testing remains one of the if not the most important facets of the software lifecycle (would you voluntarily fly in a new aircraft thats never been tested? Well, unless youre the lucky test pilot...).

Okay, back to bugs; once identified (and filed), your job as a software developer is to now identify what exactly is causing it, what the actual underlying root cause is. A large portion of this book is devoted to tools, techniques, and just thinking about how to exactly do this. Once the root cause is identified, and you have clearly understood the underlying issue, you can, in all probability, fix it. Yay!

This process of identifying a bug using tools, techniques, some hard thinking to figure out its root cause and then fixing it, is subsumed into the word debugging. Without bothering to go into details, theres a popular story regarding the origin of the word debugging: on a Tuesday at Harvard University (on Sept 9, 1947), Admiral Grace Hoppers staff discovered a moth caught in a relay panel of a Mark II computer. As the system malfunctioned because of it, they removed the moth, thus