Subrata Banik - Firmware Development: A Guide to Specialized Systemic Knowledge

This Apress imprint is published by the registered company APress Media, LLC, part of Springer Nature.

The registered company address is: 1 New York Plaza, New York, NY 10004, U.S.A.

Nowadays, firmware is one of the most critical parts within every device and every security concept. Typically, x86 firmware is hidden from the user, and little to no interaction is needed. Firmware has been closed source for the last 20 years. Whereas there are some exceptions to the general rule, the core parts of firmware are and will most likely remain closed source in the near future. From an outside perspective, this is not logical. Today, most of the software stack running on x86 platforms can be open sourced, beginning from the bootloader through the operating system up to the application level. Large hyperscalers make open source a requirement for some parts, and consumers love the freedom that open source software provides to them. Firmware is the last bastion that has not fallen. However, in recent years, the open source firmware community has leaped forward; thus, the industry is changing. Hyperscalers adopt open source firmware as the de facto firmware standard, and customers can buy open source firmware-enabled products off the shelf.

The idea of open source is not to open up the existing code base. It is rather about giving the community the ability to develop and maintain the code themselves. Two things are needed to reach this goal: sharing knowledge and providing the technical documentation. Obviously, both influence each other. Technical documentation is something that hardware vendors, OEMs, and ODMs can provide and are responsible for. This documentation enables developers to write the actual code and thus produce and maintain open source firmware. It is impossible, or at least extremely complicated, to write proper firmware code without technical documentation. Technical documentation is not about opening up the intellectual property but rather providing a path to bring up and configure the hardware components.

In addition, the community needs knowledge and experienced people who share this knowledge. Sharing knowledge enables others to become firmware developers, to understand the concepts, and ultimately to grow the community. Firmware and its development are among the most complex software systems a developer can dive into. Even though most of the components themselves are not complicated, the interaction between them makes it complex. Keeping this technical documentation closed source makes the components (unnecessarily) complicated.

This book shares some of the missing pieces in system firmware development and guides the reader through various topics. We are engineers who are passionate about open source and driving this effort forward.

Firmware is the first piece of code that runs on the target hardware after the end user has turned on a device. Depending on the types of target hardware, the operations being performed by the firmware may differ, but the fundamental operations of firmware remain the same across the target hardware: performing the bare minimal hardware initialization and either waiting for host-centric communication to initiate or handing off control to the high-level system software that allows the end-user interaction. Based on the target market segment, an OS can have multiple virtual machines, and/or various types of applications are installed that satisfy the end-user needs. Although the control goes to the OS, an instance of the firmware is still alive and available to manage a few critical tasks that for OS-based applications or drivers cannot perform.

Over time, CPU architectures have gotten more complicated, and platform requirements have evolved. This has pushed the firmware boundary and caused firmware to extend its services too.

Back in the 1950s, the only possible way to instruct a computer system to perform some operation was using assembly language. The processors were simple enough, and hence the expectations from the firmware were also minimal. Porting to different kinds of CPUs involved redundant effort. In the 1970s, with the evolution of microprocessors, which demanded the enhancement of firmware features, all firmware development started migrating to the low-level system programming language C. Later C became the de facto standard for firmware development as its easily ported from one generation to the next. Since then, a different flavor of a C-based proprietary technology or framework evolved that made the hardware programming easier and created an abstraction such that its easy for programmers to contribute to the system programming without a deep understanding of the hardware. When technology evolves in an enclosed environment like this, it actually limits the spread of specialized knowledge and positively affects the entire ecosystem.

In the 2010s, access to the Internet became cheaper, and hence demands for personal computing devices were booming. With new sets of devices becoming popular, the ecosystem was changing. Statistics suggest that between 2000 and 2019 in the United States the number of Internet users tripled. Having more computing devices meant demand for advanced user experiences. For example, an instant system response refers to less device latency, trillions of data transfers in seconds requires secure systems, and more users means the backend server capacity needs to be enhanced. These are all the driving forces for the industry to look beyond the traditional system development model and demand more transparent development of the system, be it hardware, firmware, or software. With the openness in the firmware development approach, many product differentiating ideas are evolving and intercepting in firmware which are enough to challenge the existence of traditional System Firmware development approach. This demands a revamp of the firmware development model. We wrote this book to be a bridge from the present firmware development model to the future and make sure the readers are well equipped with such knowledge that makes them ready for such a migration in future.