Luca Palmieri - Zero to Production in Rust: An Opinionated Introduction to Backend Development

Luca Palmieri

The world of backend development is vast.

The socio-technical context you operate into has a huge impact on the optimal tools and practices to tackle the problem you are working on.

For example, trunk-based development works extremely well to write software that is continuously deployed in a Cloud environment.
The very same approach might fit poorly the business model and the challenges faced by a team that sells software that is hosted and run on-premise by their customers - they are more likely to benefit from a Gitflow approach.
If you are working alone, you can just push straight to main.

There are few absolutes in the field of software development and I feel its beneficial to clarify your point of view when evaluating the pros and cons of any technique or approach.

Zero To Production will focus on the challenges of writing Cloud-native applications in a team of four or five engineers with different levels of experience and proficiency.

Defining what Cloud-native application means is, by itself, a topic for a whole new book. Instead of prescribing what Cloud-native applications should look like, we can lay down what we expect them to do.
Paraphrasing Cornelia Davis, we expect Cloud-native applications:

• To achieve high-availability while running in fault-prone environments;
• To allow us to continuously release new versions with zero downtime;
• To handle dynamic workloads (e.g. request volumes).

These requirements have a deep impact on the viable solution space for the architecture of our software.

High availability implies that our application should be able to serve requests with no downtime even if one or more of our machines suddenly starts failing (a common occurrence in a Cloud environment). This forces our application to be distributed - there should be multiple instances of it running on multiple machines.
The same is true if we want to be able to handle dynamic workloads - we should be able to measure if our system is under load and throw more compute at the problem by spinning up new instances of the application. This also requires our infrastructure to be elastic to avoid overprovisioning and its associated costs.
Running a replicated application influences our approach to data persistence - we will avoid using the local filesystem as our primary storage solution, relying instead on databases for our persistence needs.

Zero To Production will thus extensively cover topics that might seem tangential to pure backend application development. But Cloud-native software is all about rainbows and DevOps, therefore we will be spending plenty of time on topics traditionally associated with the craft of operating systems.

We will cover how to instrument your Rust application to collect logs, traces and metrics to be able to observe our system.

We will cover how to set up and evolve your database schema via migrations.

We will cover all the material required to use Rust to tackle both day one and day two concerns of a Cloud-native API.

The impact of those three requirements goes beyond the technical characteristics of our system: it influences how we build our software.

To be able to quickly release a new version of our application to our users we need to be sure that our application works.
If you are working on a solo project you can rely on your thorough understanding of the whole system: you wrote it, it might be small enough to fit entirely in your head at any point in time.
If you are working in a team on a commercial project, you will be very often working on code that was neither written or reviewed by you. The original authors might not be around anymore.

You will end up being paralysed by fear every time you are about to introduce changes if you are relying on your comprehensive understanding of what the code does to prevent it from breaking.

You want automated tests.
Running on every commit. On every branch. Keeping main healthy.

You want to leverage the type system to make undesirable states difficult or impossible to represent.

You want to use every tool at your disposal to empower each member of the team to evolve that piece of software. To contribute fully to the development process even if they might not be as experienced as you or equally familiar with the codebase or the technologies you are using.

Zero To Production will therefore put a strong emphasis on test-driven development and continuous integration from the get-go - we will have a CI pipeline set up before we even have a web server up and running!

We will be covering techniques such as black-box testing for APIs and HTTP mocking - not wildly popular or well documented in the Rust community yet extremely powerful.

We will also borrow terminology and techniques from the Domain Driven Design world, combining them with type-driven design to ensure the correctness of our systems.

Our main focus is enterprise software: correct code which is expressive enough to model the domain and supple enough to support its evolution over time.

We will thus have a bias for boring and correct solutions, even if they incur a performance overhead that could be optimised away with a more careful and chiseled approach.
Get it to run first, optimise it later (if needed).

The initial response to a random tweet about the idea of spinning up this project has been overwhelming.
Hundreds of subscribers to a rushed email newsletter.
Several emails and private messages detailing this or that specific setup and the challenges they are currently facing.

I have written and re-written the table of contents more than a couple of times trying to zero in on what I want to cover. I realised by the second version of it that I cant satisfy many of those emails and DMs.

The Rust ecosystem has had a remarkable focus on smashing adoption barriers with amazing material geared towards beginners and newcomers, a relentless effort that goes from documentation to the continuous polishing of the compiler diagnostics.
There is value in serving the largest possible audience.
At the same time, trying to always speak to everybody can have harmful side-effects: material that would be relevant to intermediate and advanced users but definitely too much too soon for beginners ends up being neglected.

I struggled with it first-hand when I started to play around with async/await.
There was a significant gap between the knowledge I needed to be productive and the knowledge I had built reading The Rust Book or working in the Rust numerical ecosystem.
I wanted to get an answer to a straight-forward question: > Can Rust be a productive language for API development?

Yes.

But it can take some time to figure out how.

Thats why I am writing this book.

I am writing this book for the seasoned backend developers who have read The Rust Book and are now trying to port over a couple of simple systems.

I am writing this book for the new engineers on my team, a trail to help them make sense of the codebases they will contribute to over the coming weeks and months.