Rick Rackow - Operating OpenShift

The goal of having an upper limit for toil is to avoid shifting back into an operations team where people spend most of the time working down toil that accumulates with both the scale of service adoption and the advancement of the software.

A part of the accumulating toil while the service adoption grows is the number of alerts an operations team gets if the alerting strategy isnt ready for scaling.If youre maintaining software that creates one alert per day per tenant, keeping one engineer busy running 10 tenants, you will need to scale the number of on-call engineers linearly with the number of tenants the team operates.That means in order to double the number of tenants, you need to double the number of engineers dedicated to reacting to alerts.These engineers will effectively not be able to work on reducing the toil created by the alerts while working down the toil and investigating the issues.

In a traditional operations team that runs OpenShift as a development platform for other organizations of the company, onboarding new tenants often is a manual task.It may be initiated by the requesting team to open a ticket that asks for a new OpenShift cluster.Someone from the operations team will pick up the ticket and start creating the requirements, kick off the installer, configure the cluster so the requesting team gets access, etc.A similar process may be set up for turning down clusters when they are not needed anymore.Managing the lifecycle of OpenShift clusters can be a huge source of toil, and as you can see it will naturally scale with the adoption of the service.

In addition to being a toil-packed process, manual lifecycle and configuration management are error-prone.When an engineer runs the same procedure several times during the week, as documented in a team-managed Wiki, chances are they miss an important step or pass a wrong parameter to any of the scripts, resulting in a broken state that may not be discovered immediately.

When managing multiple OpenShift clusters, having one that is slightly different from the others due to a mistake in the provisioning or configuration process, or even due to a customer request is dangerous and usually generates more toil.Automation that the team generated over time may not be tailored to the specifics of a single snowflake cluster.Running that automation may just not be possible, causing more toil for the operations team.In the worst case, it may even render the cluster unusable.

Automation in a traditional ops team can often be found in a central repository, that can be checked out on engineer devices so they can run the scripts they need as part of working on a documented process.This is problematic not only because it still needs manual interaction and hence doesnt scale well, but also do engineer devices often be configured differently.They can differ in the OS they use, adding the need to support different vendors in the tooling, e.g. by providing a standardized environment like a container environment to run the automation.