Unmesh Joshi - Patterns of Distributed Systems

We live in a digital world. Most of what we do is available over the network as a service. Be it ordering our favourate food or managing our finances. All these services run on some servers somewhere. These servers store data and do computations on that data handling user requests over the network. Servers typically wait for user requests, then they read the data stored on the disk into memory and process them using the CPU. CPU, Memory, Network and Disks are the four fundamental physical resources which any computation needs.

Consider a typical retail application exposed as a networked service. Users can add items to the shopping cart and buy them. Users also view their order, and can query their past orders. How many user requests will a single server be able to process? There are many factors based on the specific type of the application, but the upper bound will always be determined by the capacity of these four resources.

Lets start with the network. The network bandwidth decides the maximum limit on how much data can be transferred over the network at any given time. Consider a network bandwidth of 1Gbps. If the application is writing or reading 1KB records, the maximum number of requests that the network can support is 125000. If the records are 5KB in size, the number of requests which can be passed over the network is 25000.

The disks have a limit on the amount of data they can transfer. Following are some example numbers with typical disk bandwidths.

This is a raw hardware limitation. But in practice, there is some software component which handles the writes and reads. The software component needs to handle issues like concurrent read/writes or transactions, which further limits the number of read/write requests which can be processed on a single server.

Disk and Network are the input/output devices. But to do any work with data read from these needs CPU cycles to be consumed. So with 100K requests if the CPU is at 100% of the capacity, any more requests will be waiting to be processed for their share of CPU.

The fourth factor is that of memory. Servers load data in memory to process it. The data from requests over the network are loaded in memory to further process it. The data from storage is also loaded in memory to process it. Lets say there is 1TB of storage, the memory will typically be in GBs, so at a time only part of the data can be loaded in memory. The whole purpose of different kinds of storage engines is to reduce the need to load all the data in the memory to process. Storage engines use different data structures and maintain indexes to quickly locate the specific data items on the disk and pick and load only those for processing. But depending on the type of the request, more data might be loaded in memory. For example, if all the users are searching for books they ordered in the last year, they will need to scan through more and more data, needing more memory. If the memory is full, again the requests need to wait for their share.

One common problem when these resources reach their physical limit is that of queuing. More requests will need to wait to be processed. This in effect has an adverse effect on the servers ability to process user requests.

The disk, network, cpu and memory put upper bound on the number of requests which can be processed. If the number of requests the application needs to process goes above this upper bound, the requests start getting queued for their share of network, disk, cpu or memory. The queuing then increases the time it takes to process any requests.

The effect of reaching the resource limits is observed on overall throughput of the system as following.

This is problematic to the end users. Because when they expect the system to serve more and more users, the system actually performance starts degrading.

The only way to make sure that the requests can be served properly is to divide and process them on multiple servers. This allows using physically separated CPUs, network, memory and disks for processing user requests. In the above example, the workload needs to be partitioned in such a way that each server serves about 500 requests.

One common way to divide the architectures is as following. The architecture has two parts, a stateless part exposing functionality to the end user. This can be a web application or more commonly a web api serving user facing applications. The second part, the stateful part, is managed by a database. When user load increases the stateless services are scaled horizontally. This allows serving more and more user requests connections. The business logic executed on the data is done separately on different servers

This architecture works fine provided two basic assumptions hold true.

The database can serve a request from the stateless services in less than a few milliseconds.

The database can handle all the connections from multiple stateless service instances. This applications typically work around this constraint by adding caching layers to make sure that not all the request need to go to the database.

This architecture works very well, if most of the users can be served from caches put at different layers in the architecture. It makes sure that, out of all the requests, only a few requests need to reach the database layer. As nicely stated by Roy Fielding in his thesis on REST, "best application performance is obtained by not using the network". But caching does not work always. When most requests are writing data there is obviously no use of caching. With hyper personalized applications needing to always show latest information, the use of caching is limited. As more and more users start using the services, the assumptions start breaking down. It is caused by two reasons.

The size of the data grows, from few terabytes to several hundred of terabytes to petabytes

More and more people need to access and process that data

So the simple architecture shown above starts breaking down. The reasons it starts breaking down is again because of the physical limits of the four fundamental resources.

The impact on the classic architecture then looks as following:

One way to work around these limits is to partition the application following the domain boundaries. The popular architectural style today is that of Microservices .