API call got failed....? still you want to re-try

Clients retry the APIs to ensure completion when the network is unstable. When there are fewer clients, this strategy is effective. But what happens when there are millions of clients?

This is known as the Thundering Herd problem, Let's see together what it is, why it happens, and how to fix it.

Look at an example-

A user named Amit attempted to make an API connection to the backend, but the call was unsuccessful due to network problems.

Now the question is What to do now..? Shall we re-try, Okay cool let's see. The simplest way of doing it is.

for i=0; i<3;i++
doit[]

//assuing re-try limit is 3
 sleep = min(cap, base * 2 ** attempt)

Yes, definitely i am thinking the same as you....

A reasonable solution, although it struggles on a larger scale. Imagine that the server is overwhelmed because an API call was unsuccessful, and every connected client is retrying.

This will create server to NO ROOM TO RECOVER because of:

new requests coming in...
More and more requests because of re-try.

Then Giant question is How to Solve This...?

Exponential Backoff

--> An exponential backoff method puts some time interval between retries exponentially, up to a maximum backoff time. To say it simply, we attempt again with a time interval rather than immediately repeating what we just did.

Let's understand with an exmaple.

This provides the above mechanism's participants with some breathing room and much-needed recovery time.

However, the retries will still happen at the same time, which will put more strain on the server.

If the internet were to fall down, you might have witnessed this on Gmail.

Shall we look for bit more cleaver way to solve this, cool here the same solution comes with Jitter

Exponential Backoff with Jitter.

The solution isn’t to remove backoff. It’s to add jitter. Initially, jitter may appear to be a counter-intuitive idea: trying to improve the performance of a system by adding randomness. The time series above makes a great case for jitter – we want to spread out the spikes to an approximately constant rate. Adding jitter is a small change to the sleep function:

sleep = random_between(0, min(cap, base * 2 ** attempt))

Let's draw the diagram again.

This guarantees that retries are evenly distributed and don't worsen the issue. A few things must be ensured when using retries.

You add random Jitter
Retries are spaced out exponentially.

That's the warp buddy... 💡