Sometimes, a seemingly simple and obvious solution can lead to a series of problems later on. This is especially true when adding retries.

I’ve never run a big system like this, but like the lead character in the story, I always figured exponential backoff would be enough. Turns out there’s more.