The snark of the following comment is not directed towards you, OP, but at the tech industry at large.
What I don’t understand is why people are still surprised when this shit happens. Today, cloudflare takes down half the internet, last month it was AWS. Crowdstrike did it last year even more severely. Akamai has also caused major issues like this before, as has Google. M365/azure outages barely get reported on because they are so frequent. Yet, they are all still being used to hold up most of our infrastructure. Every single company I’ve done IT for has used at least one of these companies for critical infrastructure. There just aren’t any other realistic options due to the refusal of non IT people to learn about IT.
If you try to use something other than one of the big companies, you’re hit with one or more roadblocks.
You “don’t have the budget” to selfhost. Bean counters would rather pay $100 a month indefinitely than $5k to buy new hardware that will save $1000 a month for years.
No approval for non giant corpo option, because using AWS is cheaper and has brand recognition. This is due to the same economics and myopia that caused Walmart to be one of the only places you can get groceries.
There is no other option. Every year that goes by, more small companies get gobbled up by big tech M&A. Unless your company opts to create its own implementation of a service/software, you’re stuck with one of only a few options, even if you could get the approval to use something not run on big tech.
Even if you manage to jump all of the previous hurdles, the Internet connected software you’re using probably relies on big tech infrastructure too. Every company has to navigate all of these hurdles for every saas/infrastructure implementation, and the only ones that successfully do it have to have leadership that not only understands why the decisions have to be made, but also need to be willing to accept the extra cost. Anyone that has dealt with upper management knows that this is exceptionally rare.
So what we are left with is a system that every professional knows is deeply broken and monopolized. The people that actually make the final decisions are largely ignorant and unwilling to invest money in fixing it, instead choosing short term savings and lack of commitment over long term security and continuity.
I hear where you are coming from, but I think your criticisms are misdirected. For the majority of businesses, using an infrastructure provider is a sensible decision that leads to greater security and stability in the long run for less money than trying to build the same thing on their own. This isn’t a decision made out of stubbornness, laziness, or ignorance about IT. It’s simply that it’s the better option for each individual business.
But when most companies make the decision to use an infrastructure provider, outages and risks are centralized. As you pointed out, the services you rely on are likely to use a provider even if you don’t use one, so this isn’t a problem that a business can solve by buying a server and hiring an IT team. These massive failures aren’t a sign that businesses need to make different decisions. It’s a sign that the infrastructure providers must work harder and spend more money to improve their internal isolation.
When a bridge collapses because the pedestrians happen to walk in step with the resonant frequency of the bridge, we don’t blame the pedestrians for walking incorrectly or for deciding to take the bridge instead of a boat. We blame the designer of the bridge for failing to account for the mundane stresses that the bridge is expected to sustain.
I mean, companies avoiding self hosting isn’t just about being cheap. Cloudflare/AWS might cost $100 per mo and only have 95% uptime but you know what you’re getting. Self hosting inherently introduces risk.
That 5k machine might pay for itself in half a year OR it might self destruct in 3 months. The man hours and downtime needed to unfuck that mess might cost more than multiple years of flaky cloud hosting. Alternatively, a change in data retention regulation requires hardware redundency, then next month the revenue stream from that hardware drys up and you’re stuck holding a $10k loss instead of canceling a $100 payment.
The snark of the following comment is not directed towards you, OP, but at the tech industry at large.
What I don’t understand is why people are still surprised when this shit happens. Today, cloudflare takes down half the internet, last month it was AWS. Crowdstrike did it last year even more severely. Akamai has also caused major issues like this before, as has Google. M365/azure outages barely get reported on because they are so frequent. Yet, they are all still being used to hold up most of our infrastructure. Every single company I’ve done IT for has used at least one of these companies for critical infrastructure. There just aren’t any other realistic options due to the refusal of non IT people to learn about IT.
If you try to use something other than one of the big companies, you’re hit with one or more roadblocks.
You “don’t have the budget” to selfhost. Bean counters would rather pay $100 a month indefinitely than $5k to buy new hardware that will save $1000 a month for years.
No approval for non giant corpo option, because using AWS is cheaper and has brand recognition. This is due to the same economics and myopia that caused Walmart to be one of the only places you can get groceries.
There is no other option. Every year that goes by, more small companies get gobbled up by big tech M&A. Unless your company opts to create its own implementation of a service/software, you’re stuck with one of only a few options, even if you could get the approval to use something not run on big tech.
Even if you manage to jump all of the previous hurdles, the Internet connected software you’re using probably relies on big tech infrastructure too. Every company has to navigate all of these hurdles for every saas/infrastructure implementation, and the only ones that successfully do it have to have leadership that not only understands why the decisions have to be made, but also need to be willing to accept the extra cost. Anyone that has dealt with upper management knows that this is exceptionally rare.
So what we are left with is a system that every professional knows is deeply broken and monopolized. The people that actually make the final decisions are largely ignorant and unwilling to invest money in fixing it, instead choosing short term savings and lack of commitment over long term security and continuity.
I hear where you are coming from, but I think your criticisms are misdirected. For the majority of businesses, using an infrastructure provider is a sensible decision that leads to greater security and stability in the long run for less money than trying to build the same thing on their own. This isn’t a decision made out of stubbornness, laziness, or ignorance about IT. It’s simply that it’s the better option for each individual business.
But when most companies make the decision to use an infrastructure provider, outages and risks are centralized. As you pointed out, the services you rely on are likely to use a provider even if you don’t use one, so this isn’t a problem that a business can solve by buying a server and hiring an IT team. These massive failures aren’t a sign that businesses need to make different decisions. It’s a sign that the infrastructure providers must work harder and spend more money to improve their internal isolation.
When a bridge collapses because the pedestrians happen to walk in step with the resonant frequency of the bridge, we don’t blame the pedestrians for walking incorrectly or for deciding to take the bridge instead of a boat. We blame the designer of the bridge for failing to account for the mundane stresses that the bridge is expected to sustain.
For a lot of people who would self host, $100 at a time is easier to get together than a few thousand at once.
Not talking about individuals, homie. Talking about companies.
I mean, companies avoiding self hosting isn’t just about being cheap. Cloudflare/AWS might cost $100 per mo and only have 95% uptime but you know what you’re getting. Self hosting inherently introduces risk.
That 5k machine might pay for itself in half a year OR it might self destruct in 3 months. The man hours and downtime needed to unfuck that mess might cost more than multiple years of flaky cloud hosting. Alternatively, a change in data retention regulation requires hardware redundency, then next month the revenue stream from that hardware drys up and you’re stuck holding a $10k loss instead of canceling a $100 payment.