On Monday, an issue with Amazon's computational infrastructure caused an outage that went way beyond impacting the internet giant's online shopping website.
The AWS outage had wide-ranging global impacts, affecting some healthcare services, banks and air travel as well as disrupting popular apps such as Snapchat and Reddit.
So what caused the outage, and why was it such a big deal?
What is AWS?
AWS stands for Amazon Web Services, which is a subsidiary of Amazon that offers speciality technology services to clients across the globe.
"It's one of the world's largest cloud computing platforms," University of Queensland's Cyber Research Centre director Ryan Ko says.
"Instead of every company owning and maintaining their own physical computer servers, AWS lets them rent computing power, storage and databases from Amazon's data centres around the world.
"Think of it like electricity: most businesses don't run their own power plants; they just plug into the grid."
Why was the AWS outage so disruptive?
Because so many services use AWS.
"Millions of apps, websites, and services — from banks to streaming platforms and even government systems — rely on AWS to operate smoothly and scale up quickly," Professor Ko says.
But it wasn't just direct clients of AWS that were impacted.
That's because AWS also offers other services, such as support for something called a "software as a service" (SaaS).
"The client might need to run some software [such as] a HR system or payment software and pays a provider to use this software," says David Parry from Murdoch University's School of Information Technology.
"The provider might be AWS or might be another organisation that then buys capacity on AWS to run the software.
"It's a bit like when you buy power from your electricity company, which in turn has contracts with power stations or even people with solar panels feeding into the grid.
"It's got a very wide impact because not only does this happen on a large scale, but there are also small pieces of software that do something useful that lots of systems use."
An example of this is when a shopping website uses a piece of software they've contracted from another company to take your payment details.
"It's often much easier to use these components than build them yourself, and these often run on the AWS system," Professor Parry says.
What was the problem?
AWS said it "identified the trigger of the event as DNS resolution issues for the regional DynamoDB service endpoints".
DNS stands for Domain Name System, which is like the internet's phone book, Professor Ko says.
"Almost everything in AWS [and the internet in general] depends on DNS," he says.
"When it fails, it's like every taxi driver suddenly losing access to Google Maps.
"They may not be able to find their destinations, even though the roads are still there.
"This small but critical piece of infrastructure connects thousands of services together."
AWS's DynamoDB is a cloud database service.
"[It] was affected because the system that tells computers where DynamoDB lives on the network stopped working properly," Professor Ko says.
"So even though the servers were up, many systems couldn't find their way there."
Where was the issue detected?
The issue originated in Amazon's US-East-1 region data cluster in the US state of Virginia.
Notice we say "region"?
That's because the servers aren't located in just one building.
Amazon has "well over 100" sprawling computing warehouses in Virginia, the Associated Press reports.
"That region is one of AWS's oldest and most central hubs — it hosts critical parts of their network and management systems," Professor Ko says.
"[It is] likely that the DNS infrastructure for multiple regions is logically centralised there.
"So, when something goes wrong in that region, the impact is global."
What caused the outage?
We still don't know the root cause of the DNS issues that sparked the outage.
"That's not unusual as big tech companies often take days or weeks to release full post-incident reports," Professor Ko says.
"Sometimes they keep the deeper technical details internal for security and commercial reasons."
What could have caused the outage?
The DNS issue could have been caused by a whole range of things, but we can rule out a few potential causes.
AWS says it wasn't the result of a malicious attack, and Professor Parry says it's unlikely to be a capacity issue.
And it's probably not the result of something impacting the physical building that houses the servers.
"I'd say it's more likely to be a software issue, although people do drive bulldozers through cables," Professor Parry says.
Professor Ko says it might have been a software bug or configuration error, "where a change in one part or a software update of the system breaks something elsewhere".
Or it could have been sparked by an overloaded or failing network component.
What are the lasting impacts of the outage for AWS?
"It's highly unlikely that we'll see a mass exodus from AWS," University of Sydney Business School's Shumi Akhtar says.
"Moving your entire digital operation from one cloud provider to another is an enormous, expensive and risky undertaking.
"Furthermore, all major cloud providers — including Microsoft and Google — have had significant outages."
However, Dr Akhtar reckons this will serve as a "wake-up call".
Dr Akhtar says AWS may now be focusing on ways an outage of this scale could have been prevented.
"The goal in designing these massive systems is to limit the 'blast radius' of any single failure," she says.
How could AWS prevent such a big outage?
It's hard to say for sure without knowing the cause behind the outage.
However, there are few options, Monash University software systems and cybersecurity lecturer Mohammad Goudarzi says.
In a nutshell, it could be about spreading resources across locations.
That might mean having extra DNS servers and services as backups to primary servers or balancing the load across locations to reduce the impost on main servers in the one region.
"Many essential AWS services currently operate within the US-EAST-1 region, which previously also experienced outages leading to significant service disruptions," Dr Goudarzi says.
"Such incidents highlight the risks of regional concentration.
"Therefore, AWS and organisations relying on its infrastructure should deploy critical services across multiple regions to ensure redundancy and minimise the potential for widespread failures."
What should governments be doing about this?
"Technology service providers should have an appropriate level of insurance so that, in the event of an outage, customers/victims of outages are properly compensated," Edith Cowan University computing and security lecturer Mohiuddin Ahmed says.
"The Australian government should consider this seriously to ensure its citizens' rights in the increasingly connected online world."
"Governments should also require transparency and resilience standards from major cloud providers," Professor Ko says.
"For long-term resiliency, the Australian government could support the development of sovereign or hybrid cloud capabilities to reduce dependency on any single overseas vendor."
Do you know more, or are you affected?
