Amazon packages pile up as AWS outage spawns delivery havoc
An Amazon Web Services outage wreaked havoc on the e-commerce giant’s delivery operation, preventing drivers from getting routes or packages and shutting down communication between Amazon and the thousands of drivers it relies on, according to four people familiar with the situation.
Three delivery service partners said an Amazon.com Inc. app used to communicate with delivery drivers was down. Vans that were supposed to be on the road delivering packages sat idle with no communication from the company, one person said. Amazon Flex drivers, independent delivery people who carry parcels in their own cars, couldn’t log in to Amazon’s app to get assignments, another person said.
Amazon said late Tuesday afternoon that much of its web service was working again. “We have mitigated the underlying issue that caused some network devices in the US-EAST-1 Region to be impaired,” the company said on its AWS dashboard just before 6 p.m. Eastern time. “We are seeing improvement in availability across most AWS services.” The company didn’t comment on the issues with its delivery operation.
The problems came amid Amazon’s crucial holiday shopping season, when the company can ill afford delays that could potentially create lasting logjams. One delivery business owner on the West Coast said the company halted deliveries Tuesday and planned to regroup Wednesday.
L.A. Pizza Hut customers are being hit with an extra charge to help recover “the increased cost of operations in the state of California.”
Two delivery partners in earlier time zones said drivers who already had routes were instructed to put their phones in airplane mode and not log out of the Amazon routing app so they could continue making stops, but drivers who hadn’t already been assigned routes were sidelined.
The outage began at about 10 a.m. Eastern time, according to DownDetector. At the height of the outage, the web monitoring site reported more than 20,000 complaints for Amazon and more than 11,000 for the company’s cloud computing arm, Amazon Web Services. By 1:45 p.m., the reported outages had declined by about half for AWS and two-thirds for Amazon.
Multiple popular websites were also affected, including those operated by Coinbase, Robinhood, Disney and Netflix, according to DownDetector. Disney said that although people were able to get into its theme parks, they were having difficulty checking in online and paying for purchases. Webcast presentations from Comcast and Altice USA at UBS’ Global TMT Conference experienced disruptions earlier Tuesday, and the Charter Communications presentation was rescheduled for Wednesday.
AB 701, headed to a Senate vote this week, is the first legislation in the U.S. that would regulate warehouse performance metrics.
Some Amazon services, including music and video streaming, the voice-activated Alexa platform and security arm Ring were affected, too.
Just after 1 p.m. Eastern time, Amazon said it had identified the root cause of the issue and was working to fix it. Meanwhile, the company directed customers to alternative servers in its western region that weren’t experiencing problems. The increased errors were in the eastern North American region.
Amazon, in an update on its AWS dashboard, said it still hadn’t completely fixed the issue by 6 p.m. Eastern time, although “many services have already recovered.” Multiple Amazon cloud-computing services had been affected, including Amazon DynamoDB and Amazon Elastic Compute.
Video streaming service Netflix experienced a 26% drop in traffic after the AWS problems were reported, showing how quickly outages can ripple outward, said Doug Madory, an analyst at the network monitoring firm Kentik in San Francisco. “It gets more and more complicated with software running these services, so when something goes sideways it can take a long time to figure out what went wrong and fix it,” he said. “Complexity has risks. You introduce unknown errors.”
AWS is the leading cloud-computing provider, enabling companies to purchase computing power and software services on demand rather than maintaining their own data centers and teams in-house. Its customers include a wide range of industries and the federal government.
Bloomberg writers Scott Moritz and Christopher Palmeri contributed to this report.
Inside the business of entertainment
The Wide Shot brings you news, analysis and insights on everything from streaming wars to production — and what it all means for the future.
You may occasionally receive promotional content from the Los Angeles Times.
