On August 2nd, 2021 between 11:00 am and 01:30 pm UTC some customers on Atlassian’s Cloud Platform were unable to use Jira Software, Jira Service Management and Jira Work Management. The event was triggered by the incapacity of scaling our infrastructure to support the traffic. We exhausted some network resources in the US-east region which prevented the scale-out of the stack. The incident was detected within 2 minutes by Atlassian's staff and mitigated by cleaning resources manually which put Atlassian systems into a known good state. The total time to resolution was about 2 hours & 30 minutes.
The overall impact was between August 2nd, 2021 11:00 am, and 01:30 pm UTC. It mainly affected Jira Software, Jira Service Management, and Jira Work Management. However, the applications that relied on accessing the data from those products were not able to display that data (Confluence, Compass, Team Central, Bitbucket, Opsgenie). The incident caused service disruption to the customers in the US-east region. It translated into high latency when trying to perform an action within Jira or a complete outage.
The issue was caused by an emergency deployment attempt to the US-east region during irregular deployment hours that used up all the subnet IP resources that we configured. As a result, we could not spin up enough nodes to handle the load. This happened because, during this emergency deployment, we requested more resources than we had available to handle all the traffic.
We know that outages impact your productivity. We deploy our changes progressively (by cloud region) to avoid broad impact. However, in this case, the progressive deployment rollout to a specific cloud region (US-east) didn't scale as expected. Moving forward, to minimize the impact of deploying new changes to our environments, we will implement preventative measures such as the ones listed below.
We apologize to those customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability.
Atlassian Customer Support