On December 8, 2021, between 02:45 and 03:54 UTC, Atlassian customers using Jira and Confluence Cloud products were unable to access core functionality of the products. This event was triggered by a change to replace the Transport Layer Security (TLS) certificate for *.atlassian.net on Atlassian's Edge Network Infrastructure. The incident was detected by monitoring and mitigated by rolling back to the previous TLS certificate. The total time to resolution was one hour and nine minutes.
Jira and Confluence Cloud experienced service degradation on December 8, 2021, between 02:45 and 03:54 UTC. The incident caused service disruption to customers globally, and was caused by a failed change to renew the *.atlassian.net TLS certificate. As a result, API based clients could not perform TLS handshakes with the Atlassian Edge Network Infrastructure, and clients received HTTP 5xx errors which resulted in the following failure scenarios:
Our mechanism for replacing TLS certificate failed to validate the certificate chain. We deployed the certificate with the incorrect certificate chain to Atlassian's Edge Network Infrastructure, resulting in failed TLS handshakes with *.atlassian.net services, and therefore, Jira and Confluence Cloud were unable to service public customer requests.
We know that outages impact your productivity. While we have a number of tests and preventative processes in place, this specific issue wasn’t identified because the change was related to a very specific set of HTTP clients. This was not picked up by our automated continuous deployment suites and manual test scripts.
We are prioritizing the following improvement actions to avoid repeating this type of incident:
We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve Jira and Confluence Cloud's performance and availability.
Thanks,
Atlassian Customer Support