Jira and Confluence has degraded performance - scoped to Brazil only

Incident Report for Jira

Postmortem

Summary

On February 10, 2025, between 16:10 and 18:35 UTC, Atlassian customers in Brazil experienced intermittent failures accessing and using Jira and Confluence Cloud. This disruption was due to our Content Delivery Network (CDN) not having sufficient capacity in Brazil under a certain network configuration.

Changes in the lead-up to the incident included enabling CDN for Jira and Confluence in Brazil between January 20th and February 6th to improve performance and security.

Jira and Confluence experience success rate monitoring detected the incident within 50 minutes. The incident was mitigated by temporarily disabling CDN in Brazil, which put Atlassian systems into a known good state. The total time to resolution was two hours and 25 minutes.

IMPACT

The overall impact was on February 10, 2025, between 16:10 and 18:35 UTC to Jira and Confluence Cloud. The incident caused service disruption to Brazilian customers when they attempted to access or use any feature of those products. Customers observed a generic error page or “HTTP 500” error presented by our CDN provider for up to a third of all requests at peak.

ROOT CAUSE

Our CDN is configured to present Jira and Confluence Cloud from static IP ranges dedicated to Atlassian.

This configuration limited the number of edge locations servicing customer requests, causing a high concentration of connections in the Brazilian region on one edge location in particular.

When daily traffic volumes in Brazil peaked on February 10, connections through the edge location to our servers encountered a capacity limit and were rejected with a HTTP 500 error.

REMEDIAL ACTION PLAN & NEXT STEPS

We understand that outages impact your productivity. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Along with our CDN provider, we have already taken the following actions to avoid repeating this type of incident:

  • Our CDN provider has completed the rollout of the static, dedicated IP feature to all their edge locations and enabled Jira and Confluence Cloud to be served from all of them, thus moving to the intended configuration that will provide full network capacity.
  • We are working with our CDN provider to perform more rigorous checks of capacity and configuration for Jira and Confluence Cloud.
  • Our CDN provider increased the server connection limit for Jira and Confluence Cloud in all locations, further increasing capacity.
  • Based on the observed failure patterns, we have created additional monitoring and alerting configurations, so we will be alerted immediately if the issue reoccurs.

Additionally, we are prioritizing actions to improve the detection of failures that affect a specific region.

Thanks,

Atlassian Customer Support

Posted Feb 26, 2025 - 02:12 UTC

Resolved

Between February 10, 2025, at 17:04 UTC and 18:32 UTC, some customers for Jira Work Management, Jira Service Management, Jira, Jira Product Discovery Cloud and Confluence experienced degraded performance and 500 errors.

We found that cloud products in Brazil encountered an internal limit with our cloud service provider. We redistributed traffic to mitigate this issue and are working with our cloud provider to address this issue in the future.

The issue has been resolved and all the services are operating normally.
Posted Feb 10, 2025 - 19:26 UTC

Update

We are investigating cases of degraded performance for Jira Work Management, Jira Service Management,Jira Cloud and confluence customers. We are currently investigating it and will provide more details within the next hour.
Posted Feb 10, 2025 - 18:07 UTC

Identified

We are investigating cases of degraded performance for Jira Work Management, Jira Service Management, and Jira Cloud customers. We are currently investigating it and will provide more details within the next 30 minutes.
Posted Feb 10, 2025 - 17:47 UTC
This incident affected: Viewing content, Create and edit, Authentication and User Management, Search, Notifications, Administration, Marketplace, Mobile, Purchasing & Licensing, Signup, and Automation for Jira.