From 9pm UTC on October 23, 2023, until 3am UTC on October 25, 2023, Atlassian customers using the Jira family products that have configured notifications from custom email domains experienced a degradation as email notifications were being sent from the default atlassian.net domain instead of a custom domain. The incident was detected by Atlassian customers as they started to receive notifications from the atlassian.net domain instead of previously configured custom domains. It was mitigated by excluding redundant validations from the custom domain verification job which put Atlassian systems into a known good state. The total time to resolution was about 30 hours.
Note: the initially published timeframe of the incident was incorrect and was adjusted together with time to resolution.
The overall impact was between October 23, 2023 9pm UTC and October 25 3am UTC on Jira family products, including Jira Service Management, Jira Software and Jira Work Management. The incident caused service disruption to customers who have notifications from custom email domains enabled for their Jira products as email notifications were being sent from the default atlassian.net domain instead of the custom domain configured by the customer.
The issue was caused by a change to the service responsible for custom domain email notifications lifecycle that inadvertently enabled SPF record validation as part of a daily validation job. As a result, previously working custom domain configuration transitioned to an unverified state for some Atlassian customers, resulting in notifications from Jira products being sent from atlassian.net domain instead of a previously configured custom domain.
We are continuously improving our system's resiliency. We are prioritizing the following improvement actions to avoid repeating this type of incident:
We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability.
Thanks,
Atlassian Customer Support