SUMMARY

On 23 June, 2022, between 07:00 UTC and 07:45 UTC, some of our customers using Jira Service Management, Jira Software & Jira Work Management were seeing duplicate issues and comments being created by the Incoming Email Handlers feature in Jira. Incoming email handlers are configured in Jira to create issues or comments in any Jira project from emails via a cron job. Handlers are mapped with POP, IMAP and default mail servers. This incident only affected handlers mapped to IMAP mailboxes.

The incident was triggered by the introduction of a new state in emails' read status (“seen”). The Incoming Email Handlers started to process all emails that were not “deleted” or “read”, including emails marked in a “seen” state. This created duplicate issues and comments. 

The incident was detected by customers and our on-call engineers responded approximately 4 hours after impact began. The team was able to identify the root cause and verified that there was no risk of ongoing impact. Over the following 7 days, our engineers monitored the situation and worked to help affected customers identify and action their duplicate issues and comments.

IMPACT

Incoming mail handlers which were configured with IMAP mail servers reprocessed emails for the last 7 days that had been marked as "seen" but not "deleted".

The overall impact was between 23 June, 2022, 07:00 UTC and 30 June, 2022, 04:17 UTC. The incident did not cause service disruption in any region, for any customers. There was an impact to the expected behaviour of Incoming Email Handlers. Specifically, the features below were impacted during this time:

Affected Products:

• Jira Service Management
• Jira Software
• Jira Work Management

Impacted Features:

• Global email handlers were impacted.
• Incoming emails were processed twice as issues.
• Incoming emails were processed twice as comments.

Project level email handlers were NOT impacted in Jira Service Management.

ROOT CAUSE

While rolling back a feature that impacted the processing of attachments in incoming emails, the email-handling cron processed the mails from customers’ inboxes that had been marked as “seen” over the last 7 days, followed by marking them as “deleted”.

Global Jira Incoming Mail Handlers configured for IMAP marked the mails as “deleted” after they had been processed. The Jira team was working on a functionality change to start marking mail processed by the Global Jira Incoming Mail Handlers as “seen” instead of “deleted”. This would allow the customers to configure IMAP connections for mail processors other than Jira as well. Due to a bug in one of the open-source libraries used by Jira to process emails, we started to notice that attachments from these emails stopped getting processed.

To mitigate the issue, it was decided to rollback the change in functionality until the attachments issue was resolved. Unfortunately, the rollback triggered the email-handling cron to reprocess the mails from customers' inboxes that had been marked as “seen” over the last 7 days, followed by marking them as “deleted”. Since this processing is not idempotent (such that a mail with a unique identifier is processed only once), we ended up creating duplicate issues and comments for all such mails processed over the last 7 days from the feature rollback. We could not immediately delete all these duplicated issues because we are not allowed to modify customer data without explicit consent and customers could have configured automation, webhooks or workflows for issue-deletion that could potentially create an unreliable or inconsistent state for the customers.

We decided to proactively create support tickets for all customers over the next few days to communicate the situation, presenting mitigation options such as:

• sharing a JQL to query all the potentially duplicate issues across all the affected projects in JSW, JSM or JWM
• a script to label the duplicated issues with "Duplicated-issues-#-Atlassian" that were created in their instance
• steps for deleting issues in bulk

REMEDIAL ACTIONS PLAN & NEXT STEPS

We apologise to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s reliability.

While we have a number of tests in place, we have identified multiple things we can do to prevent duplication of issues/comments creation via incoming emails. We are prioritising the following improvement actions to avoid repeating this type of incident:

• Improvement to the Quality Assurance process while rolling back changes on production instances.
• Improve monitoring of alerts for the duplication of issues and comments for automatic detection of anomalies in the Email handling process.
• Make mail processing by incoming mails Idempotent. It should not process a mail with the same identifier ( messageId ) again. In this case, even if some other issues triggers the reprocessing of the email, duplication of issue or comments will be avoided.

Thanks,

Atlassian Customer Support