Forge applications installations are never finishing - stuck in installing state
Incident
Report for Jira Software
Resolved
Installation metrics have remained stable, so we are now treating this issue as resolved.
The cause appears to be that a group of worker nodes were processing fewer tasks than expected concurrently, so Forge installations became blocked behind a set of long-running jobs.
We will continue confirming the root cause and hardening the worker nodes in question as post-incident review actions.
The cause appears to be that a group of worker nodes were processing fewer tasks than expected concurrently, so Forge installations became blocked behind a set of long-running jobs.
We will continue confirming the root cause and hardening the worker nodes in question as post-incident review actions.
Monitoring
We have identified faulty instances in our infrastructure and replaced them.
Based on the signals and manual verification we have confirmed that this incident is now mitigated.
App installation is working correctly and the pending past installations were processed.
We will continue to monitor our system to ensure that any potential re-occurrence can be mitigated quickly. We will also continue the investigation of root cause.
Based on the signals and manual verification we have confirmed that this incident is now mitigated.
App installation is working correctly and the pending past installations were processed.
We will continue to monitor our system to ensure that any potential re-occurrence can be mitigated quickly. We will also continue the investigation of root cause.
Investigating
We have discovered incident where the Forge applications installation is stuck indefinitely. We are investigating the root cause of this problem.