Root Cause of Zendesk's July 23, 2024 Incident Explained
What was the root cause of the Zendesk incident on July 23, 2024?
The root cause of the Zendesk incident on July 23, 2024, was a new feature rollout related to managing team members' permissions. This feature allowed agents in custom roles to manage other team members and their role assignments. The rollout led to a significant increase in requests to the internal permissions service, causing capacity saturation of its database cluster.
The increased traffic resulted in the cluster reaching its maximum network bandwidth capacity, leading to a networking failure between the cluster and the service’s app servers. This failure caused the access issues experienced by customers on Pod 29.
More related questions
What happened during the Zendesk service incident on July 23, 2024?
On July 23, 2024, Zendesk experienced a service incident affecting customers on Pod 29. From 10:58 UTC to 14:57 UTC, users faced issues accessing Zendesk products, including the Admin Center, through the Product Tray. Approximately 1% of customer…
How did Zendesk resolve the service incident on July 23, 2024?
Zendesk resolved the service incident on July 23, 2024, by initially increasing the capacity of the permissions service’s database instance. This provided a short-term recovery while the root cause was being identified. Once the root cause was…
What steps did Zendesk take to prevent future incidents like the one on July 23, 2024?
To prevent future incidents similar to the one on July 23, 2024, Zendesk planned several remediation items. These include reducing network traffic from permissions checks, which is currently in progress, and scheduling additional monitors and…
What were the main errors experienced during the Zendesk incident on July 23, 2024?
During the Zendesk incident on July 23, 2024, customers on Pod 29 experienced several errors. These included the inability to access Zendesk products through the Product Tray and receiving 503 errors when accessing authenticated features within…
Interested indeflectingover 70% of your Zendesk support tickets?
Zendesk Support Tickets
Zendesk Messaging (live chat)
Join1,000+ companies reducing their support costs and freeing up support agents for more important work
“We needed an AI agent integrated within our current tools. My AskAI was the only solution that wasn't going to disrupt our operations.”
Zeffy
“At the end of last year I was given the challenge - how can we provide the same or better service, without hiring anyone?”
Zinc
“My AskAI blew everybody else out of the water. It made the selection process very easy for us.”
Customer.io($50M+ ARR)
“It now resolves 71% of queries (over 35,000 every month), meaning more time solving complex issues and improving UX.”
Freecash
“We needed an AI agent integrated within our current tools. My AskAI was the only solution that wasn't going to disrupt our operations.”
Zeffy
“At the end of last year I was given the challenge - how can we provide the same or better service, without hiring anyone?”
Zinc
“My AskAI blew everybody else out of the water. It made the selection process very easy for us.”
Customer.io($50M+ ARR)
“It now resolves 71% of queries (over 35,000 every month), meaning more time solving complex issues and improving UX.”
Freecash
Reduce support costs.Spend more time on customer success.
