Root Cause of Zendesk Email Processing Incident
What was the root cause of the Zendesk email processing incident?
The root cause of the Zendesk email processing incident was connectivity problems with Gmail. The Mail Fetcher service experienced issues where Gmail's 302 Moved responses were misinterpreted as failures by the liveness probe.
This misinterpretation led the container orchestrator to replace the Pods, halting mail processing. The issue was resolved when Gmail stopped blocking health checks, allowing the Pods to be recreated and email processing to resume normally.
More related questions
What happened during the Zendesk service incident on November 16, 2023?
On November 16, 2023, Zendesk experienced a service incident affecting inbound email processing. From 18:02 UTC to 20:00 UTC, some customers in Pods 13, 17, 19, 23, 28, and 29 faced delays or stoppages in receiving inbound emails. The issue was due…
Which Zendesk Pods were affected by the email processing issue on November 16, 2023?
The email processing issue on November 16, 2023, affected Zendesk Pods 13, 17, 19, 23, 28, and 29. Customers using these Pods experienced delays or a complete stoppage in receiving inbound emails. The incident was caused by connectivity issues…
How was the Zendesk email processing issue resolved?
The Zendesk email processing issue was resolved by restoring inbound mail traffic after Gmail stopped blocking health checks. This allowed the Support system to recreate its Pods and resume normal email processing. Once the health checks were no…
What steps are being taken to prevent future Zendesk email processing issues?
To prevent future email processing issues, Zendesk is implementing several remediation items. These include improving existing tools for email health checks, creating additional alerts, and adding correction code lines to specific applications….
Where can I find more information about Zendesk system status?
For more information about Zendesk's current system status, you can check their system status page. This page provides updates on any ongoing issues and the status of Zendesk services. Additionally, summaries of post-mortem investigations are…
Interested indeflectingover 70% of your Zendesk support tickets?
Zendesk Support Tickets
Zendesk Messaging (live chat)
Join1,000+ companies reducing their support costs and freeing up support agents for more important work
“We needed an AI agent integrated within our current tools. My AskAI was the only solution that wasn't going to disrupt our operations.”
Zeffy
“At the end of last year I was given the challenge - how can we provide the same or better service, without hiring anyone?”
Zinc
“My AskAI blew everybody else out of the water. It made the selection process very easy for us.”
Customer.io($50M+ ARR)
“It now resolves 71% of queries (over 35,000 every month), meaning more time solving complex issues and improving UX.”
Freecash
“We needed an AI agent integrated within our current tools. My AskAI was the only solution that wasn't going to disrupt our operations.”
Zeffy
“At the end of last year I was given the challenge - how can we provide the same or better service, without hiring anyone?”
Zinc
“My AskAI blew everybody else out of the water. It made the selection process very easy for us.”
Customer.io($50M+ ARR)
“It now resolves 71% of queries (over 35,000 every month), meaning more time solving complex issues and improving UX.”
Freecash
Reduce support costs.Spend more time on customer success.
