Incident Report

Summary

On Tuesday, 13 December 2022 at 07:34 PT, users of Gmail, Google Voice, and Google Chat experienced disruption in their services for a period of 2 hours and 30 minutes. To our Google Workspace customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we have conducted an internal investigation and are taking immediate steps to improve the resiliency of our service.

Root Cause

The issue was caused by an unanticipated increase in traffic to the internal user ID management system used by multiple Google Workspace products. Upon further investigation, we identified that the overwhelming amount of traffic resulted from erroneous changes made to update the membership details from one of the upstream systems.

Remediation and Prevention

Google engineers were alerted to this issue through automated monitoring, which also blocked the incoming traffic simultaneously within a few minutes. However, the high traffic during this period impacted downstream services.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

• Change the design in the internal user ID management system to not remove users, unless mentioned explicitly as allowed.
• Introduce multiple redundancies in the system, including self-throttling auto-kill mechanisms in the involved systems to keep the traffic under a defined threshold.
• Introduce robust control and process reviews for data changes.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

Detailed Description of Impact

On Tuesday, 13 December 2022 between 07:34 and 10:04 PT, various Google Workspace products (noted below) experienced impact in their services:

Google Chat

• Some users observed incorrect status indicators representing their presence and typing status.

Google Voice

Users experienced impact in various telephony services:

• Some SMS messages were delayed by an average of 1.2 minutes.
• A limited number of Session Initiation Protocol (SIP) registrations were not completed. Affected user devices may have been temporarily unable to place or receive calls.
• Active call queue processing was delayed, which may have resulted in delayed call history, billing, or voicemail delivery for some users.

Gmail

• A few users experienced intermittent issues signing in and temporary errors.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 13 Dec 2022 07:34

Incident End: 13 Dec 2022 10:04

Duration: 2 hours, 30 minutes

Affected Services and Features:

Google Voice, Google Chat

Regions/Zones: Global

Description:

Users of Google Voice and Google Chat experienced disruptions in their service, starting on 13 December 2022 at 07:34 PT for a period of 2 hours and 30 minutes globally. From preliminary analysis, the issue was due to throttling in one of Google's systems that authorizes access for customers.

Google will be completing a comprehensive incident report in the following days that will provide a full root cause.

Customer Impact:

• Users of Google Chat saw incorrect status indicators representing their presence and typing status. Some users of Google Voice experienced significant impact in various telephony services, including delay in delivery of SMS and failure in Session Initiation Protocol (SIP) device registrations, leading to users being unable to place or receive calls.
• Users also may have experienced delayed call history or voicemail delivery.

We are experiencing an issue with Google Chat. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2022-12-13 10:30 US/Pacific with current details.

For more information please refer to : https://www.google.com/appsstatus/dashboard/incidents/tLjnxF7DdVTCicccdQtj