Incident Report

Summary

On Thursday, 6 July 2023, Gmail experienced elevated errors in some of its core functionalities due to internal task issues. This occurred for 1 hour and 27 minutes starting at 15:22 PT. On Monday, 10 July 2023, the impact reoccurred for a period of 42 minutes between 18:37 to 19:19. During the period of impact, some Gmail users experienced intermittent unavailability to a number of services, including syncing email to their clients, uploading attachments, initial page loads, and email deliveries. To our Gmail users who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

Gmail uses an internal triggering process system, built on an internal Spanner queue receiver, to perform asynchronous processing of the incoming messages in the Gmail backend.

The root cause of the incident is due to an unintended configuration change made in one of the internal proxy layers that handles Spanner queue receivers. However, it inadvertently reduced the callback size limit of the Spanner queue receiver resulting in throttling of the queue, which caused elevated memory pressure of the backend servers while handling the backlog of messages.

Google engineers mitigated the issue by rolling back the incorrect change and performed a controlled release of the messages to ensure the backend servers were resourced.

Remediation and Prevention

On Thursday July 6, 2023 at 15:22, Google engineers were alerted to the issue from the monitoring tools in place, and started working on multiple mitigation strategies to restore the service.

There was a spike in traffic from a single problematic trigger, which caused memory pressures on the backend servers. Google engineers paused the processing of this trigger and implemented temporary throttles and began to slowly ramp up the processing of this trigger to clean up the backlog of tasks.

On Monday July 10, 2023, Google engineers noticed that an incorrect configuration change had been rolled out after the issue on Thursday that caused a recurrence. Google engineers implemented a controlled ramp of the processing of the trigger, similar to the mitigation that was performed on July 6. However, one partition of production traffic was missed during this procedure and caused follow-on impact starting at 18:47. To prevent further user impact, engineers paused the processing of the trigger again, and as a result user impact began to subside at around 19:15. Google engineers performed cleanup of this trigger at a controlled pace over the course of the next few weeks.

Google is committed to preventing a repeat of this issue in the future. Google engineers have set up a conservative limit for throttling the incoming load. This is to ensure the memory utilization is within the limits.

Google is also completing the following actions:

• Implement additional safeguards around change management while system improvements are being developed

Detailed Description of Impact

On Thursday, 6 July 2023 between 15:15 and 16:47 PT, a portion of Gmail users experienced some IMAP unavailability, which resulted in asynchronous syncing issues between their clients and their mailboxes. Users may have experienced errors when syncing mail and with settings in their clients, issues with initial page loads, one or more errors accessing Gmail’s external API through the API Server, an inbound delivery delay, or an outbound delay.

On Monday, 10 July 2023 between 18:37 and 19:19, a portion of Gmail users received errors attempting to sync their clients to their mailboxes, upload attachments and load ads, or load their web client for the first time. Meanwhile some users experienced IMAP errors, errors from Sync Server, and unavailability when accessing the API Server. Email delivery was also impacted, with some inbound emails temporarily delayed.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 6 July 2023 at 15:15

Incident End: 6 July 2023 at 16:47

Duration: 1 hour, 32 minutes

Affected Services and Features:

Gmail

Regions/Zones: Global

Description:

Some Gmail users experienced elevated latency and intermittent unavailability for a period of 1 hour and 32 minutes. This was due to an unexpected resource usage increase in backend servers.

From the preliminary analysis, the issue was triggered during validation of a configuration change. This triggered an increase in the number of background tasks, which resulted in elevated memory pressure. Google engineers quickly started to work on containing the issue and mitigated the impact by disabling the background tasks and returning the memory usage to normal levels.

Customer Impact:

Impacted users may have experienced elevated latency and errors while syncing their Gmail accounts to clients. Users may have also experienced intermittent failures in the page load on the web and latency in email delivery.