Overview:
This incident was caused by a technical constraint related to database query limits and a recent release aimed at improving the search feature on Messaging and Contacts.
What Happened:
On December 11, 2024, at approximately 11:00 a.m. ET, the platform experienced a surge in concurrent searches across a large user base. This triggered the database's fixed data limit for query execution, leading to search processes pooling and eventually crashing. The issue was observed in the United States, where higher user loads exceeded expectations. Canadian operations were unaffected.
Impact:
From 11:10 a.m. to 11:50 a.m. ET, Hypercare Messaging and Virtual Paging services were non-operational, and on-call schedules loaded slowly.
Root Cause:
The root cause was the interaction between the database’s inherent query size limit and the recent enhancement to our search functionality on Messaging and Contacts. Under high concurrent usage, query sizes exceeded the database's fixed threshold, resulting in the experienced performance issues.
Resolution:
The search enhancement release has been rolled back to stabilize the system and restore full functionality. All services are now operational, and additional safeguards have been implemented to prevent similar issues in the future.
We sincerely apologize for the disruption and remain dedicated to ensuring a reliable, high-performing platform for all users.