We sincerely apologize for any impact of the degraded performance incident may have caused your organization of the last few weeks. After monitoring the performance of our servers we've deduced that this incident has been resolved. Please be advised that a post mortem will be sent out by the end of this week to explain the technical issues our servers experienced along with the fix that was deployed.
Posted Nov 08, 2022 - 18:33 EST
----------- Canadian Region Only ---------- We have put in an updated version for the web platform that was causing additional stress to the server which reduces the number of unnecessary network requests sent.
We believe this should prevent any slowdowns from occurring while allowing us time to resolve the root issue.
We will continue to monitor the situation. The fix for the root issue will be coming out early next week but with the mitigation (additional servers) as well as the updated version for web platform we believe slowdowns should not occur anymore.
Posted Oct 20, 2022 - 19:26 EDT
--- Relevant to Canadian Regions and Customers --- Our system has been reacting very positively since we upgraded our infrastructure at 19:00 last night to support the additional load generated from our web platform. The root cause has yet to be resolved however you should see little to no slowdowns given this upgrade in infrastructure.
We hope to have this fully resolved soon and will continue posting updates.
Posted Oct 19, 2022 - 15:05 EDT
Our team will be running some stress tests around 2100 EST to ensure our infrastructure can absorb any potential high traffic load as a mitigation until the problem is fully resolved. Once our team confirms this stress test passes, our system will be marked as fully operational in the Canadian region.
Posted Oct 18, 2022 - 17:56 EDT
The issue seems to be identified and traced back to the web browser platform. We are currently investigating all the possible reasons why this issue is occurring on the web platform and will update you as soon as we have any updates.
Posted Oct 18, 2022 - 17:25 EDT
The engineering team remains engaged to determine the trigger of the issue and continues to explore multiple paths to resolution. So far we know that our servers are experiencing a larger than usual spike in resource consumption and is only affecting users in the Canadian region.
We’ll provide another update when additional information becomes available.
Posted Oct 18, 2022 - 15:47 EDT
---- Affecting only the Canadian Region ---- There is a Hypercare service disruption that is occurring intermittently on the Canadian servers. During a service disruption, end users will have performance issues that affect their experience with Hypercare.
We’re investigating the issue and an update will be provided in 30 minutes or sooner if we have more information.
Posted Oct 18, 2022 - 14:07 EDT
This incident affected: Messages (Communication, Sending Images, Delivery Receipts), On-Call (Messaging On-Call Roles, Code Team Activations), Login & Single Sign On (Login and Signup), and Virtual Pager.