We sincerely apologize for any impact of the degraded performance incident may have caused your organization of the last few weeks. After monitoring the performance of our servers we've deduced that this incident has been resolved. Please be advised that a post mortem will be sent out by the end of this week to explain the technical issues our servers experienced along with the fix that was deployed.
----------- Canadian Region Only ----------
We have put in an updated version for the web platform that was causing additional stress to the server which reduces the number of unnecessary network requests sent.
We believe this should prevent any slowdowns from occurring while allowing us time to resolve the root issue.
We will continue to monitor the situation. The fix for the root issue will be coming out early next week but with the mitigation (additional servers) as well as the updated version for web platform we believe slowdowns should not occur anymore.
--- Relevant to Canadian Regions and Customers ---
Our system has been reacting very positively since we upgraded our infrastructure at 19:00 last night to support the additional load generated from our web platform. The root cause has yet to be resolved however you should see little to no slowdowns given this upgrade in infrastructure.
We hope to have this fully resolved soon and will continue posting updates.
Our team will be running some stress tests around 2100 EST to ensure our infrastructure can absorb any potential high traffic load as a mitigation until the problem is fully resolved. Once our team confirms this stress test passes, our system will be marked as fully operational in the Canadian region.
The issue seems to be identified and traced back to the web browser platform. We are currently investigating all the possible reasons why this issue is occurring on the web platform and will update you as soon as we have any updates.
The engineering team remains engaged to determine the trigger of the issue and continues to explore multiple paths to resolution. So far we know that our servers are experiencing a larger than usual spike in resource consumption and is only affecting users in the Canadian region.
We’ll provide another update when additional information becomes available.
---- Affecting only the Canadian Region ----
There is a Hypercare service disruption that is occurring intermittently on the Canadian servers. During a service disruption, end users will have performance issues that affect their experience with Hypercare.
We’re investigating the issue and an update will be provided in 30 minutes or sooner if we have more information.
This incident affected: Canadian Region (Login and Single Sign On (Canadian Region), Messaging (Canadian Region), File Attachments (Canadian Region), Viewing Who is On-Call (Canadian Region), Code Teams (Canadian Region)).