[CA & US Region] Login flow is not functioning as expected

Incident Report for Hypercare

Postmortem

This incident caused an unplanned interruption to organizations that use the Single Sign-On (SSO) service to log into Hypercare via web browsers. This incident occurred during routine maintenance on our web client platform, which was intended to improve overall performance.

What Happened:

On July 30, 2024 at approximately 4 p.m. EST, our team deployed an update to the web client platform. This update, designed to enhance performance, had been thoroughly tested in our Staging environment without any issues. However, upon release to the Production environment, an unexpected outcome caused an interruption to our SSO service.

Impact:

As a result of this interruption, Users experienced difficulties with new Login attempts via SSO on all web browsers from 4 p.m. to 4:50 p.m. EST. SSO Login attempts on our mobile applications and users already logged into the platform were not impacted.

Root Cause:

The root cause of the issue was an unforeseen interaction between the updated web client platform and the Production environment’s SSO configuration. This interaction did not present during our Staging environment tests, leading to the unexpected service disruption.

We understand the inconvenience this caused and sincerely apologize for any disruption to your operations.

Posted Jul 30, 2024 - 17:25 EDT

Resolved

The fix is deployed and users with SSO should be able to login now.
Posted Jul 30, 2024 - 16:53 EDT

Identified

There is an issue with logging in through single-sign-on (SSO) for new sessions. Users who are already logged in are not impacted. Issue has been identified and the fix is deploying.
Posted Jul 30, 2024 - 16:25 EDT
This incident affected: Canadian Region (Login and Single Sign On (Canadian Region)) and United States Region (Login and Single Sign On (U.S. Region)).