The outage with the third party vendor has been resolved.
Posted Jun 12, 2025 - 16:42 EDT
Monitoring
The third-party services have been recovered. All Hypercare services are operational. We're currently monitoring the services.
Posted Jun 12, 2025 - 16:29 EDT
Update
Automated code escalation has been manually restored. All code team functions are now fully operational.
Posted Jun 12, 2025 - 16:09 EDT
Update
In addition to the previously reported Doppler outage, we have now identified a separate issue affecting virtual paging. This is due to a service disruption with Twilio, a third-party vendor. As a result, virtual paging is currently unavailable and may affect message delivery for some users.
We continue to monitor both the Doppler and Twilio incidents and will provide updates as services are restored. Thank you for your continued patience.
Posted Jun 12, 2025 - 15:51 EDT
Identified
The engineering team is manually updating the secrets to restore the services. Everbridge integration has been restored.
Posted Jun 12, 2025 - 15:44 EDT
Investigating
We are currently experiencing a partial service disruption caused by an ongoing outage with Doppler, one of our third-party service providers. As a result, the following Hypercare features are impacted:
- Escalation notifications related to code team activations (note: code team activations remain fully operational) - Schedule change email notifications - Schedule and Equity report downloads - Everbridge file generation
We are actively monitoring the situation and will provide updates as Doppler resolves the issue.
This incident affected: United States Region (Virtual Pager (U.S. Region), Notifications and Real-Time Syncing (U.S. Region), Code Teams (U.S. Region), Administration and Scheduling (U.S. Region), API & Integrations (U.S. Region)) and Canadian Region (Notifications and Real-Time Syncing (Canadian Region), Code Teams (Canadian Region), Administration and Scheduling (Canadian Region), API & Integrations (Canadian Region), Virtual Page (Canadian Region)).