After an hour of monitoring, Hypercare team believes this slow down incident is now fully resolved. Hypercare engineering team will be modifying our notes functionality to ensure it does not affect performance and bring the feature back into the ecosystem.
Posted Jan 31, 2023 - 13:15 EST
Update
We believe the issue is now fully resolved. We pinpointed the root cause to the "notes feature" on our platform that unfortunately was pointing towards another database. That database hit a performance limitation. Given that Android did not implement notes yet, Android was not affected and was fully functional.
We will continue to monitor the situation for the next hour. If there's continues to be no issues, we will be closing the issue.
Posted Jan 31, 2023 - 12:21 EST
Update
Our engineering team has been working tirelessly on this issue.
We believe we have highly probable suspicion which is iOS and Web Platforms differ in the amount of information asked of the server in critical actions such as pulling chats, pulling read receipts, and loading your app for the first time. This is most likely causing the inability to perform critical actions during peak periods while Android behaves normally as it could result in the server not being able to respond to the platform in a timely manner resulting in a timeout.
The next step is for iOS and Web to minimize the amount of information asked of the server to reduce the time for server to return the result which in return should fix the problem. We are hoping to create a new version of web and android ready for release by tomorrow. We will keep everyone posted on the incident until it is fully resolved.
Posted Jan 30, 2023 - 18:14 EST
Update
We are currently suspecting that may not be a performance issue since Android is behaving normally. We are now currently looking at potentially the root cause as slowdowns are still happening intermittently. We will continue to update as we get more information.
Posted Jan 30, 2023 - 15:45 EST
Monitoring
The system is currently loading as is. We are monitoring to ensure we notify if the server is slowing down again.
Posted Jan 30, 2023 - 13:29 EST
Investigating
The hypercare team is currently investigating a slowdown in the Canadian region. This currently affects loading chats from your account but may not involve sending messages for example. Our team is looking into temporary measures until the full root issue is resolved. This is currently flagged as the number one priority to resolve.
Posted Jan 30, 2023 - 13:02 EST
This incident affected: Canadian Region (Login and Single Sign On (Canadian Region), Viewing Who is On-Call (Canadian Region), Code Teams (Canadian Region), Self-serve Scheduling (Canadian Region), Administration and Scheduling (Canadian Region)).