What went wrong

Our 'Connect' device client in our older UI (V4) was inaccessible and not accepting mouse movements or keyboard inputs.

Who was impacted

All users using our 'Connect' device client in our V4 UI.

Why it went wrong

We were deploying new update to our connect software and during this roll out process we were alerted by a member of our team that devices were inaccessible.

How did we fix it

We stopped our deployment and reverted any of the servers that had already been updated.
We are currently reviewing this issue with out software engineering team.

Our monitor period has ended and our incident has been resolved. Thank you for your patience.

Symptoms

Users attempting to access the Practice Labs platform were unable to login.

What went wrong

Our database cluster had a spike in memory usage which caused a failover of the primary node to one of the secondary nodes. During this period the cluster become unresponsive.

Who was impacted

All users attempting to login to the Practice Labs platform.

Why it went wrong

Memory exhaustion in our database cluster.

How did we fix it

We have upgraded the memory in all 4 nodes in our database cluster. One node is running with less RAM than we have allocated due to a minor hardware fault which is being addressed by our maintenance provider.

Our database cluster is now operating with reduced processing times, in some cases up to 60% faster with the additional RAM. We have monitored this closely for 7 days and are now comfortable that we can come out of monitoring.

We have completed our emergency maintenance to upgrade our CDC database servers as part of the remediation plan from yesterdays outage. We will continue to closely monitor the platform.

We are performing emergency maintenance at 9am UTC to upgrade our CDC database servers as part of the remediation plan from yesterdays outage, this should not impact users but are monitoring closely.

Our database clusters primary node automatically failed over to one of its secondary nodes which restored user access to the platform.

We are investigating further corrective actions and will continue to monitor. We appreciate your understanding and patience during this incident.

We are currently investigating an issue where users are unable to access the platform either by logging in or launching a lab from another platform.

We apologies for any inconveniences this has caused.

Symptoms

Users accessing the Practice Labs v4 interface would have experienced either slow or long delays in accessing the platform. In some circumstances; due to the platform taking so long to connect, if a customer was being proxied other errors may have appeared. This behaviour was intermittent throughout the day.

What went wrong

Our v4 interface accessed files in a different data centre when logging in. The latency on this link exceeded normal operations and caused these files to be pulled slowly, and in some cases very slowly when accessing the platform. As the latency of the link was fluctuating, this caused the intermittent nature of the problem.

Who was impacted

A small number of customers who use our v4 interface may have experienced issues, although the majority of customers now use the v5 platform. Users may have also experienced this issue when saving their settings in the v5 interface.

Why it went wrong

This went wrong due to a link between our data centres becoming abnormally congested with increased latency.

How did we fix it

We have now moved the files that are in question to the local data centre with them being replicated asynchronously to the previous data centre.

We implemented a fix and are continuing to monitoring the platform.

Next update will be tomorrow 09/14/2023 9am UTC.

We are still investigating and monitoring the V4 platform at this time.
Next update will be tomorrow 09/13/2023 10am BST.

Please be advised, the incident involving our North America data center has been monitored closely throughout the weekend. Our screenshot feature and our persistent labs in North America are working as intended.

This incident has been closed.

Please be advised, the incident involving our North America data center has been resolved. Our screenshot feature and our persistent labs in North America are working as intended.

This incident has been placed into monitoring until further notice.

The situation remains unchanged at the moment. The affected services remain limited to Persistent labs and lab screenshot functions out of the North America data center. Efforts to resolve the matter remain underway and more updates will follow.

The situation remains unchanged at the moment. The affected services remain limited to Persistent labs and lab screenshot functions out of the North America data center. Efforts to resolve the matter remain underway and more updates will follow.

There has been no change in the status of this incident, as we continue to seek the best possible solution. Affected services remain limited to persistent labs and lab screenshot capabilities out of our North America data center.

The investigation remains ongoing as we progress in our discovery. More updates will follow as news develop. Next update set for 1830 09/01/2023

We have identified issues with the screen capture function relating to this ongoing incident as we continue our investigation. Next update will be at 19:00 09/01/2023

We are continuing to work on a fix for this incident, we apologies for any inconvenience that this has caused. The next update will be posted at 19:00 09/01/2023.