Symptoms
Users accessing the Practice Labs v4 interface would have experienced either slow or long delays in accessing the platform. In some circumstances; due to the platform taking so long to connect, if a customer was being proxied other errors may have appeared. This behaviour was intermittent throughout the day.
What went wrong
Our v4 interface accessed files in a different data centre when logging in. The latency on this link exceeded normal operations and caused these files to be pulled slowly, and in some cases very slowly when accessing the platform. As the latency of the link was fluctuating, this caused the intermittent nature of the problem.
Who was impacted
A small number of customers who use our v4 interface may have experienced issues, although the majority of customers now use the v5 platform. Users may have also experienced this issue when saving their settings in the v5 interface.
Why it went wrong
This went wrong due to a link between our data centres becoming abnormally congested with increased latency.
How did we fix it
We have now moved the files that are in question to the local data centre with them being replicated asynchronously to the previous data centre.