api experiencing degraded performance

Websites
100.0% uptime

www

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

voucher

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

portal

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

api

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

store

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

act

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
betareports (Data Warehouse)
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
api Reporting service
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

Insights

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Lab Services
100.0% uptime
Intranet
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Lab RDP Services
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Azure Labs
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Cisco Labs
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

Persistent Labs

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Internet Access
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Lab Availability
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
My Files
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Lab Reports
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Performance Labs
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Screenshots
Operational
Communications
100.0% uptime
Telephony
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Email Notifications
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Data Center (ATL01) Internet
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

Data Center (CDC01) Internet

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Lab Content
100.0% uptime
Assessment Items
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Exams
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Third Party: Intercom → Intercom Web Application
Operational

Notice history

Nov 2023

Issues accessing the Practice Labs platform
  • Resolved
    Resolved

    Symptoms

    Users attempting to browse to www.practice-labs.com will have encountered a HTTP 503 error, rendering the Practice Labs platform inaccessible.

    What Went Wrong

    A planned network change implemented during the maintenance window caused an outage that extended beyond the scheduled timeframe.

    Who Was Impacted

    All users attempting to access or use the Practice Labs platform were affected by the outage.

    Why It Went Wrong

    The intended modification aimed to facilitate the border switch ahead of a web layer migration. However, a spanning tree configuration on a non-Cisco device disrupted the priority on the core switch for the carrier VLAN, resulting in the outage. Upon reverting the change, we designated the core switch as the root bridge, resolving the issue successfully.

    How We Fixed It

    We promptly rolled back the intended network change, restoring normal functionality to the Practice Labs platform as the time exceeded the maintenance window.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are investigating an issue where users are unable to access the platform either by logging in or launching a lab from another platform.

Oct 2023

Sep 2023

Issues accessing the Practice Labs platform
  • Resolved
    Resolved

    Symptoms

    Users attempting to access the Practice Labs platform were unable to login.

    What went wrong

    Our database cluster had a spike in memory usage which caused a failover of the primary node to one of the secondary nodes. During this period the cluster become unresponsive.

    Who was impacted

    All users attempting to login to the Practice Labs platform.

    Why it went wrong

    Memory exhaustion in our database cluster.

    How did we fix it

    We have upgraded the memory in all 4 nodes in our database cluster. One node is running with less RAM than we have allocated due to a minor hardware fault which is being addressed by our maintenance provider.

    Our database cluster is now operating with reduced processing times, in some cases up to 60% faster with the additional RAM. We have monitored this closely for 7 days and are now comfortable that we can come out of monitoring.

  • Monitoring
    Update

    We have completed our emergency maintenance to upgrade our CDC database servers as part of the remediation plan from yesterdays outage. We will continue to closely monitor the platform.

  • Monitoring
    Update

    We are performing emergency maintenance at 9am UTC to upgrade our CDC database servers as part of the remediation plan from yesterdays outage, this should not impact users but are monitoring closely.

  • Monitoring
    Monitoring

    Our database clusters primary node automatically failed over to one of its secondary nodes which restored user access to the platform.

    We are investigating further corrective actions and will continue to monitor. We appreciate your understanding and patience during this incident.

  • Identified
    Identified

    We are currently investigating an issue where users are unable to access the platform either by logging in or launching a lab from another platform.

    We apologies for any inconveniences this has caused.

  • Investigating
    Investigating

    We are investigating an issue where users are unable to access the platform either by logging in or launching a lab from another platform.

Degraded V4 performance
  • Resolved
    Resolved

    Symptoms

    Users accessing the Practice Labs v4 interface would have experienced either slow or long delays in accessing the platform. In some circumstances; due to the platform taking so long to connect, if a customer was being proxied other errors may have appeared. This behaviour was intermittent throughout the day.

    What went wrong

    Our v4 interface accessed files in a different data centre when logging in. The latency on this link exceeded normal operations and caused these files to be pulled slowly, and in some cases very slowly when accessing the platform. As the latency of the link was fluctuating, this caused the intermittent nature of the problem.

    Who was impacted

    A small number of customers who use our v4 interface may have experienced issues, although the majority of customers now use the v5 platform. Users may have also experienced this issue when saving their settings in the v5 interface.

    Why it went wrong

    This went wrong due to a link between our data centres becoming abnormally congested with increased latency.

    How did we fix it

    We have now moved the files that are in question to the local data centre with them being replicated asynchronously to the previous data centre.

  • Monitoring
    Update

    We implemented a fix and are continuing to monitoring the platform.

    Next update will be tomorrow 09/14/2023 9am UTC.

  • Monitoring
    Update

    We are still investigating and monitoring the V4 platform at this time.
    Next update will be tomorrow 09/13/2023 10am BST.

  • Monitoring
    Monitoring

    We are currently investigating an issue where we are seeing degraded performance when attempting to use our V4 platform. We are continuing to monitor the performance.

Sep 2023 to Nov 2023