api experiencing degraded performance

Websites
100.0% uptime

www

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

voucher

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

portal

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

api

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

store

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

act

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
betareports (Data Warehouse)
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
api Reporting service
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

Insights

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Lab Services
100.0% uptime
Intranet
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Lab RDP Services
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Azure Labs
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Cisco Labs
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

Persistent Labs

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Internet Access
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Lab Availability
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
My Files
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Lab Reports
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Performance Labs
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Screenshots
Operational
Communications
100.0% uptime
Telephony
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Email Notifications
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Data Center (ATL01) Internet
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

Data Center (CDC01) Internet

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Lab Content
100.0% uptime
Assessment Items
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Exams
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime
Third Party: Intercom → Intercom Web Application
Operational

Notice history

Nov 2023

Issues accessing the Practice Labs platform
  • Resolved
    Resolved

    Symptoms

    Users attempting to browse to www.practice-labs.com will have encountered a HTTP 503 error, rendering the Practice Labs platform inaccessible.

    What Went Wrong

    A planned network change implemented during the maintenance window caused an outage that extended beyond the scheduled timeframe.

    Who Was Impacted

    All users attempting to access or use the Practice Labs platform were affected by the outage.

    Why It Went Wrong

    The intended modification aimed to facilitate the border switch ahead of a web layer migration. However, a spanning tree configuration on a non-Cisco device disrupted the priority on the core switch for the carrier VLAN, resulting in the outage. Upon reverting the change, we designated the core switch as the root bridge, resolving the issue successfully.

    How We Fixed It

    We promptly rolled back the intended network change, restoring normal functionality to the Practice Labs platform as the time exceeded the maintenance window.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are investigating an issue where users are unable to access the platform either by logging in or launching a lab from another platform.

Integration issues with LTI 1.0, 1.1 and 1.2
  • Investigating
    Update

    An automated workaround has been implemented to help remediate potential recurrences of this incident. Further investigations into the root cause are continuing and updates will be provided as more information becomes available.

  • Investigating
    Investigating

    What is wrong

    LTI v1.0, 1.1 and 1.2 integrations via api.practice-labs.com may notice an increase in loading times or in certain instances see a HTTP 503 'Service Unavailable' error when accessing Practice Labs or Assessments.

    Who is impacted

    Any customer or users using LTI v1.0, 1.1 or 1.2 integration methods from their LMS.

    Why is it wrong

    Our monitoring platform has confirmed that we have had two occurrences of this on consecutive days overnight during approximately the following time windows:

    Monday 20th November, 02:30am UTC - 07:30am UTC
    Tuesday 21st November, 02:00am UTC - 05:00am UTC

    How will we fix it

    We are currently continuing investigations into this issue and will provide updates as more information becomes available.

Oct 2023

Assessment Platform Load Testing Schedule for October 28th, from 06:00 to 09:00 UTC
  • Completed
    October 28, 2023 at 9:00 AM
    Completed
    October 28, 2023 at 9:00 AM

    Maintenance has completed successfully

  • In progress
    October 28, 2023 at 6:00 AM
    In progress
    October 28, 2023 at 6:00 AM

    Maintenance is now in progress

  • Planned
    October 28, 2023 at 6:00 AM
    Planned
    October 28, 2023 at 6:00 AM

    We are scheduling a load testing session for our production ACT assessment platform on Saturday, October 28th. During this window, performance of our ACT assessment platform may be degraded. This does NOT affect our Exam Prep assessment product.

Scheduled Maintenance - 18th October 2023 06:00 - 07:00 UTC
  • Completed
    October 18, 2023 at 7:00 AM
    Completed
    October 18, 2023 at 7:00 AM

    Maintenance has completed successfully

  • In progress
    October 18, 2023 at 6:00 AM
    In progress
    October 18, 2023 at 6:00 AM

    Maintenance is now in progress

  • Planned
    October 18, 2023 at 6:00 AM
    Planned
    October 18, 2023 at 6:00 AM

    We are planning a scheduled maintenance on Wednesday, October 18th, to replace a faulty module in our Core Cisco Switch.

    We are not expecting any outages to occur from this maintenance, but our Infrastructure team will be on hand in the event of an outage to ensure the platform is up and running smoothly, all work is planned to be completed within the maintenance window.

    We appreciate your understanding, and we apologize in advance for any disruption as we carry out these critical Data Centre operations.

Scheduled Maintenance - 11th October 2023 06:00 - 07:00 UTC
  • Completed
    October 11, 2023 at 7:00 AM
    Completed
    October 11, 2023 at 7:00 AM

    Maintenance has completed successfully

  • In progress
    October 11, 2023 at 6:00 AM
    In progress
    October 11, 2023 at 6:00 AM

    Maintenance is now in progress

  • Planned
    October 11, 2023 at 6:00 AM
    Planned
    October 11, 2023 at 6:00 AM

    We are planning for a scheduled maintenance during that time.
    During this maintenance window we will be disabling TLS versions 1 and 1.1 as a crucial step in ensuring the security and integrity of our digital communications.  

Atlanta (ATL01) Hardware issue
  • Investigating
    Investigating

    We are currently addressing a memory hardware issue in one of our storage servers. Our team is actively monitoring the situation and working closely with our hardware maintenance provider to swiftly rectify the problem. We want to assure our users that this issue is not affecting the performance or availability of our platform and we are committed to resolving this matter as quickly as possible. Thank you for your understanding.

Sep 2023

V4 Issue - Unable to Interact with Lab Devices
  • Resolved
    Update

    What went wrong

    Our 'Connect' device client in our older UI (V4) was inaccessible and not accepting mouse movements or keyboard inputs.

    Who was impacted

    All users using our 'Connect' device client in our V4 UI.

    Why it went wrong

    We were deploying new update to our connect software and during this roll out process we were alerted by a member of our team that devices were inaccessible.

    How did we fix it

    We stopped our deployment and reverted any of the servers that had already been updated.
    We are currently reviewing this issue with out software engineering team.

  • Resolved
    Resolved

    Our monitor period has ended and our incident has been resolved. Thank you for your patience.

  • Monitoring
    Monitoring

    We are aware of an issue on our V4 platform where users were unable to interact with devices with Connect Device Client enabled.
    We have implemented a fix that resolved the issue and now we are monitoring the V4 platform to ensure this issue does not occur again.

Emergency maintenance - Upgrading our CDC database
  • Completed
    September 20, 2023 at 2:01 PM
    Completed
    September 20, 2023 at 2:01 PM

    Maintenance has completed successfully, we will continue to monitor the performance of our platform to ensure stability for users.

  • In progress
    September 20, 2023 at 11:45 AM
    In progress
    September 20, 2023 at 11:45 AM

    Our Emergency maintenance is now in progress.

  • Planned
    September 20, 2023 at 8:37 AM
    Planned
    September 20, 2023 at 8:37 AM

    We are performing emergency maintenance at 9am UTC to upgrade our CDC database servers as part of the remediation plan from yesterdays outage, this should not impact users but are monitoring closely.

Issues accessing the Practice Labs platform
  • Resolved
    Resolved

    Symptoms

    Users attempting to access the Practice Labs platform were unable to login.

    What went wrong

    Our database cluster had a spike in memory usage which caused a failover of the primary node to one of the secondary nodes. During this period the cluster become unresponsive.

    Who was impacted

    All users attempting to login to the Practice Labs platform.

    Why it went wrong

    Memory exhaustion in our database cluster.

    How did we fix it

    We have upgraded the memory in all 4 nodes in our database cluster. One node is running with less RAM than we have allocated due to a minor hardware fault which is being addressed by our maintenance provider.

    Our database cluster is now operating with reduced processing times, in some cases up to 60% faster with the additional RAM. We have monitored this closely for 7 days and are now comfortable that we can come out of monitoring.

  • Monitoring
    Update

    We have completed our emergency maintenance to upgrade our CDC database servers as part of the remediation plan from yesterdays outage. We will continue to closely monitor the platform.

  • Monitoring
    Update

    We are performing emergency maintenance at 9am UTC to upgrade our CDC database servers as part of the remediation plan from yesterdays outage, this should not impact users but are monitoring closely.

  • Monitoring
    Monitoring

    Our database clusters primary node automatically failed over to one of its secondary nodes which restored user access to the platform.

    We are investigating further corrective actions and will continue to monitor. We appreciate your understanding and patience during this incident.

  • Identified
    Identified

    We are currently investigating an issue where users are unable to access the platform either by logging in or launching a lab from another platform.

    We apologies for any inconveniences this has caused.

  • Investigating
    Investigating

    We are investigating an issue where users are unable to access the platform either by logging in or launching a lab from another platform.

Degraded V4 performance
  • Resolved
    Resolved

    Symptoms

    Users accessing the Practice Labs v4 interface would have experienced either slow or long delays in accessing the platform. In some circumstances; due to the platform taking so long to connect, if a customer was being proxied other errors may have appeared. This behaviour was intermittent throughout the day.

    What went wrong

    Our v4 interface accessed files in a different data centre when logging in. The latency on this link exceeded normal operations and caused these files to be pulled slowly, and in some cases very slowly when accessing the platform. As the latency of the link was fluctuating, this caused the intermittent nature of the problem.

    Who was impacted

    A small number of customers who use our v4 interface may have experienced issues, although the majority of customers now use the v5 platform. Users may have also experienced this issue when saving their settings in the v5 interface.

    Why it went wrong

    This went wrong due to a link between our data centres becoming abnormally congested with increased latency.

    How did we fix it

    We have now moved the files that are in question to the local data centre with them being replicated asynchronously to the previous data centre.

  • Monitoring
    Update

    We implemented a fix and are continuing to monitoring the platform.

    Next update will be tomorrow 09/14/2023 9am UTC.

  • Monitoring
    Update

    We are still investigating and monitoring the V4 platform at this time.
    Next update will be tomorrow 09/13/2023 10am BST.

  • Monitoring
    Monitoring

    We are currently investigating an issue where we are seeing degraded performance when attempting to use our V4 platform. We are continuing to monitor the performance.

Scheduled Maintenance - 6th September 2023 06:00 - 07:00 UTC
  • Completed
    September 06, 2023 at 8:23 AM
    Completed
    September 06, 2023 at 8:23 AM

    Maintenance has completed successfully.

  • Planned
    September 06, 2023 at 6:00 AM
    Planned
    September 06, 2023 at 6:00 AM

    Please be advised we are planning scheduled maintenance on this date and the platform may be unavailable between 06:00 - 07:00 UTC.
    Persistent Labs in our ATL Data Center will be unavailable for the next 20 minutes.
    This is required to perform platform changes and maintenance.

Sep 2023 to Nov 2023

Contact

Telephone: UK: +44 203 588 7250 (9am-10pm) | US: +1 612 200 0993 (4am-5pm EST)

Email: support@practice-labs.com (24x7)

Service Status Detail: Website Status | Data Center Internet Status | Data Center RDP Status