Provation Apex is currently experiencing technical difficulties

Incident Report for Provation

Postmortem

Issue Resolution Report

Description:

On February 24, 2026, between 8:51AM and 9:22 AM CST, some customers experienced intermittent difficulties uploading video documentation and accessing other platform features.

Impact Summary:

On February 24, 2026, between 14:51-15:22 UTC (8:51am-9:22am CST), the Provation production environment experienced a 31-minute degradation affecting customer-facing operations. During the peak degradation window (14:51-15:20 UTC). The incident manifested as intermittent failures across multiple services dropping  customer-facing success rates to 81% during the worst minutes.

Auto-scaling triggered at 14:00 UTC (8:00am CST) provided initial capacity expansion. Operations executed additional mitigation measures  and the system achieved gradual improvement through 15:21 UTC (9:21am CST) and reached full recovery at 15:22 UTC (9:22am CST) when customer-facing success rates returned to 98%+.

No data loss occurred, but clinical documentation workflows experienced intermittent failures during the incident window. Primary failure modes included server errors on shared APIs.

Response

The Development team coordinated mitigation activities to stabilize platform performance. This effort included isolating the affected service paths, adjusting system resources to support essential workflows, and continuously monitoring platform behavior to validate the most effective mitigation steps. Responsiveness, error patterns, and system health indicators were tracked to guide decision making during recovery. As mitigation actions took effect, performance improved which lead to restoration by 15:15 UTC.  

Follow-up Tasks

  • Split VideoAPI onto independent ASP-DevOps Team

  • Evaluate VNet Integration + Service Endpoints for Video API/Document API-DevOps Team

  • Evaluate private endpoints for Storage-DevOps Team

  • Evaluate connection pooling refactor-Development Team

  • Evaluate port monitoring and alerting thresholds-DevOps Team

  • Evaluate circuit breaker implementation for video endpoints-Development Team

  • Evaluate rate limiting for video uploads and vitals polling-Development Team

  • Evaluate gradual re-enable plan for disabled tenants-Development Team
Posted Mar 04, 2026 - 13:11 CST

Resolved

Provation Apex has fully recovered. We apologize for the inconvenience.
Posted Feb 24, 2026 - 09:26 CST

Investigating

Provation Apex is currently experiencing a partial outage, resulting in some users finding it unreachable. Investigation is underway. New information will be posted here as it becomes available.

Please reference this link for Apex Desktop App offline mode instructions -

Offline Mode Instructions

Click the "Subscribe to Updates" button on this page to get email updates sent to your inbox whenever a change is made to this page.
Posted Feb 24, 2026 - 09:08 CST
This incident affected: Provation Apex.