Summary

Following the conclusion of the July 21, 2024 scheduled maintenance, at 00:00 MT Versaic by Benevity was unavailable to users for approximately 7 hours. Further intermittent service degradation extended through to 16:00 MT on July 22.

Impact

All Versaic users were impacted for the duration of the initial outage. During the initial 7 hours, users were unable to access Versaic.

For the period from July 21 07:00 MT to July 22 16:00 MT users may have intermittently experienced slower performance and the inability to complete payments.

Root Cause

The root cause is in two parts:

1. During scheduled maintenance of a portion of the systems supporting the Versaic application, that component was put into a state of failure by an incompatibility in the code supplied by the vendor and the configuration that Benevity has on that component.
2. Benevity teams implemented a recovery process to remediate the previous failure. The recovery process solved for the failure of the system. However, during the execution of the recovery process, the configuration of that system was changed such that a portion of the application (primarily the payments integration to Benevity’s platform services) was no longer able to interact with that infrastructure.

Note: This was not related in any way to the global CrowdStrike outage the previous week. Benevity was not impacted by the CrowdStrike outage.

Future Mitigation

1. We are adjusting our scheduled maintenance for this infrastructure to include additional validation that would allow us to detect and remediate similar failure conditions before the system enters a failure state.
2. The configuration change made to fix the post failure system interaction is more resilient to failure than the previous configuration. This should reduce the impact should a similar failure occur in the future.

Timeline of Events

July 20, 2024

• 18:00 MT - Start of Benevity maintenance window
• 21:40 MT - Versaic health checks fail after system maintenance completes; troubleshooting begins

July 21, 2024

• 00:00 MT - Benevity maintenance window ends; start of SEV1 incident
• 06:41 MT - Service is no longer failing; maintenance page removed; repair replication started
• 10:08 MT - Reports of intermittent payments failures received and confirmed
• 14:00 MT - Fix applied for some of the payments failures; pending confirmation

July 22, 2024

• 05:22 MT - Determined that not all payment processes were fixed; further troubleshooting begins
• 12:04 MT - Applied further remediation to solve for payments issues; received confirmation of successful payments
• 13:38 MT - Payments failures detected after extended period of no failures
• 14:57 MT - Determined that a single client facing server was failing; others were successful. Server was removed from serving new requests.
• 15:59 MT - No failures detected after 14:57. Declared the incident remediated, and entered extended monitoring state.

Jul 23 2024

• 14:37 - No failures detected since 14:57 the previous day. Declared the incident resolved.