Summary

As a result of a conflict with a third party plugin on November 16, 2022, users were unable to log into the Benevity Reporting site from 02:45 to 10:00 MT.

Impact

Users were unable to access the Benevity Reporting service for the duration of the outage period.

Root Cause

Benevity uses a third party monitoring solution that provides performance metrics for browser interactions. An update for that solution was in the process of being rolled out to services. That version of the solution had a conflict with the code that runs on the Benevity Reporting login page.

Additionally, our monitoring on the Benevity Reporting service does not include the login workflow for the application. As a result, our existing monitoring did not alert us to the failure because the service was never fully offline, which extended the downtime period.

Future Mitigation

• We have determined that the third party browser monitoring is no longer required for Benevity Reporting and Benevity Analytics. We are no longer running the browser monitoring solution for those services.
• We are communicating with the vendor about how updates to the monitoring service can be implemented so that we can better test updates before they are introduced to our production environment.
• We are investigating additional monitoring that would include a more thorough user path.