Requests to some Spark URL paths are responding with HTTP 403
Incident Report for Benevity
Postmortem

Summary

Between Mar 23, 2023 8:20am MT and Mar 24, 2023 12:10pm MT, Spark users may have encountered an error when attempting to access certain pages related to HR, profile and tax receipt information. The issue was initially identified and raised through client success channels, and then was quickly escalated to the engineering team. 

Investigation determined the cause to be the result of an unanticipated behaviour change related to a recent security update of a third party service. Benevity's Engineering team were able to develop and successfully test a software patch to accommodate the introduced behaviour change, which was then deployed through all of Benevity's environments. After deployment, access and functionality of the HR, profile, and tax information pages was restored for all Spark users.

Impact

During the incident, Spark users were unable to access certain pages related to HR, profile and, tax receipt information between Mar 23, 2023 8:20pm MT and Mar 24, 2023 12:10pm MT.  There was no impact to user logins or donation flows, and all other pages were accessible.

Root Cause

As part of Benevity's commitment to security, all of our systems are updated and patched on a regular cadence to ensure they comply with, or exceed, the latest published security baselines. 

A security update for a third party service, deployed on Mar 23, introduced a change in behaviour specific to site URLs with space characters, resulting in the HR, profile, and tax receipt information pages being inaccessible. 

This update was made in accordance with our standard change management procedures, but the behaviour change it introduced was not caught in the automated test suite as URLs containing space characters is not a common pattern used throughout Spark and therefore not explicitly tested.

Future Mitigation

  • Spark code to be updated to remove spaces from URLs
  • Additional tests to add coverage to missing areas

Timeline of Events

  • Mar 23, 2023 08:20 - First report of the issue
  • Mar 23, 2023 08:52 - Issue escalated to engineering & investigation began
  • Mar 23, 2023 11:52 - Cause identified
  • Mar 23, 2023 12:15 - Patch produced & testing begins
  • Mar 24, 2023 13:03 - Release deployed & Issue mitigated
  • Mar 24, 2023 13:10 - Systems fully operational
Posted Mar 31, 2023 - 16:34 MDT

Resolved
This incident has been resolved.
Posted Mar 27, 2023 - 09:47 MDT
Monitoring
The fix has been deployed and verified to be working. We will monitor for the next couple of hours to ensure the fix is behaving as expected.
Posted Mar 24, 2023 - 13:12 MDT
Update
The identified fix is going through the release process and will be going live in the near future
Posted Mar 24, 2023 - 08:47 MDT
Update
Artifacts containing the fix for this issue are building. We will post an update when the fix will be going live.
Posted Mar 23, 2023 - 13:44 MDT
Update
The fix for this issue has been verified and will be deployed shortly
Posted Mar 23, 2023 - 12:24 MDT
Identified
The issue has been identified and a potential fix is being tested in lower environments.
Posted Mar 23, 2023 - 12:11 MDT
Investigating
We are currently investigating an issue where requests to some Spark URL paths are responding with HTTP 403
Posted Mar 23, 2023 - 12:00 MDT
This incident affected: Benevity Spark.