Summary

A new cause search functionality for Spark was introduced in December 2021. This change introduced a non-optimized query whose performance was immaterial under normal system load. As Giving Tuesday (Nov 29th) volume rapidly increased around 8:26am MST, this query pushed search performance over the limit causing extensive degradation across multiple pages for 133 minutes.

Impact

All Spark clients started experiencing extremely slow response times on the Cause Search and Cause Profile pages.  While these pages were not completely unresponsive, response times significantly degraded Spark's ability to fulfill our clients' usability for over two hours. 

Root Cause

Spark's cause search engine capacity was exhausted due to non-optimized queries, and load test coverage failed to identify the issue. Initial attempts to add additional capacity to the search service while under load were unsuccessful which contributed to a longer time to remediate the issue.

Future Mitigation

• Optimize the inefficient query so that it is performant under increased load
• Increase load test coverage to capture additional use cases around searching for causes
• Improve processes and training around usage of search queries optimization and how to use them

Timeline of Events

• 08:26 AM MT - Initial alert received
• 08:31 AM MT - Major contributing cause identified
• 08:34 AM MT - Initial attempt to add additional search capacity
• 09:55 AM MT - Disabled donation progress bar display on Spark for all clients to reduce load
• 10:22 AM MT - Extra Spark's Cause Search capacity added to allow the system to catch-up on pending requests
• 10:40 AM MT – Major Client impact solved, system back to acceptable performance
• 01:04 PM MT - Incident resolved, systems fully operational