Spark sites not fully loading the dashboard

Summary

On September 21st, 2023 from 2:56pm MDT to 3:41pm MDT, some Spark sites experienced issues with the dashboard fully loading as portions of the dashboard were missing from their view.

Impact

The outage affected a subset of users when trying to access content within Spark. This issue started September 21st at 2:56pm, and was remediated by 3:41pm. Some users attempting to access Spark would have been unable to see all the various artifacts displaying on the dashboard.

Root Cause

During a deploy there was an issue with previously cached information preventing the latest code from running as expected.

Future Mitigation

A process change is being put in place to create additional checks for issues that might require backwards compatibility.

Timeline of Events

• 02:56pm MT - Latest code released into production
• 03:19pm MT - Initial alert received via PagerDuty about some sites seeing issues
• 03:30pm MT - The fix was identified and was initiated to resolve issue
• 03:41pm MT - The process to fix completed and sites were verified as loading correctly
• 03:41pm MT - Incident declared resolved; systems fully operational