Summary

On Sunday, October 2, 2022 at 6pm an internal certificate that we use to allow the versaic application access cloud storage solution had expired.  The certificate was updated as part of our routine maintenance.  However, a step was missed to give the application access to the certificate.  This was caught by our regression testing team on Monday morning, who informed SRE.  SRE was able to update the access to the certificate and access was restored on Monday at 09:16 am.

Impact

The application was available, however access to some attachments were unavailable.

Root Cause

An internal certificate had expired and was not replaced in time

Future Mitigation

We are adjusting our process to give a minimum of 30 days notice to update certificates.  We have an a process in place where deployment scripts will correctly set access so this is not missed.

We will also be adjusting our validation tests for the public storage service to better inform us if the service is available

Timeline of Events

Provide a high level timeline of the incident

• Oct 02, 2022: 18:00 MT - cdn.versaic.internal certificate had expired
• 21:00 MT - 2022-10-02 maintenance window commences
• 21:34 MT - certificate replaced on web and job nodes
• 21:37 MT - Inspec tests run showing the cdn login page was available and the app was started
• Oct 03, 2022: 08:07 MT - Regression testing show CDN has an issue
• 08:20 MT - SRE contacted via slack to about the issue
• 08:59 MT - SRE identifies the issue with the certificate
• 09:16 MT - fix is pushed out to impacted systems