Versaic Quick Search functionality is not working
Incident Report for Benevity
Postmortem

Summary

The Quick search functionality for Versaic stopped working for some clients on Dec 18, 2023. The issue was reported by one of the clients on Jan 5, 2024. About 78 Versaic clients were unable to search for proposals using the 'Quick Search' feature on Jan 5, 2024 when the issue was reported. The workaround for searching for proposals was to use the 'Advanced Search' feature. The team fixed the issue for the reporting client by adding a new event for search indexing. A temporary fix for all other clients was implemented on Jan 8,2024 and a permanent fix was deployed on Feb 27, 2024.

Impact

About 78 Versaic clients were unable to use the 'Quick search' functionality in Versaic on Jan 5, 2024. The functionality stopped working for a few clients starting Dec 18, 2023. There was a workaround and these clients were able to search for proposals using the 'Advanced Search' feature. One of the client was unable to close the program cycle and the team had to manually close it.

Root Cause

The Versaic Quick search functionality stopped working because one of the jobs in the Scheduler service was getting stuck for unknown reasons and not completing. Since the release on Dec 18, 2023, there were about 78 Versaic clients for which newly submitted/created proposals were not indexed for search mechanisms.

Future Mitigation

A permanent fix for the issue was implemented on Feb 27, 2024. The team will continue to monitor and check logs post deployment to ensure the issue doesn't occur again.

Timeline of Events

  • Jan 5, 2024 04:15 AM MT - Issue reported for one of the clients.
  • Jan 5, 2024 12:56 PM MT - The issue for reporting client was fixed by adding a new event for search indexing.
  • Jan 8, 2024 07:00 PM MT -  Scheduler service restarted for all clients as a temporary fix.
  • Feb 27, 2024 07:00 PM MT - Permanent fix deployed to production; Incident resolved.
Posted Apr 30, 2024 - 18:35 MDT

Resolved
We have applied a permanent fix and not noticed reoccurrence of the issue in production. This incident is now resolved.
Posted Mar 26, 2024 - 17:34 MDT
Monitoring
We have applied some corrective action, but are monitoring for recurrence/further issues.
Posted Jan 10, 2024 - 11:20 MST
Investigating
We are currently experiencing a temporary disruption with the functionality of Quick Search.
Our technical team is actively investigating this issue to understand and resolve it as quickly as possible.
Posted Jan 05, 2024 - 10:50 MST
This incident affected: Versaic by Benevity (Versaic Production Application).