Summary
Between December 15 and December 17, 2025, an incident occurred on our Managed File Transfer (MFT) system that resulted in the accidental deletion of critical payroll folder structures. This impacted two primary workflows: the receipt of Success Files from clients and the delivery of Deduction Files to clients. While most data was restored via backups, a small window of data was unrecoverable, and some deliveries were delayed.
Impact
The incident created two distinct challenges for our clients:
- Issue 1: Payroll Success Files - Due to a gap between the deletion and our last system backup, 28 files uploaded by clients between 12:08 AM and 2:51 PM on December 15 were lost. These clients were contacted directly to resubmit their data.
- Issue 2: Payroll Deduction Files - The deletion prevented the system from automatically generating deduction files for some clients. By December 17, our teams manually recovered and redelivered all affected files to ensure client payroll deadlines were met.
Root Cause
During a routine administrative cleanup of inactive user accounts on the MFT system, an unintended cascading deletion occurred.
The investigation revealed a hidden dependency: when specific administrative user profiles were removed, the system automatically deleted the directory structures associated with those profiles. Unfortunately, these directories housed critical payroll folders. Because this "link" between the user account and the production folder was not explicitly flagged by the system, the deletion was not caught until automated processing errors were reported.
Future Mitigation
- Increased snapshot frequency for critical payroll directories to reduce the recovery window.
- All account cleanups now require a peer-review protocol and a dependency check.
- Implement monitoring and alerting for bulk file/folder deletions in payroll directories.
Timeline of Events
- Dec 15, 12:08 AM: Accidental deletion occurs during routine account maintenance.
- Dec 16, 9:15 AM: Incident detected and response teams engaged.
- Dec 16, 2:30 PM: Core folder structure restored from snapshots; 15-hour data gap identified.
- Dec 17, 12:52 PM: Deduction file folders restored; manual re-generation of files begins.
- Dec 17, 6:00 PM: All 67 missing deduction files successfully delivered to clients.
- Dec 18 – Jan 20: Direct outreach to the affected clients to coordinate success file resubmissions.
- Jan 25, 2026: Incident officially closed following confirmation of all data reprocessing.