Spark users are experiencing intermittent PayPal Subscription issues

Summary

On Tuesday, September 26th, our production support team discovered several errors with Benevity's API processing of PayPal instant payment notifications (IPNs), which prevented these transactions from being recorded in our donation flows and made them ineligible for disbursement.

Impact

A recent PayPal update caused payment IPNs to be missing critical information needed to complete donation transactions on Platform. As a result, 187 PayPal subscriptions processed on September 27th failed. Spark users would not be able to see their donations in their donation history, nor would they receive donation receipts (for tax-deductible donations). The PayPal errors only affected a subset of transactions (failure rate ~4%), and failures occurred across multiple merchant accounts/foundations and clients.

Root Cause

A recent PayPal release introduced a problem with payment IPNs missing critical information required for the completion of transactional donation flows on Platform.

Timeline of Events

• September 26 14:02 MT - Initial alert received.
• September 26 14:02 MT - Issue was escalated to vendor (Paypal) via support ticket
• September 27 09:36 MT - PayPal support escalated internally to their development team.
• October 25 14:37 MT - PayPal confirmed it refunded all transactions identified.
• October 25 14:37 MT - Incident resolved; systems fully operational.