Post-incident review
Incident: Checkout unavailable for 47 minutes
Date: 11 October 2027
Severity: High
Incident lead: Hamza Idrissi
Review written by: Anna Berg
Review date: 14 October 2027
This review is blameless. Its aim is to learn and to improve systems and processes, not to blame people.
Summary
On 11 October from 14:03 to 14:50 customers could not pay. A database change filled the disk and the checkout service stopped.
Impact
About 310 failed payment attempts. Estimated lost sales of EUR 9,000. 42 support tickets. No data was lost.
Timeline
14:03 Alert for failed payments
14:08 On-call engineer starts looking
14:21 Cause found: disk full
14:38 Old logs removed, service restarted
14:50 Payments working, incident closed
Causes
Direct cause: the disk reached 100 percent.
Underlying causes: no alert for disk use, log files were never rotated, the change was not tested with real data volumes.
What went well
The payment alert fired in 3 minutes. The team worked together and kept customers informed on the status page.
What went badly or was lucky
Finding the cause took 18 minutes because the dashboard did not show disk use. The runbook was out of date.
Follow-up actions
| Action | Owner | Due date |
|---|---|---|
| Add a disk alert at 80 percent | Hamza Idrissi | 21 October |
| Set up log rotation | Daniel Roche | 21 October |
| Update the runbook | Anna Berg | 28 October |
Lessons for others
All services should have disk alerts and log rotation.