Resolved — monitoring
Post-incident review: automated issue delivery outage
Incident apq-production-2026-08-29 · KDTIX Agent Project Queue · known since on or before 29 August 2026 · service restored 2 September 2026 22:22 UTC. This is the customer-facing summary; the full review with dated preventive actions is being completed by the recovery team and this page will be updated as items land.
What customers experienced
From 29 August the APQ web application was unreachable and automated issue delivery (worker → independent review → merge → closure) was paused. Automated delivery had in fact been stalled since 22 August by an internal scheduling fault that was only visible in service telemetry, so no customer work was delivered in that window. No customer action was required and no customer data was affected.
What happened
- A protective maintenance hold, applied correctly after an earlier data-consistency finding, stopped the application from restarting cleanly after a deployment step failed; a host maintenance window and a container-runtime fault extended the outage.
- Independently, a stale recovery record for one work item had been blocking the scheduler from dispatching any work since 22 August, without raising an alarm.
- Recovery required releasing the hold, clearing the stale record, and correcting several scheduler states by hand under explicit operator authorization; the first work item was then delivered end to end on 2 September.
Preventive actions
| Action | Target |
|---|---|
| Alarm when the scheduler is armed with queued work but nothing dispatches, and bound any internal dispatch pause. | Design by 2026-09-09; delivery scheduled with the September plan |
| Reconcile worker results durably after timeouts or lost receipts so a completed change never needs manual closure. | Design by 2026-09-09; delivery scheduled with the September plan |
| Sanctioned, audited recovery actions replacing hand edits of scheduler state. | Scoped by 2026-09-16 |
| Rehearse recovery steps on a canary project before production; validate provider output formats before dispatch. | Scoped by 2026-09-16 |
| Independent status page and outage notice on the application landing page (already live). | Done 2026-09-02 |
Support
A support landing page for incident reports and support requests is planned; until then, contact your KDTIX representative. Current service state: status.projectit.ai · machine-readable status.json.