Two concurrent work units writing the same (workspace, observer, observed)
collection deadlocked on times_derived reinforcement UPDATEs issued in
batch order (DEV-1975, 682 events in 90 days). The deadlock was swallowed
per-document, the loop cascaded PendingRollbackErrors against the dead
session, the whole batch was lost, and the queue item was marked processed.
- serialize writers per collection with a transaction-scoped advisory lock
(pg_advisory_xact_lock + SET LOCAL lock_timeout), skipped for insert-only
batches; covers all three row-lock sites in one move
- hoist external-vector-store dup-candidate resolution ahead of the first
DB statement so the lock's critical section contains no network calls
- abort the batch on SQLAlchemyError instead of continuing through an
aborted transaction; per-document skip semantics kept for non-DB errors
- classify transient errors (new src/utils/retryable_errors.py) and retry
them via a bounded in-process counter instead of marking items errored
Aakash Kattelu