fix(actions): retry workflow insertion on database deadlock #221

2026-05-26T20:20:10Z

jmiller commented

2026-05-26 20:20:10 +00:00

Summary

Fixes Actions workflows silently not triggering for API-created PRs due to an InnoDB deadlock in InsertRun / UpdateRepoRunsNumbers
Adds db.IsErrDeadlock() for cross-engine deadlock detection (MySQL 1213, PostgreSQL 40P01, SQLite BUSY)
Wraps PrepareRunAndInsert in a retry loop (3 attempts, exponential backoff) with proper run-state reset between retries

Root Cause

When multiple workflows match a single event, each InsertRun transaction acquires:

An X-lock on the repository row (via UpdateRepoRunsNumbers doing SELECT count(*) FROM action_run)
An index lock on action_run (via the INSERT)

Two concurrent transactions deadlock when each holds one lock and waits for the other. InnoDB kills the lighter transaction, but handleWorkflows only logged the error and moved on -- silently dropping the workflow run.

API-created PRs were hit harder because they tend to be created during active CI, increasing lock contention. This matches upstream Gitea #36234.

Test plan

go build ./models/db/ ./services/actions/ -- compiles clean
go vet ./models/db/ ./services/actions/ -- no issues
TestIsErrDeadlock -- all 6 cases pass
Create PR via API on a repo with 2+ workflow files -- verify all workflows trigger
Create PR via web UI -- verify no regression

Closes #220

## Summary - Fixes Actions workflows silently not triggering for API-created PRs due to an InnoDB deadlock in InsertRun / UpdateRepoRunsNumbers - Adds db.IsErrDeadlock() for cross-engine deadlock detection (MySQL 1213, PostgreSQL 40P01, SQLite BUSY) - Wraps PrepareRunAndInsert in a retry loop (3 attempts, exponential backoff) with proper run-state reset between retries ## Root Cause When multiple workflows match a single event, each InsertRun transaction acquires: 1. An X-lock on the repository row (via UpdateRepoRunsNumbers doing SELECT count(*) FROM action_run) 2. An index lock on action_run (via the INSERT) Two concurrent transactions deadlock when each holds one lock and waits for the other. InnoDB kills the lighter transaction, but handleWorkflows only logged the error and moved on -- silently dropping the workflow run. API-created PRs were hit harder because they tend to be created during active CI, increasing lock contention. This matches upstream Gitea #36234. ## Test plan - [x] go build ./models/db/ ./services/actions/ -- compiles clean - [x] go vet ./models/db/ ./services/actions/ -- no issues - [x] TestIsErrDeadlock -- all 6 cases pass - [ ] Create PR via API on a repo with 2+ workflow files -- verify all workflows trigger - [ ] Create PR via web UI -- verify no regression Closes #220

jmiller added 5 commits 2026-05-26 20:20:11 +00:00

Merge pull request 'rc: fix dashboard icon' (#203 ) from rc/05.03.06 into main b7f9743ade

Merge pull request 'rc: footer help + login logo' (#206 ) from rc/05.03.08 into main 3c55a3baca

Merge pull request 'rc(v05.04.00): login notifications, help footer, login logo, checksum fix' (#210 ) from rc/05.04.00 into main 1e1441f8bd

Merge pull request 'rc(v05.05.00): org 2FA, wiki slugs, http render fix' (#219 ) from rc/05.05.00 into main c633024a9c

fix(actions): retry workflow insertion on database deadlock

Branch Policy Check / Verify merge target (pull_request) Successful in 1s

Details

PR RC Release / Build RC Release (pull_request) Successful in 2s

Details

Universal: Pre-Release / Build Pre-Release (${{ inputs.stability || 'development' }}) (pull_request) Failing after 21s

Details

dd6fc4b69c

When multiple workflows are triggered by a single event (e.g. a
pull_request with several matching workflow files), each InsertRun
transaction acquires an X-lock on the repository row via
UpdateRepoRunsNumbers and an index lock on action_run. Two concurrent
transactions can deadlock when each holds one lock and waits for the
other. InnoDB kills the lighter transaction, but handleWorkflows only
logged the error and silently dropped the workflow run — making it
appear as though pull_request events were never fired.

This was the root cause of API-created PRs appearing to not trigger
Actions workflows: the notification pipeline was correct, but the DB
insert was lost to an unretried deadlock.

The fix wraps PrepareRunAndInsert in a retry loop (up to 3 attempts
with exponential backoff) that detects deadlock errors across MySQL,
PostgreSQL, and SQLite. On deadlock, the rolled-back run fields are
reset before the next attempt.

Also adds db.IsErrDeadlock() for cross-engine deadlock detection and
unit tests for the same.

Closes #220

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

jmiller merged commit e8ce4ae60b into dev

2026-05-26 22:04:59 +00:00

Sign in to join this conversation.

Priority -

Type -

Status —

Priority —

Type —

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: MokoConsulting/MokoGitea-APP#221