File Locking and Shared Resource Contention in Worktree Swarms
Git worktrees isolate checkouts, but swarms still need locks.

- Written by
- Tobias Orel
- Updated
- November 21, 2026
- Reading time
- 8 min read
- Sources cited
- 3 sources ↓
What this covers
Git worktrees separate agents’ files, but they do not make a shared repository safe for unrestricted concurrent operations. Reliable swarms require coordination at three distinct layers, namely registry provisioning, runtime lock management, and shared runtime resources. File locking and resource contention must be handled at each layer because isolation at one does not protect the others.
Why worktrees alone are not enough to run reliable agent swarms
Each agent can have a dedicated working directory and index without duplicating the repository. Standard git branches provide the integration path, while a shared object store keeps the arrangement economical. Teams have described how they run multiple Claude agents at the same time, each on a different feature branch in its own worktree. The appeal is straightforward and structurally sound. Agents can edit separate checkouts without silently overwriting one another’s files, leaving conflicts for standard git tooling to detect at merge time.
The object store, worktree registry, and ref files are still shared, and locks control writes to parts of that state. Separate indexes therefore remove one source of interference without eliminating repository-wide contention. The remaining failures follow the worktree lifecycle. Provisioning can race before an agent starts, runtime locks can outlive their owners, and cleanup can fail after removing the metadata needed for recovery.
How git's file-based locking model was designed, and what breaks when agents stress it
Git uses file-based locks, including .git/index.lock and .git/config.lock, to serialize writes to repository state. This model suits sequential, human-paced operations, but concurrent agents place sustained pressure on the same coordination points. Each worktree has its own index, so an index lock error does not necessarily mean that agents in separate worktrees are competing for one index. It can instead indicate a lock left behind by a crashed process in that worktree.
Agents in different worktrees can contend over loose ref updates, ref packing, object creation, and the worktree registry. These operations reach beneath the separate working directories into common repository state. Directory separation cannot serialize those writes for the caller. The structural mismatch is therefore not that worktrees fail to separate checkouts, but that checkout isolation is being asked to provide coordination it does not supply. Understanding which resource a lock protects is the first step toward choosing a response that can succeed.
The worktree registry is not safe for concurrent writes
The .git/worktrees/ registry is a shared data structure whose mutations are not atomic. During git worktree add, git reads the registry entries that are already there while it creates the new entry. A concurrent add can encounter a sibling directory before its commondir file has been written. The directory exists, but its metadata is incomplete. Reading that intermediate state can produce a fatal error before either agent begins its assigned work.
codeArbiter fixed this race by using a dedicated worktreeChain lock for creation, removal, and prune operations. The registry lock is separate from the merge lock because the two protect different operations with different durations. Changes to the registry take little time, but merges can hold locks much longer. Combining them would delay worktree setup behind unrelated merges without improving correctness. Retry backoff runs outside the registry lock so that a worker waiting to try again does not block other workers from provisioning.
Prune is particularly dangerous when it escapes this serialization. It deletes sibling metadata directories that a concurrent add may be scanning. Every writer to the registry needs to use the same lock. An orchestrator that launches concurrent git worktree add calls without it remains vulnerable regardless of how carefully it separates agents’ working directories.
Stale lock files from crashed agents block all subsequent writes in that worktree
A crashed process can leave an index lock indefinitely. Subsequent writes in that worktree then fail even though no process remains to release the lock. If instructions classify every index lock error as contention and require agents to retry without deleting anything, the agent can loop indefinitely.
Exponential backoff is appropriate when concurrent operations are genuinely competing, because the current owner can finish and release its lock. Stale-lock recovery answers a different question. The corrected heuristic is to remove a lock that survives a retry when no git process is running in that worktree, and otherwise report its path to the user.
A killed git process can leave objects/info/commit-graph.lock or objects/pack/multi-pack-index.lock, and the corresponding write commands still fail despite an empty process table and old locks. Index-lock handling alone does not cover these operations. An agent may also exit after a commit fails because of a lock, leaving changes uncommitted. Automatic cleanup can then permanently destroy work that the agent never successfully recorded.
Finalizing and pruning worktrees under concurrent load leaves half-removed, irrecoverable state
Finalization can fail after dismantling the worktree’s git linkage but before removing its files. In the documented failure, finalize removes the .git pointer file and the admin directory under <anchor>/.git/worktrees/<id>, then fails to remove the tree because concurrent operations are competing for locks. A commit-graph-chain.lock held by a fetch in the shared clone is one example. The remaining directory is no longer a valid worktree. Management state can still report it as active, even though its git metadata is gone.
Re-running finalize cannot repair that state through the normal path because the operation no longer has a valid git worktree to act on. The documented rule against deleting managed worktrees by hand leaves the operator with repair work. A safer finalizer would acquire the required git locks with backoff before changing anything, or remove the tree before deleting its git linkage and registry entry. Recording a finalize-incomplete state would also let a later run complete idempotently.
Prune adds a related lifecycle hazard for managed worktrees whose agents are temporarily idle. Manual, scheduled, and orchestrator-triggered prune operations all need to be considered. The mitigation is to mark every managed worktree with the .git/worktrees/<id>/locked file so prune skips it. This protects the directory from removal, while application-level ownership must still control who may use it.
Application-layer ownership locks fill the gap that git's own locking leaves
Git does not provide an agent-level claim of exclusive worktree access. An orchestrator’s WorktreeManager.create() can find an existing worktree for a branch and reuse it without detecting that another process is already using it. Two callers can consequently receive the same directory. The existence check answers whether the worktree is present. Ownership requires a separate atomic decision.
mattpocock/sandcastle issue #427, from April 2026, describes creating an ownership lock with O_EXCL. Exactly one of two concurrent acquire() calls succeeds, while the other receives a contention error immediately. The lock records the PID, branch, and acquisition timestamp. Those fields show who owns the lock and help recover from a stale lock. A liveness check through process.kill(pid, 0) allows a dead owner’s lock to be removed before ownership is reacquired atomically.
The agent-locks MCP server stores lightweight coordination files under the repository’s shared .git/ directory, where every worktree can see them and git cannot commit them. Provisioning needs a different scope of exclusion. A per-repository advisory flock can serialize the entire worktree add and hooks-wiring critical section, then release before the long-lived agent starts. Unlike a fail-fast ownership claim, this lock queues setup work without serializing whole agent sessions.
Augment Code’s Intent implements ownership and sequencing at the orchestration layer. Each Space creates a dedicated git branch and worktree, and specialist agents execute in parallel within isolated directories. The Coordinator manages merge sequencing after validation. This places coordination above individual git commands, where the lifecycle can be managed. It is a distinct architectural role from the locks git uses to protect individual repository writes.
Git worktrees do not isolate what runs inside them — only what files they contain
Worktrees are a filesystem primitive. Agents in separate checkouts can still reach the same databases, ports, caches, credentials, and shared services if their permissions allow it. They can also execute commands outside their assigned directories. This contention layer therefore requires mechanisms beyond registry serialization and git lock recovery.
Parallel workers sharing a global SQLite database encountered SQLite WAL errors. The fix was to set XDG_DATA_HOME to a per-worker directory, giving each worker its own database instance. Git had no mechanism to provide that separation. The collision arose from where the running application stored its state. Correcting the environment removed the shared runtime resource that worktree isolation had left untouched.
Filesystem confinement introduces a different boundary problem. Git operations that rewrite the shared packed-refs file must write outside the linked worktree directory. If agent permissions allow writes only inside that directory, those operations can fail silently or produce opaque errors. Stricter confinement can therefore obstruct legitimate repository maintenance while still requiring separate decisions about service isolation. Worktrees alone cannot give agents their own databases, development servers, or ports.
A layered mitigation strategy matched to each contention point
Layer 1, registry provisioning, requires a single per-repository lock around git worktree add, remove, and prune. That lock should remain separate from merge and agent ownership locks so that short registry mutations do not queue behind long-running work. Prune must participate in the same serialization. Managed worktrees also need git lock files to protect them from scheduled or manual pruning outside the normal agent lifecycle. Finalization should take the required locks before it deletes anything, remove git linkage last, and record how far it got so a run that stops can resume.
Layer 2, runtime lock management, starts with distinguishing live contention from dead ownership. Backoff belongs to the first case, while stale-lock detection and removal belong to the second. Ownership locks created with O_EXCL and backed by PID tracking give orchestrators an atomic way to claim worktrees and recover abandoned claims. Prompt guidance needs the same distinction. Instructions to retry and never delete must not turn a stale lock into an endless loop, and object-store locks need recovery handling separate from index locks.
Layer 3, runtime resource isolation, needs each agent to have its own environment state, which more git locking cannot provide. Setting XDG_DATA_HOME, XDG_RUNTIME_DIR, and equivalent variables to per-agent directories separates databases, caches, and sockets that would otherwise collide. Shared async resources, including event loops and database connections, should go through one runtime that coordinates their use. Agents that need their own services require containers. Permission boundaries must also allow the shared .git writes that required git operations perform, or the workflow must accept the constraint and operate without ref packing.
Intent centralizes provisioning, ownership, and merge sequencing, with the Coordinator managing task delegation and the spec lifecycle, Implementors working in isolated worktrees, and the Verifier validating output before merge. That arrangement addresses the coordination responsibilities of Layers 1 and 2 without requiring each team to build the serialization and ownership logic independently. Runtime separation remains a distinct responsibility. The architectural choice is how to manage that full lifecycle without mistaking an isolated checkout for an isolated execution environment.
Methodology & sources
- glama.ai
Described the agent-locks MCP server that keeps lightweight coordination files in the shared .git directory.
- cflx 0.6.322
Covered index lock handling, including how stale index locks left by dead processes are detected and distinguished from live contention.
- 9 Open-Source Agent Orchestrators for AI Coding (2026)
Supplied the description of Augment Code's Intent, with per-Space worktrees, a Coordinator that sequences merges, and a Verifier that checks output.