Troubleshooting UnixLocal Workspace I/O and Cancellation Safety

Why workspace I/O is an event-loop correctness problem

The event loop is the core of an asyncio application. It runs asynchronous tasks and callbacks, network I/O operations, and subprocesses. That makes the placement of workspace operations a correctness concern when those operations can perform substantial filesystem work.

The UnixLocal implementation addresses this boundary by routing selected workspace operations through worker-based I/O rather than performing all of their work directly in the asynchronous method.

Failure timeline: recursive removal

Recursive directory removal is one distinct workspace-I/O path. The UnixLocal implementation routes recursive directory removal through a worker instead of calling shutil.rmtree directly in the async method.

The relevant invariant is narrow: recursive removal belongs in the worker path. This is separate from the archiving and extraction paths, which have their own synchronous filesystem work.

Failure timeline: archiving

Workspace archiving is another distinct path. The UnixLocal implementation performs workspace archiving in a worker by wrapping tarfile.open and tar.add in a synchronous function passed to run_blocking_workspace_io.

That wrapper keeps the archiving operation associated with the worker-based workspace-I/O helper. It is not the same implementation boundary as recursive removal: archiving has its own synchronous function and its own tarfile operations.

Failure timeline: extraction

Workspace extraction has a third distinct path. The UnixLocal implementation performs workspace extraction in a worker by wrapping root creation, tarfile.open, and safe_extract_tarfile in a synchronous function passed to run_blocking_workspace_io.

The extraction invariant therefore covers all three operations in that synchronous function: creating the root, opening the archive, and safely extracting it. Moving only one of these operations would not be the same as the documented implementation.

Act I fix: route each workspace operation through worker-based I/O

The implementation separates the three workspace operations while giving them the same event-loop-avoidance treatment:

• recursive directory removal is routed through a worker;
• archiving wraps tarfile.open and tar.add in a synchronous function passed to run_blocking_workspace_io;
• extraction wraps root creation, tarfile.open, and safe_extract_tarfile in a synchronous function passed to run_blocking_workspace_io.

These are three implementation paths, not one generalized filesystem rule. A review or regression check should preserve each path explicitly.

Why cancellation creates a separate worker-ownership boundary

Keeping workspace work in a worker does not by itself describe what happens when the caller is cancelled. Cancellation introduces a separate boundary: the relationship between the caller's cancellation and the worker task that is already performing the operation.

The helper's behavior is explicit. run_blocking_workspace_io starts asyncio.to_thread as a separate task, continues waiting when the caller receives asyncio.CancelledError, and retrieves the worker task result before re-raising the cancellation.

This is a worker-ownership invariant. The cancellation is propagated only after the worker task result has been retrieved. It should be reviewed separately from the question of whether workspace I/O was routed away from the event loop.

Failure timeline: cancellation before the worker result

The cancellation timeline has two participants: the caller and the worker task created for asyncio.to_thread.

1. run_blocking_workspace_io starts asyncio.to_thread as a separate task.
2. The caller receives asyncio.CancelledError while the worker task exists.
3. The helper continues waiting rather than immediately finishing cancellation handling.
4. The worker task result is retrieved.
5. The cancellation is re-raised.

The important boundary is between steps three and four. The helper does not re-raise cancellation before retrieving the worker task result.

Cancellation-safe invariant: wait before re-raising

The cancellation-safe rule is therefore distinct from event-loop avoidance: after cancellation is received, run_blocking_workspace_io continues waiting for the worker task and retrieves its result before re-raising asyncio.CancelledError.

This rule does not replace the three Act I routing rules. It governs ownership of the worker task after cancellation has been delivered to the caller.

Regression shape: observe extractor completion during cancellation

A separately executed cancellation regression test records extractor start and end around cancellation, asserts asyncio.CancelledError, and confirms that the archive stream remains open.

That test shape checks the cancellation boundary directly. It observes the extractor's interval, verifies the caller-facing cancellation, and checks the archive-stream state. It is distinct from a test that only checks whether removal, archiving, or extraction uses the worker path.

Verified regression result

At commit 494ea5978778a4daa48b513419dc5786084802d4, the cancellation regression test selected one hydrate_workspace_cancellation test and passed, with 14 tests deselected.

This is a bounded test result for that commit and selection. It is not a production performance benchmark or a general claim about other runtimes or repositories.

Operational checklist

Review the two correctness boundaries separately:

1. **Event-loop avoidance:** recursive removal is routed through a worker; archiving wraps tarfile.open and tar.add in a worker function; extraction wraps root creation, tarfile.open, and safe_extract_tarfile in a worker function.
2. **Cancellation-safe ownership:** run_blocking_workspace_io continues waiting after asyncio.CancelledError, retrieves the worker task result, and only then re-raises cancellation.
3. **Regression evidence:** the cancellation test observes extractor start and end, asserts asyncio.CancelledError, confirms that the archive stream remains open, and has a recorded passing execution at the specified commit.

Treating these as separate checks prevents the worker-routing invariant from being mistaken for the cancellation-ownership invariant.