Work That Outlives the Process

Architecture · July 2, 2026 · 9 min read

A request that takes two hundred milliseconds lives entirely inside one process, one code version, one machine. Almost every convenience of ordinary programming holds: local variables persist, sleep works, the call stack means something, and if the process dies the client simply retries.

Extend that to three weeks and every one of those conveniences fails. The process will be replaced several times. The code will change underneath the running instance. The machine will be recycled. A local variable is now a promise you cannot keep, and sleep is a bet that nothing will be deployed for half a month.

Long-running workflows are hard for one reason with many consequences: the computation outlives its host.

State has to live somewhere you can lose the process

The first consequence is that the workflow’s position — which steps ran, what they returned, what it is waiting for — cannot live in memory. It has to be written down, after every meaningful transition, somewhere durable.

This sounds obvious and is routinely half-done. A common shape is a workflow whose data is persisted but whose control flow is not: the records are in the database, but the knowledge that step four has completed and step five is next exists only in the running program. Kill it and the data survives while the progress does not, and recovery means a human working out where it had got to.

The test is blunt. Terminate the process at an arbitrary point. Can something else pick up every in-flight instance and continue, without a person deciding where to resume? If not, the state is not durable, whatever the database contains.

Timers are state, not sleeps

Long workflows wait. A trial ends in thirty days. A supplier has five working days to confirm. A machine needs recalibrating in six months. Escalate if nothing happened in forty-eight hours.

Every one of those is a durable timer, and the way to get it wrong is to implement it in the process — a sleeping thread, a delayed task in an in-memory scheduler, a background timer object. These evaporate on restart. Worse, they evaporate silently: nothing errors, the wait simply never ends, and the workflow sits in a state that looks like patience.

A durable timer is a row: this instance becomes runnable at this moment. Someone sweeps for due rows and wakes the instances. The mechanism is unremarkable, and it is the difference between a wait that survives a deployment and one that does not. Cloud queues offering delayed delivery cover the short end of this well; for waits measured in months, a table you can query and audit is easier to reason about, because you can answer “what is waiting, and for what” without asking a broker.

Deploying into a running population

Here is the problem that has no clean answer. Instances started last month are running the code path of last month’s version. You are about to deploy this month’s, which adds a step, renames another, and changes what step three returns.

Three approaches exist and each has a real cost.

Let old instances finish on old code. Cleanest semantically, most expensive operationally: you are now running multiple versions concurrently and cannot retire any of them until the last instance drains. With month-long workflows that can mean many versions alive at once.

Version the definition and pin each instance. The instance records which definition it started under, and the engine keeps the old definitions available. This is what mature workflow engines do, and it turns the problem into one of code hygiene: definitions accumulate and someone must eventually prune them.

Migrate in-flight instances. Write a migration that maps old positions to new ones. Sometimes trivial, sometimes impossible — there is no correct answer for an instance sitting in a step that the new version deleted. Reserve this for changes that are genuinely additive, and accept that a subset may need to be finished manually.

What does not work is deploying and hoping. The failure surfaces days later as instances stuck at a step name that no longer exists, and by then the population is large.

Determinism as a constraint you did not ask for

Durable execution engines that recover by replaying history impose a rule that surprises people: the workflow code must produce the same sequence of decisions when replayed. Read the clock directly, generate a random value, or query a database from inside the workflow body, and the replay can take a different branch than the original run — at which point the recorded history and the code disagree, and the engine has no way to reconcile them.

The discipline that follows is to push all non-determinism into recorded steps. The current time comes from the engine, not the system clock. Random values are generated inside a step so the result is written into history. External reads happen in steps. The workflow body itself becomes pure orchestration: it decides what to do next based only on what history says already happened.

This is a genuine restriction and it is the main reason teams bounce off durable execution frameworks on first contact. It is also the property that makes recovery reliable rather than approximate.

Cancellation is a first-class feature

Short work does not need cancellation; you wait for it. Work spanning weeks absolutely does. The customer withdrew. The order was replaced. The asset was decommissioned. Something must stop the instance, and stopping it is more than deleting a row.

A running instance may hold a lease, occupy a slot in a rate limit, have an outstanding request with a third party, and be scheduled to wake in nine days. A proper cancellation releases the lease, cancels the timer, and decides what to do about work already dispatched — which is usually a compensating action rather than a deletion.

Building this after the fact is painful, because by then the instance’s external effects are scattered across steps with no inventory of them. Recording each external effect as it happens, in the instance’s own history, is what makes cancellation and cleanup tractable later.

The honest summary

Nothing here is conceptually deep. It is a list of properties you get for free in short-lived code and must pay for explicitly in long-lived code: durable position, durable timers, versioned definitions, deterministic decisions, explicit cancellation.

The reason teams write their own workflow engine — and most do, at least once, usually calling it a status column and a cron job — is that the first version of this is genuinely easy. The cost arrives when instances live long enough that deployments happen in the middle of them. That is the point at which the status column stops being sufficient, and it always arrives later than the decision that created it.