Durable Execution for Voice and AI Agent Campaigns

Updated October 2026. Everything below was run on 19 October 2026 on a Linux VPS with Python 3.12.3, dbos 3.3.0, temporalio 1.34.0 and the Temporal dev server that package starts (CLI 1.9.1, server 1.32.0). I read the vendor documents quoted here on the same day. The carrier is a fake I wrote, so there is no real telephony, no LLM and no cost data in this post.

1. The problem: a campaign that dies at lead 17

An outbound campaign is a loop that can run for hours. For each lead it places a call, waits for an outcome, asks a model to classify the transcript, and sometimes sends a WhatsApp confirmation. Somewhere in that loop the process will die: a deploy, an out-of-memory kill, a host reboot. The question I wanted answered is what happens to the leads already dialed.

The obvious fixes are not enough, and I wanted numbers rather than opinions. So I built the smallest campaign I could and killed it on purpose. Forty leads, a fake carrier that appends one line to a log file for every real dial, and a hard os._exit(137) (a stand-in for a SIGKILL) right after the carrier records the dial for lead 17 and before the program records that it happened. Then I restarted the program and counted lines in the carrier log.

2. Why this is hard

There are three places a crash can land, and they behave differently:

1. Before the side effect. The call was never placed. Retrying is safe.
2. After the side effect, before the record. The call happened and nothing remembers it. A retry places it again. This is the window that matters.
3. After the record. Resuming skips the call. Safe.

Case 2 cannot be closed by the orchestrator alone, because the orchestrator cannot make a phone call and write a database row atomically. Whatever you choose has to be paired with something at the carrier boundary. I come back to that in section 6.

The second difficulty is the model. A classifier or an agent step is non-deterministic: ask twice, get two answers. Durable engines that re-run your workflow function after a crash need that function to make the same decisions the second time. A model call placed in workflow code breaks that, and I show it break in section 7.

3. Reference architecture

The shape is the same in every engine I tried. One function decides what happens next and contains no side effects. Separate units of work, called steps in DBOS and activities in Temporal, perform every side effect and have their outputs recorded. A state store holds those records. Something starts recovery after a crash. And the carrier gets an idempotency key it can check.

                 +-----------------------------+
                 |  workflow function          |   deterministic: loops, ifs,
                 |  for lead in leads: ...     |   no clock, no random, no I/O
                 +--------------+--------------+
                                | calls
          +---------------------+----------------------+
          v                     v                      v
   dial(lead)             classify(lead)         whatsapp(lead)
   side effect            LLM call               side effect
   key = wf_id:lead       output recorded        output recorded
          |
          v
   carrier checks key  --> same key seen? return the old result, do not dial

   state store (SQLite in my lab, Postgres in the DBOS docs) records
   workflow input, each step's output, workflow status
   recovery: restart the process (DBOS) or retry after timeout (Temporal)

Two rules fall out of the diagram. Anything that touches the outside world or a model goes in a step. Anything in the workflow function must give the same answer on a second run.

4. The lab

The fake carrier is the measuring instrument. Every real dial appends one JSON line, and a call with an idempotency key that is already in the log returns the old result without appending. Duplicate dials are then just lines minus distinct leads.

def place_call(lead_id, idempotency_key=None):
    seen = set()
    if idempotency_key and os.path.exists(LOG):
        for line in open(LOG):
            seen.add(json.loads(line).get("key"))
        if idempotency_key in seen:
            return {"lead": lead_id, "deduped": True}
    with open(LOG, "a") as f:
        f.write(json.dumps({"t": round(time.time(), 3), "pid": os.getpid(),
                            "lead": lead_id, "key": idempotency_key}) + "\n")
    return {"lead": lead_id, "deduped": False}

I ran six variants. Each one crashes once, at lead 17, between the carrier write and the record, then restarts and runs to completion. Forty leads means 40 dials is the correct answer.

naive-memory:        dials=58 unique_leads=40 duplicate_dials=18
naive-checkpoint:    dials=41 unique_leads=40 duplicate_dials=1
dbos(use_key=False): dials=41 unique_leads=40 duplicate_dials=1
dbos(use_key=True):  dials=40 unique_leads=40 duplicate_dials=0
temporal(use_key=False): dials=41 unique_leads=40 duplicate_dials=1
temporal(use_key=True):  dials=40 unique_leads=40 duplicate_dials=0

All six rows are observed in this run, with a single crash and a single trial each. The plain loop's 18 is simply leads 0 to 17 dialed twice. A file written after every lead has the same one-lead exposure window as a durable engine. What the engines add is not a smaller window. They add recorded step outputs, a status you can query, and resume-on-restart, which a hand-rolled file leaves for you to build.

5. The two implementations

DBOS

DBOS is a Python library. I decorated the three side-effecting functions as steps and the loop as a workflow. The line I care about is the key built from the workflow ID, which is stable across restarts.

@DBOS.step()
def dial(lead: int) -> dict:
    key = f"{DBOS.workflow_id}:{lead}" if USE_KEY else None
    res = place_call(lead, idempotency_key=key)
    if lead == CRASH_AT and not os.path.exists("crashed.marker"):
        open("crashed.marker", "w").close()
        os._exit(137)          # after the dial, before DBOS records the result
    return res

@DBOS.workflow()
def campaign(leads: list) -> int:
    n = 0
    for lead in leads:
        dial(lead)
        if classify(lead) == "interested":
            whatsapp(lead); n += 1
    return n

DBOS.launch()                  # also recovers a PENDING workflow from the last process
with SetWorkflowID("campaign-2026-10-19"):
    h = DBOS.start_workflow(campaign, LEADS)

Recovery needed no extra code. I ran python camp.py, it exited with code 137, I ran it again, and the second process picked up the pending workflow and finished it. DBOS printed one warning on the second run saying the workflow already exists in the internal queue, which I take to mean recovery had already claimed that workflow ID before my code tried to start it. After the run I read the state tables directly:

operation_outputs: [('classify', 40), ('dial', 40), ('whatsapp', 14)]
workflow_status:   [('campaign-2026-10-19', 'SUCCESS', 2)]

That is one recorded output per step execution and a workflow that shows two recovery attempts counted. The DBOS documentation says "If a DBOS process is interrupted while executing a workflow and restarts, it resumes the workflow from the last completed step" and that "An assigned workflow ID acts as an idempotency key: if a workflow is called multiple times with the same ID, it executes only once" (docs.dbos.dev, workflow tutorial, read 19 October 2026). Note that the workflow-level key protects the campaign from being started twice. It does not protect the individual calls. That is why I built a per-call key as well.

One caveat about storage. DBOS's architecture page says it checkpoints to Postgres. The DBOSConfig docstring in 3.3.0 says the system database defaults to SQLite, and that is what I used. I did not test Postgres, so treat my results as the engine's behavior, not as a statement about production database load.

Temporal

Temporal separates the pieces into a server, a worker and a client. I used the dev server that the Python package starts in-process, and ran the worker as a subprocess so I could kill it without killing the server. The driver restarts the worker when it exits.

@activity.defn
async def dial(lead: int) -> dict:
    info = activity.info()
    key = f"{info.workflow_id}:{lead}" if USE_KEY else None
    res = place_call(lead, idempotency_key=key)
    if lead == CRASH_AT and not os.path.exists("tcrashed.marker"):
        open("tcrashed.marker", "w").close()
        os._exit(137)          # the worker dies after the dial

@workflow.defn
class Campaign:
    @workflow.run
    async def run(self, leads: list) -> int:
        n = 0
        opts = dict(start_to_close_timeout=timedelta(seconds=4),
                    retry_policy=RetryPolicy(maximum_attempts=5,
                                             initial_interval=timedelta(seconds=1)))
        for lead in leads:
            await workflow.execute_activity(dial, lead, **opts)
            if await workflow.execute_activity(classify, lead, **opts) == "interested":
                await workflow.execute_activity(whatsapp, lead, **opts); n += 1
        return n

The driver printed worker exited with 137; restarting (#1) and then the workflow result. Recovery here is different from DBOS. The server noticed the activity never completed when my 4-second start_to_close_timeout expired, and it scheduled a retry for the restarted worker. Temporal's documentation says "If an Activity attempt fails, it is automatically retried using its Retry Policy" and recommends that activities be "idempotent, so retries can be processed without duplicate side effects" (docs.temporal.io/activities, read 19 October 2026). My one duplicate dial is exactly that retry.

One thing I did not chase. The first Temporal run printed several Task exception was never retrieved ... RPCError('transport error') tracebacks. They came from abandoned result-polling tasks in my own driver loop, where I wrap handle.result() in asyncio.wait_for. The second run was clean, and the duplicate counts come from the carrier log, not from the SDK output, so I left it. A production driver should not poll like that.

6. The idempotency key is the real fix

Look at the two zero rows. In both, the interrupted step ran again after the restart. The carrier received the same key, found it in its log and returned without dialing. The engine did not prevent the second attempt. The carrier boundary did.

This is the point I would put in front of anyone buying or building this. DBOS's architecture page says "Steps should be idempotent, meaning it should be safe to retry them multiple times." Temporal's says activities should be idempotent. Both vendors tell you that a step can run more than once, and my run confirms it for one crash. So the engine reduces your risk from "lose all progress" to "repeat at most the step in flight", and your design has to make that repeat harmless.

What does a usable key look like for a real voice campaign? Mine was workflow ID plus lead ID, which is stable across restarts and unique per lead per campaign. If a lead can be called twice in one campaign on purpose, such as a retry after no answer, add the attempt number to the key. Whether your telephony provider accepts an idempotency key on call creation is something I did not test, because I did not call any provider. If it does not, the same effect needs a check-then-write on your own table keyed the same way, inside the step, before the dial. That narrows the window but does not close it, since a crash between the dial and the write is case 2 again. For a live outbound call I would treat "a lead might get one extra call after a crash" as a business rule to decide on, not an engineering bug to hide.

7. The determinism rule, measured

The other half of the architecture is the rule that the workflow function must not decide anything from a source that changes between runs. I tested what happens when it does. The workflow below chooses between two steps with random.random(), a stand-in for an LLM that routes the campaign. It then crashes once, and the process is restarted.

@DBOS.workflow()
def bad():
    if random.random() < 0.5:      # decision made in workflow code, not in a step
        r = call_a()
    else:
        r = call_b()
    crash_once()
    return r

On restart the function runs from the top again, draws a new random number, and may choose the other branch. I ran the crash and restart 20 times. In 8 of the 20 trials the restarted workflow failed with:

DBOSUnexpectedStepError: DBOS Error 11: During execution of workflow nd-1 step 1,
function call_b was recorded when call_a was expected.
Check that your workflow is deterministic.

Eight of twenty is close to the one-in-two I would expect from a coin flip, which is the point: the bug shows up half the time, only after a crash, and only in production. Both engines state the rule. DBOS says "a workflow function must be deterministic: if called multiple times with the same inputs, it should invoke the same steps with the same inputs in the same order." Temporal says that "any time your Workflow code is executed it makes the same Workflow API calls in the same sequence, given the same input" and that a mismatch makes the execution return a non-deterministic error (docs.temporal.io/workflow-definition, read 19 October 2026). I only reproduced the failure in DBOS. I did not write the equivalent Temporal test, partly because I did not want to guess how its Python sandbox treats random.

The practical consequence for agent work: the model call goes in a step, and the branch the model chose is the step's recorded return value. In my DBOS lab, classify returned a random label from inside a step. The "interested" count differed between my two DBOS experiments (23 and 14). That is expected, because each experiment began a new campaign on a fresh state file and the classifier draws labels at random.

8. DBOS and Temporal side by side

I am not going to name a winner. The two solve the same problem from different places, and what I measured does not separate them on correctness. This is what I observed, what the vendors state, and what I did not test.

The package license row covers the Python SDK packages only. I did not read the license of the Temporal server binary, so I make no claim about it.

9. Failure modes I did not test

A single crash on one machine is a narrow test. These are the things I would test before putting a real campaign on either engine, and I list them so you do not read my zero-duplicate rows as a guarantee.

• Two processes recovering the same workflow. My runs had one process at a time. DBOS and Temporal both have a story for this, but I did not exercise it.
• Deploying new workflow code while a campaign is mid-flight. The determinism rule means a changed loop can fail on recovery. DBOS's tutorial links an "Upgrading Workflow Code" page that I did not read, and Temporal has its own versioning guidance that I did not read either.
• Losing the state store. My Temporal dev server kept state in memory and my DBOS lab used a local SQLite file. Neither says anything about a database failover.
• A live call in progress. A step can be replayed, but audio on a phone line cannot. A crash during a conversation drops that conversation. Durable execution protects the campaign around the call, not the call. That is my reading of the problem, not something I measured.
• Throughput and rate limits. Forty leads is not a campaign. I measured nothing about speed, and I did not check any carrier's concurrency or calling-hour rules.

10. Decision framework

Here is how I would choose, as a judgment based on the lab and the vendors' own statements. It is my opinion, not a measured ranking.

In the first three rows the carrier-side key stays. It is the only part of the design that changed my duplicate count from 1 to 0.

11. Recommendation

If you run outbound campaigns that last longer than a deploy cycle, do three things in this order. First, give every side-effecting step a key that survives a restart, and make the carrier or your own table honour it. Second, move every model call and every external call out of the workflow function and into a step, so the function stays a deterministic loop. Third, pick the engine by what you already operate: a database you trust for a library-style engine, or a platform team for Temporal.

I would not buy an engine to fix the first problem, because the checkpoint-file run showed the same exposure window. I would buy one for the second and for the visibility: a queryable record of exactly which step each lead reached. That is the part I would have had to build myself.

If you want this designed against your own dialer, CRM and compliance rules, that is the kind of work I take on through my AI agents practice. To size the effort first, use the ROI calculator or the AI readiness assessment. For what happens after a call goes wrong, see my post on voice agent handoff and escalation, and for the build-or-buy side of the voice stack see Vapi vs Retell vs self-hosted.

12. Reproduce it

The lab is five small files, under 200 lines in total. To run it yourself, create a virtualenv, pip install dbos temporalio, save the carrier and the variants, set CRASH_AT=17 and USE_KEY=0 or 1, run each script, run it a second time after it exits with code 137, and call the carrier's report function. The Temporal variant needs network access the first time, because the SDK downloads the dev server. Changing CRASH_AT moves the crash window, and setting it to -1 gives a clean run to compare against.

Ready to Get Started?

Book a free 30-minute discovery call. We'll identify your biggest opportunities and show you exactly what AI automation can do for your business.

Book a Free Discovery Call

Suyash Raj Founder, Voxdonna AI and AiSewak. Writes here about n8n, AI agents, and voice automation.