Async Long-Running Tool Calls

Overview

Overview

Standard LLM tool use assumes a tool returns inside the inference timeout, typically a few seconds. A data pipeline, batch job, video transcode or fine-tuning run can take five minutes to an hour, and bridging that gap without holding a server connection open takes patterns from both agent orchestration and distributed systems.

The fundamental mismatch

LLMs are built for synchronous request-response cycles. API servers, load balancers and connection pools assume the same, with hard timeouts typically in the 30-60 second range, so a 5-minute pipeline run breaks every timeout at every layer.

The fix is never to hold the connection open. The agent instead:

• Initiates the long-running task.
• Records that it started, by storing job_id in state.
• Releases the connection immediately.
• Resumes from a checkpointer snapshot when the task completes.

• Checkpointer — Persists agent state while it is suspended
• Webhook mechanism — Lets the external task announce completion
• Idempotent tool invocations — Makes retry and replay safe

The three patterns

• Checkpoint-suspend-webhook-resume — Start the job, store job_id, interrupt; the webhook writes the result and re-invokes the same thread_id — No — The canonical LangGraph choice
• Async polling — A polling node checks a status endpoint and loops with exponential backoff — Yes — The service exposes only a status endpoint, no webhooks
• SSE streaming — Server-Sent Events push progress ("started", "20% complete", "complete, resuming agent") to the browser — No — User experience, decoupled from agent resumption

> [!NOTE] For jobs longer than a few minutes, checkpoint-suspend-webhook beats polling because polling keeps a live execution thread busy.

Pattern 1 state

The tool node starts the pipeline, stores the job_id and triggers a graph interrupt. The webhook handler looks up the thread_id for that job_id, writes the result with graph.update_state() and calls graph.invoke() on the same thread_id.

> [!IMPORTANT] Generate the idempotency key in state before triggering the pipeline. On a crash and replay, the same key goes to the pipeline service, which returns the existing job_id instead of starting a second run.

Durable execution: Temporal and Step Functions

For workflows spanning hours or days, LangGraph checkpointers are often not enough; they are built for conversation-length persistence.

Temporal and AWS Step Functions provide durable execution primitives:

• Activity retries with configurable policies.
• Workflow versioning.
• Event history replay.
• Saga patterns for multi-step compensation.

> [!TIP] Reach for Temporal or Step Functions when you need exactly-once execution, multi-hour fault tolerance or saga compensation logic.

Interview questions

• How do you handle long-running asynchronous tool calls? For example, if an agent triggers a data pipeline that takes 5 minutes, how do you manage the agent's state and the user experience?
• What happens if the agent process crashes while waiting for the 5-minute pipeline? How do you recover?
• How would you design the user experience so the user knows what is happening during a 5-minute wait?

More Agentic AI topics

• Context Window Management for Persistent Agents
• Multi-Agent Systems with LangGraph