Rebuilding an Agent From Scratch When the Original Is Unrecoverable

Rebuilding an agent from scratch when the original is unrecoverable is an infrastructure reset. The work starts with the layers that let a system survive failure, including durable state, versioned artifacts, scoped credentials, and automatic rollback. Recreating the old behavior before those layers exist only recreates the conditions that brought it down.

Why agents become unrecoverable in the first place

A failed agent may have had silent model degradation as inputs drifted, a hard deployment failure, or state that was corrupted after a mid-run crash. Each failure needs its own recovery path, which has to be built before it can be used. Temporal's Preeti Somal describes the pattern plainly. "They had to move really fast, but they didn't take care of the plumbing. Things crash and burn, and then they're back to rebuilding with the reliable foundation."

Brittle connections to legacy systems and undocumented internal APIs work in clean environments, then break against real customer inputs. Teams can also think that saving conversation history means they have saved execution state. State records where the workflow stands and what has completed, while memory carries context across interactions. Protecting one does not protect the other, so an agent can remember the conversation but have no reliable account of its own actions.

What "unrecoverable" actually means and why the distinction matters for the rebuild

Unrecoverable is a spectrum. A missing checkpoint removes the place to resume, a missing versioned artifact removes the rollback target, and a missing audit trail removes the record of what happened. Those are independent failures that compound each other. If an email was sent, a record modified, or an API request submitted before a crash, lost state leaves no reliable distinction between finished and unfinished work.

That distinction sets the rebuild's scope. If a checkpoint exists, recovery may still be possible through compensating actions. If credentials remain intact but state is gone, the job may be a partial rebuild. If both are gone, it becomes a full rebuild. Credential custody therefore belongs in the diagnosis, because lost access and lost execution history create different recovery problems.

Why the rebuild order matters — infrastructure before logic

Rebuilding behavior first feels productive because it produces something visible. But without durable state, a version registry, and rollback triggers, that behavior remains as fragile as before. The parallel with cloud lift-and-shift is direct. Moving workloads without redesigning their underlying architecture left organizations spending more and getting less. Rebuilding an agent on its old scaffold repeats that mistake.

The dependency chain begins with versioning before any new agent logic is written. Model weights, prompts, tool configurations, and environment specs belong in one immutable artifact identified by a content hash. Once the configuration has a fixed identity, checkpointing can give execution a reliable place to resume. But resuming is not enough when an action has already changed something outside the agent, so compensating actions must be defined before live execution.

Only then can monitoring judge whether the reconstructed workflow is safe to expose. Schema compliance, tool call success rate, and latency per agent node supply the signals for that judgment. Shadow mode follows, letting the agent generate and log what it would do without executing it. Reviewing that dry run before cutover completes the chain from a known configuration to an observed workflow.

State checkpointing — the layer that turns crashes into resumptions

Checkpointing determines whether a crash leads to a retry or hours of manual cleanup. Without it, restarting workflows repeats inference work, raises costs, and adds latency. For an agent running over hours, a network failure near the end can erase the value of everything that came before. That makes persistence part of execution design from the outset. A checkpoint belongs before every critical action.

A useful checkpoint captures the agent's goals, working variables, and position in the workflow. It also captures tool-call history, distinguishing completed, in-flight, and failed calls. Credentials and connection state need scoped handling that keeps cached secrets confidential. Execution state belongs in the orchestration layer, while memory belongs in retrieval and conversation layers. They need separate storage and recovery paths because they answer different questions about what the agent was doing and what it knew.

Abridge, a healthcare workflow customer of Temporal's, needs to be able to recover at each stage. Its workflow spans audio processing, summarization, multiple model calls, and after-visit generation. With real-world side effects across those stages, restarting from the beginning is not an adequate recovery plan. The useful boundary is the last completed stage.

Rollback infrastructure — version registries, blue-green cutover, and deployment gates

Checkpoints help when a run fails, but a bad deployment needs another route to recovery. That route starts with an immutable snapshot of the whole model artifact, including weights, configuration, prompt templates, preprocessing pipelines, and environment specs. Rolling back weights while keeping newer prompts can make the agent behave in unexpected ways. MLflow 3.0, released June 2025, connects model weights to prompts, evaluation runs, and dataset versions in a single lineage graph.

The registry needs three aliases called stable, candidate, and previous. Stable points to the proven version in production, candidate points to the version being tested, and previous points to the last stable version. With those roles established, rollback becomes an alias reassignment. Blue-green deployment adds two identical environments in parallel, with the new version validated against shadow traffic before the load balancer switches traffic. That pattern works for schema-breaking changes, regulatory audit requirements, and workflows that cannot be disrupted.

Canary deployment offers an incremental alternative by sending the new version a small slice of traffic. Automated gates watch for threshold breaches and halt the rollout or trigger reversion before full cutover. Those thresholds need to be in the deployment configuration before release. A rollback plan invented after deployment is already a recovery project.

Rebuilding the integration layer with narrower credential scope

Saved state and sound deployments cannot bring back access that the operator no longer controls. The Composio breach of May 2026 exposed that custody problem. Composio could not revoke most API-key-scheme connections from its side, and under 5% of affected connections could not be revoked at all. Agent operators therefore had to finish the cleanup that the integration platform could not.

The rebuild is the point to narrow every credential's reach. Use user-level authorization and limited OAuth scopes in place of broad API keys where they are supported. Each credential needs a named owner, a record of what it can reach, and responsibility for rotation. Revocation needs to work for each connection as well as for the whole account. Audit logs must also show which connections were active when failure occurred, or cleanup begins without a clear boundary.

Managed integration still has real value because it handles OAuth flows, credential refresh, rate-limiting, error handling, and standardized tool access. Composio provides an MCP Gateway compatible with Claude, Cursor, OpenAI Codex, and other frameworks. That convenience does not remove custody obligations. Its compliance tier also needs to be checked before it is chosen, since it has independently audited SOC 2 Type II and ISO 27001:2022, but HIPAA BAA and zero data retention cost extra per call on lower tiers and are included only on enterprise plans.

Choosing the runtime the rebuilt agent will run on

Runtime choice follows from the recovery requirements. The mid-2026 landscape separates self-hosted operator runtimes, including OpenClaw and Hermes Agent, from coding-agent surfaces, including Claude Code and OpenAI Codex. They are not direct substitutes. Orchestration breadth, persistent cross-session memory, and deep codebase reasoning serve different needs. Deliberate combinations can be more useful than forcing one runtime to own every task.

Hermes Agent, launched by Nous Research in February 2026, focuses on self-improving skills and persistent cross-session memory. That helps when the rebuilt agent needs to run with less hands-on work. Claude Code instead supports deep codebase work by reading across a codebase, planning across files, and iterating on test failures. Its lack of model flexibility makes an Anthropic API outage a meaningful risk for unattended production-critical work because there is no model fallback. The rebuild should make runtime ownership explicit, assigning subtasks deliberately rather than letting the architecture emerge by accident.

Evals, monitoring, and the three signals that catch regressions before they become incidents

The runtime still needs evidence that its behavior is acceptable. Unit evals test each node and tool call on its own. Integration evals test whole workflows with inputs like those they will face, including the messy inputs missing from the pilot. Adversarial evals test prompt injection, boundary cases, and inputs meant to trigger hallucination or memory drift. Together, those layers cover failures that a clean end-to-end demonstration can miss.

Production monitoring then watches three different surfaces. Schema compliance shows whether outputs match the expected structure. Tool call success rate shows whether outside calls complete or silently fail. Latency per agent node shows where work slows, which can be an early sign of a degrading model call or a rate-limited integration. With thresholds in place, those numbers drive deployment gates, so someone does not have to work out what they mean during an incident.

That matters most for silent degradation, where outputs worsen gradually while the agent appears to function. Without thresholds, falling business KPIs may be the first clear sign, weeks after performance began to slip. The same monitoring layer needs to show costs because long workflows spend tokens across many calls. Orchestration visibility exposes the "token tax" of failed and retried work. Monitoring therefore has to show both whether the agent is working and where recovery is consuming resources.

What the rebuild looks like when the original was serving multiple clients

For a multi-client operator, shared failure changes the scale of the rebuild. An incident on a shared instance reaches every client there, while an isolated instance confines the blast radius to one. The appropriate unit is an isolated, persistent agent per client, provisioned and torn down through an API. Each client then has a separate checkpoint history, credential scope, and rollback surface.

Stammer AI provides white-label infrastructure for agencies selling chatbots and voice agents, with a rebrandable dashboard, domain, and API, client sub-accounts, prepaid usage wallets, and configurable subscription packages. Kore.ai offers an enterprise compliance stack covering SOC 2 Type II, ISO/IEC 27001:2022, PCI DSS, GDPR, CCPA, and HIPAA-aligned requirements. KiloClaw provides managed hosting for production OpenClaw agents. NemoClaw, announced by NVIDIA at GTC March 2026, adds OpenShell runtime sandboxing and policy-based privacy guardrails to OpenClaw through a single-command installation.

If provisioning still means an SSH session, a Docker command, or a hand-configured environment, the operator is the infrastructure. The rebuild should make onboarding-triggered provisioning the default. A signup should create an isolated instance with its own recovery boundaries, without a founder touching a terminal.

What the rebuilt agent has that the original did not

The rebuilt agent is a different class of system because its recovery layers are now load-bearing. Its plan-execute separation makes actions reversible or defers them until the final moment, with compensating actions recorded for external side effects. Its immutable versions and checkpoints provide known places to return to. Its credentials have owners and revocation boundaries, while its deployment gates act without waiting for a person to notice failure.

Deloitte's 2026 State of AI in the Enterprise report found that only one in five companies has a mature governance model for autonomous AI agents. The rebuild must define what the agent can do without approval, who is liable for a wrong decision, and how recovery works. The discipline that lasts is to version everything, monitor everything, and set rollback triggers before deployment. Production survival starts with that discipline in the first commit of the rebuilt version.