A Failed Deployment Is Still a State Transition

The deployment reaches configuration and stops.

The network exists. The virtual machine exists. Configuration did not finish, and the health check never ran. The pipeline is red, but the provider still has live resources.

Both facts can be true. The command failed, and the infrastructure changed.

The original article appeared on HackerNoon. This version includes the HybridOps state and regression contract.

A failed command is not a rollback

Infrastructure workflows often expose one action while making several remote calls underneath it. A deployment can create a network, reserve an address, create a virtual machine, wait for access, configure the operating system, and run service probes.

Those calls do not become one atomic transaction because one command started them. If the fourth step fails, the first three may remain complete.

That distinction is easy to lose in CI. Green means that the process reached its expected end. Red means that it did not. Neither result proves which provider mutations committed before execution stopped.

A blind retry is therefore unsafe whenever the first attempt has an uncertain outcome. The provider may have accepted a create request before the response was lost. Repeating the call can create a duplicate, return a conflict, or attach later work to the wrong object.

Idempotency helps when the caller and provider share a stable operation identity. It does not explain which earlier steps completed, which later steps never started, or whether the remaining object should be resumed or removed.

That requires operation state.

Persist the transition before mutation

The most important write happens before control passes to the provider.

After validation and preflight pass, the runtime should store a nonterminal state such as running. It needs the operation identity, target, resolved inputs, timestamp, and any outputs already known from an earlier run. Only then should mutation begin.

If that write fails, the provider call should not start. Blocking a deployment because runtime storage is unavailable can feel strict, but the alternative is an infrastructure change with no durable record that it began.

A reduced record looks like this:

{
  "module_ref": "platform/example/service",
  "status": "running",
  "active_command": "apply",
  "run_id": "<run-id>",
  "resolved_inputs_file": "<run-record>/resolved.inputs.yml",
  "outputs": {
    "resource_id": "previously-known-resource"
  }
}

There is a conservative edge case: the process may persist running and stop before the provider receives the request. The next operator then has to reconcile an uncertain operation that made no remote change. That is safer than retaining a stale destroyed marker after a provider may have created something.

Failure should update state, not erase it

When execution fails, the runtime should preserve what it already knows.

{
  "module_ref": "platform/example/service",
  "status": "error",
  "failed_command": "apply",
  "last_error": "provider failed after create",
  "outputs": {
    "resource_id": "previously-known-resource"
  },
  "rerun_inputs_file": "<runtime-root>/config/rerun-inputs.yml",
  "run_id": "<run-id>"
}

On a first creation, the provider may create an object and fail before returning its identifier. The runtime cannot preserve a value it never received. It can still retain the operation identity, inputs, timestamp, error, and run record needed to inspect the provider and reconcile the result.

Provider state and operation state serve different purposes. Provider state maps declared resources to remote objects. Operation state records which lifecycle action was attempted across engines, where it stopped, and what the next safe actions are. Neither should impersonate the other.

Pending work did not fail

A dependency graph needs to distinguish failure from work that never ran.

network          ok
virtual machine  error
configuration    pending
health check     pending

Configuration and health checks did not fail. They were blocked by an earlier step.

Marking every downstream node as failed hides the actual boundary and can lead an operator to rerun completed work. A useful lifecycle vocabulary separates complete, error, pending, retained, absent, and destroyed states.

Reconciliation is more useful than retry here. Resume from facts that can be checked, not from the assumption that a failed process changed nothing.

Destroy is also a verified operation

An error state must be treated as potentially live.

Destroy logic that skips every step not marked ok will skip the resources most likely to need cleanup after a partial apply. A safer rule is to skip only state already confirmed absent or destroyed. An error record remains eligible for provider inspection and teardown.

Parent ownership matters as well. A child resource may disappear when its parent is removed. The child should not be marked destroyed when parent deletion starts. It reaches terminal state only after the parent is confirmed absent. If deletion fails, the child remains unresolved.

This is the same operational discipline described in Disaster Recovery as a Governance System: the request is not the outcome, and the outcome needs evidence.

The HybridOps regression contract

HybridOps Core tests this boundary directly. The regression path begins with a previous destroyed marker, starts a new apply, and returns a partial provider failure.

The contract requires the runtime to:

1. persist running before provider mutation;
2. stop before the provider if that persistence fails;
3. replace running with ok after success;
4. replace it with error after failure without discarding known outputs; and
5. keep the failed operation eligible for reconciliation and verified destroy.

The public implementation is covered by the mutation-state regression suite and the original partial-mutation change.

These tests do not make remote infrastructure atomic. They preserve an honest account of what the process attempted and what the provider may now contain.

A red pipeline proves that execution ended badly. It does not prove that the environment rolled back. Recovery should begin with durable state and a valid reconciliation path, not a provider console and a reconstruction exercise.