Terraform provider reference / guides / retries

Retries, 409 and 503

The provider repeats some failures for you and deliberately refuses to repeat others. The rules are not arbitrary - each one follows from what the platform does with a half-finished request.

What is repeated automatically

Failure Behaviour
429 too many requests repeated, honouring Retry-After
409 resource busy repeated - the refusal is issued before any work starts
409 your own idempotency key is still running repeated for up to two minutes: your own answer is being prepared
connection dropped, timeout repeated for reads, and for writes that carry an idempotency key
503 on a read repeated
503 on a write not repeated - handed back to you
other 5xx not repeated

The provider also keeps its own pace under the published rate limit, so a large plan does not spend its first seconds collecting 429s. When the minute's share is spent - by a big plan, or by another program using the same token - the provider waits for the next window, which can take up to a minute. The terminal shows nothing during that wait; the log does: run with TF_LOG=INFO and look for waiting for the public API rate limit window or the public API asked to retry later.

Why a write is not repeated on 503

Because the platform does not remember the outcome of a 5xx, and it frees the idempotency key when one happens. A repeat therefore runs the whole handler again, from the beginning - it is a new request wearing an old key.

For most resources that is merely wasteful. For a server it can cost money: a 5xx on create does NOT mean there is no server - the machine may already be built when the failure is reported, and a repeat would build a second, billed one. The address is a smaller matter: when a create fails after the address was reserved for a known reason (no capacity, the chosen host has no room, the application's password or email, the machine could not be created), the platform releases the new address of that request itself; an outcome it cannot name (a bare 500, a dropped connection) keeps the address, and you see it in the project.

So the provider stops and tells you. The right move is to look at what actually happened - read the project's servers and addresses - and then decide. terraform refresh (or simply the next plan) is usually enough to find out.

An interrupted run

Stopping apply with Ctrl-C (or a stop signal) while a server is being created is safe: the error says the run was interrupted, the server is recorded in state and marked tainted, and the warning names it. Check the server, then either terraform untaint <resource address> or let the next apply replace it.

A hard kill (kill -9, a CI runner that died, an out-of-memory kill) gives Terraform no chance to record anything: the server exists on the platform and is billed, but it is not in state. The next plan then proposes to create it again, and warns that a server with the same name already exists in the project, naming its ID. Import that one instead of creating a second: terraform import <resource address> <ID>. The platform allows equal names, so the provider warns rather than refuses - two resources may carry one name on purpose.

Idempotency keys

Every create the provider sends carries a key, generated per request. That is what makes a repeat after a dropped connection safe: the platform returns the answer it already gave, together with the identifier of the resource it already made.

You do not set these keys yourself, and you should not try to: a key reused across two genuinely different requests would hand you the wrong answer.

A refusal of the request itself (a 4xx other than 402 and 409) is stored under the key as well, for a day. That matters when you fix a rejected request by hand and retry it: the platform answers a repeat under the same key with the stored refusal, and a changed body under that key is rejected outright as idempotency_key_reused (422). A 402, a 409 and any 5xx are NOT stored: the key is freed, so after paying a debt or waiting out a busy server you repeat the same body with the same key. The provider generates a fresh key per request, so a corrected apply is not affected - but a script of your own that reuses keys will be.

What a 409 usually means in practice

Operations on one project are queued. A 409 with Retry-After normally means a neighbour is being submitted - often another run of your own pipeline, or someone working in the client area at the same moment. The provider waits and retries, so you rarely see it.

You do see it when the wait is not enough: several parallel terraform apply runs against one project will fight over the same queue. Run them one at a time.

When the platform says the queue itself is unavailable

A 503 whose message speaks about serialisation means the platform could not queue the operation at all, and it refused rather than running it unqueued. This is a temporary condition on our side. The provider hands a write like that back to you unchanged - see above for why.

Generated from the provider schema for version 0.11.0. How to install the provider: Terraform provider.