Terraform provider reference / guides / retries
Retries, 409 and 503
The provider repeats some failures for you and deliberately refuses to repeat others. The rules are not arbitrary - each one follows from what the platform does with a half-finished request.
What is repeated automatically
| Failure | Behaviour |
|---|---|
429 too many requests |
repeated, honouring Retry-After |
409 resource busy |
repeated - the refusal is issued before any work starts |
409 your own idempotency key is still running |
repeated for up to two minutes: your own answer is being prepared |
| connection dropped, timeout | repeated for reads, and for writes that carry an idempotency key |
503 on a read |
repeated |
503 on a write |
not repeated - handed back to you |
other 5xx |
not repeated |
The provider also keeps its own pace under the published rate limit, so a large plan does not spend
its first seconds collecting 429s. When the minute's share is spent - by a big plan, or by another
program using the same token - the provider waits for the next window, which can take up to a
minute. The terminal shows nothing during that wait; the log does: run with TF_LOG=INFO and look for
waiting for the public API rate limit window or the public API asked to retry later.
Why a write is not repeated on 503
Because the platform does not remember the outcome of a 5xx, and it frees the idempotency key when
one happens. A repeat therefore runs the whole handler again, from the beginning - it is a new
request wearing an old key.
For most resources that is merely wasteful. For a server it can cost money: a 5xx on create does
NOT mean there is no server - the machine may already be built when the failure is reported, and a
repeat would build a second, billed one. The address is a smaller matter: when a create fails after
the address was reserved for a known reason (no capacity, the chosen host has no room, the
application's password or email, the machine could not be created), the platform releases the new
address of that request itself; an outcome it cannot name (a bare 500, a dropped connection)
keeps the address, and you see it in the project.
So the provider stops and tells you. The right move is to look at what actually happened - read the
project's servers and addresses - and then decide. terraform refresh (or simply the next plan)
is usually enough to find out.
An interrupted run
Stopping apply with Ctrl-C (or a stop signal) while a server is being created is safe: the error
says the run was interrupted, the server is recorded in state and marked tainted, and the warning
names it. Check the server, then either terraform untaint <resource address> or let the next
apply replace it.
A hard kill (kill -9, a CI runner that died, an out-of-memory kill) gives Terraform no chance to
record anything: the server exists on the platform and is billed, but it is not in state. The next
plan then proposes to create it again, and warns that a server with the same name already exists
in the project, naming its ID. Import that one instead of creating a second:
terraform import <resource address> <ID>. The platform allows equal names, so the provider warns
rather than refuses - two resources may carry one name on purpose.
Idempotency keys
Every create the provider sends carries a key, generated per request. That is what makes a repeat after a dropped connection safe: the platform returns the answer it already gave, together with the identifier of the resource it already made.
You do not set these keys yourself, and you should not try to: a key reused across two genuinely different requests would hand you the wrong answer.
A refusal of the request itself (a 4xx other than 402 and 409) is stored under the key as well,
for a day. That matters when you fix a rejected request by hand and retry it: the platform answers a
repeat under the same key with the stored refusal, and a changed body under that key is rejected
outright as idempotency_key_reused (422). A 402, a 409 and any 5xx are NOT stored: the key is
freed, so after paying a debt or waiting out a busy server you repeat the same body with the same key.
The provider generates a fresh key per request, so a corrected apply is not affected - but a script
of your own that reuses keys will be.
What a 409 usually means in practice
Operations on one project are queued. A 409 with Retry-After normally means a neighbour is being
submitted - often another run of your own pipeline, or someone working in the client area at the
same moment. The provider waits and retries, so you rarely see it.
You do see it when the wait is not enough: several parallel terraform apply runs against one
project will fight over the same queue. Run them one at a time.
When the platform says the queue itself is unavailable
A 503 whose message speaks about serialisation means the platform could not queue the operation at
all, and it refused rather than running it unqueued. This is a temporary condition on our side. The
provider hands a write like that back to you unchanged - see above for why.
Generated from the provider schema for version 0.11.0. How to install the provider: Terraform provider.