Terraform provider reference / guides / retries
Retries, 409 and 503
The provider repeats some failures for you and deliberately refuses to repeat others. The rules are not arbitrary - each one follows from what the platform does with a half-finished request.
What is repeated automatically
| Failure | Behaviour |
|---|---|
429 too many requests |
repeated, honouring Retry-After |
409 resource busy |
repeated - the refusal is issued before any work starts |
409 your own idempotency key is still running |
repeated for up to two minutes: your own answer is being prepared |
| connection dropped, timeout | repeated for reads, and for writes that carry an idempotency key |
503 on a read |
repeated |
503 on a write |
not repeated - handed back to you |
other 5xx |
not repeated |
The provider also keeps its own pace under the published rate limit, so a large plan does not spend its first seconds collecting 429s.
Why a write is not repeated on 503
Because the platform does not remember the outcome of a 5xx, and it frees the idempotency key when
one happens. A repeat therefore runs the whole handler again, from the beginning - it is a new
request wearing an old key.
For most resources that is merely wasteful. For a server it costs money: the public address is reserved before the machine is built, so every repeat of a failed create leaves one more billable address behind in the project.
So the provider stops and tells you. The right move is to look at what actually happened - read the
project's servers and addresses - and then decide. terraform refresh (or simply the next plan)
is usually enough to find out.
Idempotency keys
Every create the provider sends carries a key, generated per request. That is what makes a repeat after a dropped connection safe: the platform returns the answer it already gave, together with the identifier of the resource it already made.
You do not set these keys yourself, and you should not try to: a key reused across two genuinely different requests would hand you the wrong answer.
A refusal is stored under the key as well, for a day. That matters when you fix a rejected request
by hand and retry it: the platform answers a repeat under the same key with the stored refusal, and
a changed body under that key is rejected outright as idempotency_key_reused (422). The provider
generates a fresh key per request, so a corrected apply is not affected - but a script of your own
that reuses keys will be.
What a 409 usually means in practice
Operations on one project are queued. A 409 with Retry-After normally means a neighbour is being
submitted - often another run of your own pipeline, or someone working in the client area at the
same moment. The provider waits and retries, so you rarely see it.
You do see it when the wait is not enough: several parallel terraform apply runs against one
project will fight over the same queue. Run them one at a time.
When the platform says the queue itself is unavailable
A 503 whose message speaks about serialisation means the platform could not queue the operation at
all, and it refused rather than running it unqueued. This is a temporary condition on our side. The
provider hands a write like that back to you unchanged - see above for why.
Generated from the provider schema for version 0.3.0. How to install the provider: Terraform provider.