Skip to main content

Retry and recover

A batch of calls where one endpoint is broken. The run retries it, gives up, records what happened, and processes the rest. The output is a file with a row for every input — which is the point: a run that ends on the first bad record tells you nothing about the other ninety-nine.

The workflow​

Global variables:

endpoints:
  type: array
  value:
    - type: string
      value: 'https://jsonplaceholder.typicode.com/users/1'
    - type: string
      value: 'https://this-host-does-not-exist.invalid/users/2'
    - type: string
      value: 'https://jsonplaceholder.typicode.com/users/3'
CreateTable("outcomes", schema={"columns": [{"name": "endpoint", "isKey": True}, {"name": "status"}, {"name": "detail"}]})

for endpoint in endpoints:  # _name="Call each endpoint" _description="One row per endpoint whatever happens; a failure must not end the batch"
    try:
        with RetryOnError(retries=3, delay=2):
            record = HttpRequest("{{ endpoint }}", "GET")
        TableUpsertRow("outcomes", {"endpoint": "{{ endpoint }}", "status": "ok", "detail": "{{ record.name }}"})
    except Exception as err:
        TableUpsertRow("outcomes", {"endpoint": "{{ endpoint }}", "status": "failed", "detail": "{{ err }}"})

export = TableExport("outcomes", "C:/Temp/rinkt-examples/outcomes.csv", format="csv")
WriteLine("{{ export.rows }} endpoints recorded in {{ export.filePath }}")

The run​

Twelve seconds — most of it the three retries, two seconds apart — and three rows:

endpoint,status,detail
https://jsonplaceholder.typicode.com/users/1,ok,Leanne Graham
https://this-host-does-not-exist.invalid/users/2,failed,"Maximum number of retries reached — Get ""https://this-host-does-not-exist.invalid/users/2"": dial tcp: lookup this-host-does-not-exist.invalid: no such host"
https://jsonplaceholder.typicode.com/users/3,ok,Clementine Bauch

The run completed. Nothing about it is green-washed: the failure is in the file, with the error that caused it, and the two good records are there as well.

Worth taking from it​

RetryOnError goes inside try, not instead of it. They answer different questions. with RetryOnError(retries=3, delay=2): handles the failure that goes away on its own — a timeout, a rate limit, a page that had not finished loading. try / except handles the one that does not. Retrying alone still ends the run when the retries are exhausted; catching alone gives up on the first hiccup.

Wrap the smallest thing that can fail. Only the request is inside RetryOnError here. Putting the row-write in there too would rewrite a row three times to recover a request.

except Exception as err binds the error. {{ err }} is the message, and writing it into the row is the difference between a file that says something failed and a file that says why. The message above names DNS resolution — you can act on that without opening the run.

Retries cost wall-clock time. Three retries two seconds apart turned a sub-second batch into twelve seconds. On a batch of a thousand records with a systematically broken endpoint, that arithmetic is the whole run. Retry counts want to be small.