Error Handling & Retries

Cloud storage over the network is inherently unreliable. Connections drop, APIs get rate-limited, and reads can timeout. To provide a stable interface, s3fs includes a robust, built-in retry mechanism wrapped around its asynchronous calls.

This guide explains how s3fs handles failures internally and how you can extend that behavior for your specific use cases.

The Default Retry Strategy

Whenever s3fs communicates with S3 (e.g., calling get_object or put_object), the operation is wrapped in an _error_wrapper function.

By default, s3fs attempts the operation up to 5 times. If a retryable error occurs, it pauses execution using an exponential backoff algorithm. The wait time is calculated as min(1.7^attempt * 0.1, 15). This means delays start small (~100ms) and cap out at 15 seconds between retries.

Default Retryable Exceptions

s3fs automatically catches and retries the following transient network exceptions:

  • socket.timeout
  • botocore.exceptions.HTTPClientError
  • urllib3.exceptions.IncompleteRead
  • fsspec.asyn.FSTimeoutError
  • botocore.parsers.ResponseParserError
  • aiohttp.ClientPayloadError (if aiohttp is installed)

Additionally, it catches botocore.exceptions.ClientError and inspects the response. It automatically retries if the error message contains known rate-limiting or synchronization indicators, such as:

  • SlowDown
  • reduce your request rate
  • XAmzContentSHA256Mismatch

Extending Retry Logic

In some distributed architectures or strict enterprise environments, you might encounter custom proxies, firewalls, or internal S3 wrappers that return unique exception types or HTTP codes that s3fs does not recognize as transient.

1. Registering Custom Error Types

If you have an exception class that you know represents a temporary failure, you can add it to the global list of retryable errors using add_retryable_error. 

import s3fs
from requests.exceptions import ConnectionError

# Instruct s3fs to treat requests.exceptions.ConnectionError 
# as a transient, retryable failure globally.
s3fs.add_retryable_error(ConnectionError)

# Now create your filesystem
s3 = s3fs.S3FileSystem()

2. Implementing Custom Error Handlers

Sometimes simply matching an exception class isn't enough. You might need to inspect the contents of the error message to determine if it should be retried.

You can set a custom handler using set_custom_error_handler. This function must accept a single argument (the exception instance) and return True (retry) or False (do not retry and raise immediately).

import s3fs

def specific_failure_handler(e):
    # Only retry if the string "Gateway Timeout" is in the error message
    if isinstance(e, Exception) and "Gateway Timeout" in str(e):
        return True
    return False

s3fs.set_custom_error_handler(specific_failure_handler)

3. Handling AWS ClientErrors

botocore.exceptions.ClientError is a "catch-all" exception for HTTP 4xx and 5xx errors returned by AWS.

s3fs checks these against its internal patterns first. If it does not match its internal SlowDown patterns, it passes the exception to your custom error handler. You can use this to retry specific AWS error codes.

from botocore.exceptions import ClientError
import s3fs

def my_client_error_handler(e):
    if isinstance(e, ClientError):
        # Extract the specific AWS error code
        error_code = e.response.get("Error", {}).get("Code")

        # Retry if our custom infrastructure returns a specific throttling code
        if error_code == "CustomThrottlingCode":
            return True

    return False

s3fs.set_custom_error_handler(my_client_error_handler)

Best Practices for Data Pipelines

  1. Fail Fast on Permissions: Do not configure custom handlers to retry 403 Forbidden or 404 Not Found errors (unless you expect eventual consistency delays). Retrying auth failures only slows down pipeline failures.
  2. Understand Eventual Consistency: While AWS S3 is now strongly consistent for read-after-write, some alternative S3 providers are not. If you are using an alternative provider, you might need a custom handler to retry 404 Not Found errors briefly immediately after creating a file.
  3. Logs are your Friend: If you suspect operations are failing and hitting the max retries, enable the debug logger (export S3FS_LOGGING_LEVEL=DEBUG). It will print messages indicating exactly which error triggered a retry.