Usage Guide

This guide covers the core concepts, advanced configurations, and operational workflows required to use s3fs effectively in production environments.

1. Authentication & Credentials

Properly managing credentials is the foundation of using s3fs. By default, s3fs relies on botocore's standard credential resolution chain.

If you instantiate S3FileSystem without explicit keys, it will search for credentials in the following order:

  1. Environment Variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN.
  2. Shared Configuration Files: Typically located at ~/.aws/credentials and ~/.aws/config.
  3. IAM Roles/Metadata: If running on an EC2 instance, ECS container, or Lambda function, it will automatically query the AWS metadata service for temporary credentials.
import s3fs

# Uses automatic resolution
s3 = s3fs.S3FileSystem(anon=False)

Specific AWS Profiles

If you have multiple AWS accounts configured in your ~/.aws/credentials file, you can explicitly request a specific profile via the profile parameter:

s3 = s3fs.S3FileSystem(profile='my_production_profile')

Explicit Credentials

While not recommended for production code (to avoid hardcoding secrets), you can pass credentials explicitly:

s3 = s3fs.S3FileSystem(key='AKIA...', secret='SECRET...')

Temporary Credentials for Distributed Systems

In a distributed computing environment (like a Dask cluster), passing raw credentials between machines is a security risk. Instead, you can use the get_delegated_s3pars() method. This method requests short-lived, temporary credentials from AWS STS (Security Token Service).

# Get temporary credentials valid for 1 hour (3600 seconds)
temp_creds = s3.get_delegated_s3pars(exp=3600)

# temp_creds is a dictionary: {'key': '...', 'secret': '...', 'token': '...', 'anon': False}
# These can be safely passed to worker nodes to instantiate their own S3FileSystem.

2. Asynchronous Operations

s3fs is built on aiobotocore and provides first-class asynchronous functionality. This is highly beneficial because S3 operations are heavily I/O bound and subject to network latency.

For every synchronous method (e.g., ls, cat, info), there is an asynchronous equivalent prefixed with an underscore (e.g., _ls, _cat, _info).

Setting up Async s3fs

To use s3fs asynchronously, you must pass asynchronous=True to the constructor. Crucially, you must explicitly await the creation of the underlying HTTP session using set_session() before making any calls.

import asyncio
from s3fs import S3FileSystem

async def fetch_multiple_files():
    # 1. Initialize in async mode
    s3 = S3FileSystem(anon=True, asynchronous=True)

    # 2. Await the session establishment
    session = await s3.set_session()

    # 3. Perform concurrent network requests
    bucket_path = 's3://noaa-hrrr-bdp-pds/hrrr.20140730/conus/'

    # Concurrently get info for three different files
    tasks = [
        s3._info(f"{bucket_path}file1.grib2"),
        s3._info(f"{bucket_path}file2.grib2"),
        s3._info(f"{bucket_path}file3.grib2")
    ]

    # gather executes them simultaneously
    results = await asyncio.gather(*tasks)
    print([res['size'] for res in results])

    # 4. Clean up the session
    await session.close()

asyncio.run(fetch_multiple_files())

3. Caching Strategies

s3fs optimizes S3 reads by caching data in memory. This prevents making a separate HTTP request for every small .read() call.

You can control this behavior using the default_cache_type and fill_cache arguments in the S3FileSystem constructor, or per-file in the s3.open() method.

  • readahead (Default): Fetches data ahead of the current file pointer. Best for sequential reading.
  • bytes: Reads specific byte blocks as requested.
  • none: Disables caching. Every read call directly hits the network. Useful only if you are managing buffers yourself or downloading exact, large blocks.
# Disable cache filling if you only need small, random reads
with s3.open('my-bucket/huge_file.dat', 'rb', fill_cache=False) as f:
    f.seek(1000000) # Seek to 1MB
    data = f.read(10) # Reads only 10 bytes without fetching surrounding blocks

4. Write Concurrency & Multipart Uploads

When writing files, s3fs uses S3 Multipart Uploads under the hood. Data is written to an internal buffer. Once the buffer hits the block_size (default 50MB), that block is uploaded to S3 as a part.

Concurrency limits

By default, s3fs will upload up to 10 parts concurrently. You can adjust this via the max_concurrency argument in S3FileSystem:

# Increase concurrency for high-bandwidth environments
s3 = s3fs.S3FileSystem(max_concurrency=20)

Note: Increasing max_concurrency increases memory usage, as multiple 50MB blocks will be held in RAM simultaneously while waiting for the network.

5. Advanced S3 Features

Bucket Version Awareness

If your S3 bucket has versioning enabled, you can read historical versions of objects. Set version_aware=True.

Warning: Enabling this causes s3.ls() to use the ListObjectVersions API, which lists all historical markers and can be significantly slower than standard listing. Only enable this if you need to read historical data.

s3 = s3fs.S3FileSystem(version_aware=True)

# Get all versions of a file
versions = s3.object_version_info('my-bucket/data.csv')

# Read a specific version
old_file = s3.open('my-bucket/data.csv', version_id=versions[1]['VersionId'])

Requester Pays

To access "Requester Pays" buckets (where you, the downloader, agree to pay the AWS egress charges), set requester_pays=True:

s3 = s3fs.S3FileSystem(anon=False, requester_pays=True)

Server-Side Encryption (SSE)

To enforce Server-Side Encryption, you must pass arguments directly to the underlying botocore client via s3_additional_kwargs.

from s3fs.utils import SSEParams

# Using standard AES256
s3 = s3fs.S3FileSystem(
    s3_additional_kwargs={'ServerSideEncryption': 'AES256'}
)

# Using KMS
params = SSEParams(
    server_side_encryption='aws:kms', 
    sse_kms_key_id='arn:aws:kms:region:account:key/YOUR-KEY-ID'
)

s3_kms = s3fs.S3FileSystem(s3_additional_kwargs=params.to_kwargs())

6. Multiprocessing Pitfalls

When using Python's multiprocessing library, the start method must be set to either spawn or forkserver.

If you use fork (the default on older Linux Pythons), child processes will inherit the open network sockets and the asynchronous event loop from the parent process. This leads to broken connections, IncompleteRead errors, and deadlocks.

import multiprocessing as mp

if __name__ == '__main__':
    # MUST be executed before any multiprocessing pools or s3fs instances are created
    mp.set_start_method('spawn')
    # ... application code

7. Logging & Debugging

If operations are failing or hanging, s3fs includes a built-in logger to reveal network requests and cache hits.

Set the environment variable S3FS_LOGGING_LEVEL before running your script:

export S3FS_LOGGING_LEVEL=DEBUG
python my_script.py

This will dump all aiobotocore interactions and s3fs API calls to stderr, making it easy to spot 403 Forbidden or 404 Not Found errors originating from AWS.