Using S3-Compatible Storage Providers

Because s3fs is built on botocore, it is not strictly limited to Amazon Web Services (AWS). It can be used to interact with any storage provider that implements the S3 API.

Common alternatives include MinIO, Ceph Object Gateway, Cloudflare R2, Scaleway, OVH, and Storj DCS.

Core Concept: The Endpoint URL

To point s3fs at a non-AWS provider, the most critical parameter is the endpoint_url. By default, botocore attempts to resolve endpoint URLs based on AWS regions (e.g., s3.us-east-1.amazonaws.com).

You can override this directly in the S3FileSystem constructor.

MinIO (Local or Self-Hosted)

MinIO is widely used for local testing and private cloud storage.

import s3fs

# Connecting to a local MinIO instance
s3 = s3fs.S3FileSystem(
    key='your-minio-root-user',
    secret='your-minio-root-password',
    endpoint_url='http://127.0.0.1:9000',
    # Many local MinIO setups do not use SSL.
    use_ssl=False 
)

Using Environment Variables via fsspec

If you are using pandas or dask and relying on their fsspec integrations, it is often easier to set connection parameters via environment variables. fsspec intercepts variables prefixed with FSSPEC_S3_.

export FSSPEC_S3_ENDPOINT_URL="http://127.0.0.1:9000"
export FSSPEC_S3_KEY="minioadmin"
export FSSPEC_S3_SECRET="minioadmin"

import pandas as pd

# Pandas will automatically pick up the FSSPEC_S3_ environment variables
df = pd.read_csv("s3://my-local-minio-bucket/data.csv")

Cloudflare R2

Cloudflare R2 is highly compatible with S3 but has specific requirements regarding multipart uploads. R2 generally requires that all parts in a multipart upload (except the last) be exactly the same size.

You must pass the fixed_upload_size=True flag to accommodate R2.

s3_r2 = s3fs.S3FileSystem(
    key='your-r2-access-key-id',
    secret='your-r2-secret-access-key',
    endpoint_url='https://<ACCOUNT_ID>.r2.cloudflarestorage.com',
    fixed_upload_size=True, # Crucial for Cloudflare R2
)

Scaleway (Europe)

Scaleway Object Storage requires you to define the specific region name within client_kwargs to ensure signatures match properly.

s3_scw = s3fs.S3FileSystem(
    key='scaleway-api-key',
    secret='scaleway-secretkey',
    endpoint_url='https://s3.fr-par.scw.cloud',
    client_kwargs={
        'region_name': 'fr-par'
    }
)

OVH Cloud

Similarly, OVH requires specific regions and explicitly forcing the Signature Version 4 (s3v4) via config_kwargs.

s3_ovh = s3fs.S3FileSystem(
    key='ovh-s3-key',
    secret='ovh-s3-secretkey',
    endpoint_url='https://s3.GRA.cloud.ovh.net',
    client_kwargs={
        'region_name': 'GRA'
    },
    config_kwargs={
        'signature_version': 's3v4'
    }
)

Troubleshooting Alternative Providers

When working with non-AWS providers, you may encounter specific errors. Here are the most common pitfalls and solutions:

1. Virtual-Hosted vs. Path-Style Requests: AWS S3 defaults to virtual-hosted style requests (https://bucket-name.s3.region.amazonaws.com). Many self-hosted providers (like MinIO on localhost) require path-style requests (http://localhost:9000/bucket-name). To force path-style addressing, modify the config_kwargs:

   s3 = s3fs.S3FileSystem(
       endpoint_url='http://127.0.0.1:9000',
       config_kwargs={'s3': {'addressing_style': 'path'}}
   )

2. Signature Mismatches: If you receive a 403 Forbidden with a "SignatureDoesNotMatch" error, it often means the region_name inferred by botocore does not match what the endpoint expects. Explicitly set client_kwargs={'region_name': '...'}.

3. Missing Features: Not all S3-compatible providers support all S3 APIs. Features like Bucket Versioning, Server-Side Encryption, or specific Canned ACLs may throw 501 NotImplemented or 400 Bad Request errors. Disable these features in s3fs when communicating with minimal S3 implementations.