API Reference

This document provides an in-depth reference for the public classes and methods exposed by s3fs.

Core Classes

s3fs.S3FileSystem

The S3FileSystem class is the main entry point. It inherits from fsspec.asyn.AsyncFileSystem and manages the aiobotocore session, configuration, and directory-level tasks.

Initialization

S3FileSystem(
    anon=False, 
    endpoint_url=None, 
    key=None, 
    secret=None, 
    token=None, 
    use_ssl=True, 
    client_kwargs=None, 
    requester_pays=False, 
    default_block_size=52428800, 
    default_fill_cache=True, 
    default_cache_type='readahead', 
    version_aware=False, 
    cache_regions=False, 
    asynchronous=False, 
    config_kwargs=None, 
    session=None, 
    max_concurrency=10, 
    fixed_upload_size=False, 
    **kwargs
)

Parameters:

  • anon (bool, default False): Whether to use an anonymous connection (for public buckets only). If False, uses the provided credentials or AWS's default credential resolution.
  • endpoint_url (str, default None): Specify a custom URL for S3-compatible providers (e.g., MinIO, R2).
  • key (str, default None): AWS Access Key ID. Takes precedence over client_kwargs.
  • secret (str, default None): AWS Secret Access Key. Takes precedence over client_kwargs.
  • token (str, default None): AWS Security token (for assumed roles/STS).
  • use_ssl (bool, default True): Whether to encrypt the connection to S3. Disabling may increase speed on secure local networks but is otherwise insecure.
  • s3_additional_kwargs (dict, default None): Global parameters passed to S3 API calls (e.g., {'ServerSideEncryption': 'AES256'}).
  • client_kwargs (dict, default None): Parameters passed directly to the aiobotocore.client (e.g., {'region_name': 'eu-west-1'}).
  • config_kwargs (dict, default None): Parameters passed to botocore.client.Config (e.g., {'signature_version': 's3v4'}).
  • requester_pays (bool, default False): If True, adds RequestPayer='requester' to API calls.
  • default_block_size (int, default 50MB): The byte size of blocks used when buffering files in open().
  • default_fill_cache (bool, default True): If True, fills the buffer between sections to best support random access.
  • default_cache_type (str, default "readahead"): The caching strategy for opened files (e.g., "readahead", "bytes", "none").
  • version_aware (bool, default False): If True, ls will fetch object versions, and methods will respect version_id parameters.
  • cache_regions (bool, default False): Whether to cache bucket-to-region mappings internally.
  • asynchronous (bool, default False): If True, the class operates in an async context, and you must await set_session().
  • max_concurrency (int, default 10): The maximum number of concurrent threads/coroutines to use per file during multipart uploads (put).
  • fixed_upload_size (bool, default False): If True, forces all multipart upload chunks (except the last) to be exactly chunksize. Required by Cloudflare R2.

File & Directory Operations

  • open(path, mode='rb', block_size=None, acl=False, version_id=None, cache_type=None, ...) Opens a file at the given S3 path. Returns an S3File instance. mode must be a binary mode ('rb', 'wb', 'ab') or text mode. If writing, data is buffered and uploaded in multipart chunks.

  • ls(path, detail=False, refresh=False, versions=False) Lists files in a given bucket/path. If detail=True, returns a list of dictionaries with metadata (Size, LastModified, etc.). If versions=True (and version_aware is configured), returns historical versions.

  • find(path, maxdepth=None, withdirs=None, detail=False, prefix="") Recursively lists all files below a path, similar to the POSIX find command.

  • info(path, refresh=False, version_id=None) Retrieves a dictionary of metadata for a specific key or bucket. Raises FileNotFoundError if missing.

  • exists(path) Returns True if the bucket or key exists, False otherwise.

  • mkdir(path, acl=False, create_parents=True) Creates a bucket (if path is top-level) or a directory placeholder.

  • rmdir(path) Deletes an empty bucket or directory placeholder. Raises OSError if not empty.

  • rm(path, recursive=False) Removes file(s) or directories. If recursive=True, deletes all objects under the prefix using bulk delete.

  • cp_file(path1, path2, preserve_etag=None) Copies a file within S3. If preserve_etag=True and the file was multipart-uploaded, it attempts to recreate the exact multipart boundaries to preserve the ETag hash.

  • mv(path1, path2, recursive=False) Moves file(s). Equivalent to cp followed by rm.

Data Transfer & Utilities

  • get_file(rpath, lpath) / put_file(lpath, rpath) Downloads an S3 file to local disk, or uploads a local file to S3. Handled via concurrent byte range requests / multipart uploads for large files.

  • cat_file(path, start=None, end=None) Fetches the contents of a file directly into memory as bytes. Accepts byte ranges.

  • url(path, expires=3600, client_method='get_object') Generates an HTTP presigned URL valid for expires seconds. Useful for granting temporary download access to external clients.

  • checksum(path) Returns the unique ETag hex value for the file. Useful for validating sync operations.

  • get_tags(path) / put_tags(path, tags, mode='o') Retrieves or sets AWS Resource Tags on an S3 object. mode='m' merges with existing tags, 'o' overwrites.

  • setxattr(path, **kw_args) Modifies the AWS S3 Object Metadata. Because S3 metadata is immutable, this performs an in-place copy_object with the REPLACE metadata directive.

s3fs.S3File

The file-like object returned by S3FileSystem.open(). It extends fsspec's AbstractBufferedFile.

Key Methods

  • read(length=-1) Reads length bytes from the file. If -1, reads to the end. Leverages internal buffers and readahead caching.

  • write(data) Writes bytes to the file buffer. If the buffer exceeds block_size, a background multipart upload part is dispatched.

  • seek(loc, whence=0) Changes the file pointer position. If seeking outside the current buffer, it may trigger a new network request.

  • flush() Forces the internal write buffer to dispatch a multipart chunk to S3, even if it hasn't reached block_size.

  • close() Finalizes the file. If writing, it issues the complete_multipart_upload API call to AWS, making the file visible in the bucket.

  • commit() / discard() If the file was opened with autocommit=False, close() will not finalize the upload. You must explicitly call commit() to save the file or discard() to abort the upload.

Utility Functions

s3fs.S3Map

S3Map(root, s3, check=False, create=False)
Creates a mutable mapping (dictionary-like object) out of an S3 path. Keys correspond to object names, and values are byte strings. Setting a key uploads a file; deleting a key removes the file.

s3fs.add_retryable_error

add_retryable_error(exc)
Registers a new Exception class to the global tuple of retryable errors. If this exception is raised by aiobotocore, s3fs will back off and retry instead of failing immediately. See Error Handling.

s3fs.set_custom_error_handler

set_custom_error_handler(func)
Registers a custom function to evaluate whether an exception should trigger a retry. The function must accept a single Exception argument and return a boolean.

s3fs.utils.SSEParams

SSEParams(
    server_side_encryption=None, 
    sse_customer_algorithm=None, 
    sse_customer_key=None, 
    sse_kms_key_id=None
)
A data structure helper to generate valid kwargs for Server-Side Encryption. Call .to_kwargs() on the instance and pass the result to s3_additional_kwargs.