S3Fs: Pythonic File Interface for Amazon S3

S3Fs is a high-performance, Pythonic file interface to Amazon S3. Built on top of aiobotocore, it provides a seamless, synchronous, and asynchronous filesystem interface for S3 storage.

By leveraging the fsspec (Filesystem Specification) architecture, s3fs allows you to interact with S3 buckets and objects as if they were local files on your machine. This abstraction enables a wide array of data science, machine learning, and engineering tools to read and write directly to cloud storage without requiring custom boto3 implementation.

Why Use S3Fs?

Working directly with S3 SDKs (like boto3) often requires managing pagination, multipart uploads, byte-range requests, and network retries manually. s3fs handles all of this under the hood, providing a familiar Python file object interface.

  • Pythonic Interface: Use standard open(), read(), write(), and filesystem operations like ls(), cp(), and rm().
  • Seamless Ecosystem Integration: Works out-of-the-box with pandas, dask, xarray, zarr, and other data libraries that support fsspec.
  • High Performance: Supports concurrent multipart uploads, asynchronous I/O via aiobotocore, and customizable intelligent caching (readahead, block, mmap).
  • Resilient: Built-in, customizable retry logic for transient network errors, rate limits, and incomplete reads.

When to Use This

  • Data Engineering Pipelines: Reading and writing large parquet, CSV, or JSON datasets directly from S3 using tools like pandas or Dask.
  • Machine Learning: Streaming large training datasets from S3 into memory without downloading them entirely to local disk.
  • Cloud-Native Applications: Applications that need to treat cloud object storage as a traditional POSIX-like filesystem.

Core Concepts

The library revolves around two primary classes:

  1. S3FileSystem: The main entry point. It manages authentication, connection pooling, and directory-level operations (listing, moving, removing). See the API Reference for details.
  2. S3File: The file-like object returned when you call open() on an S3FileSystem. It mimics Python's built-in file objects, buffering reads and handling multipart uploads for writes automatically.

Project Support & Maintenance

s3fs is community-maintained and acts as the official S3 backend for the fsspec project.

Work on this repository is supported in part by:

"Anaconda, Inc. - Advancing AI through open source."