Quick Start

This guide will get you up and running with s3fs quickly. Because s3fs faithfully copies the standard Python file interface, it is highly intuitive. If you know how to read and write files locally in Python, you already know how to use s3fs.

1. Authentication Setup

Before running the code, you need to ensure s3fs can talk to AWS. s3fs automatically uses the standard boto3 credential resolution chain.

The easiest way to authenticate locally is to have the AWS CLI installed and configured:

aws configure

Alternatively, you can export environment variables in your terminal:

export AWS_ACCESS_KEY_ID="your_access_key"
export AWS_SECRET_ACCESS_KEY="your_secret_key"
export AWS_DEFAULT_REGION="us-east-1"

If you only need to access publicly available data, you can skip this step and use anon=True.

2. Basic Setup and Directory Listing

First, import the library and instantiate the file system. 

import s3fs

# Create a filesystem instance. 
# anon=False tells s3fs to look for the credentials we set up above.
s3 = s3fs.S3FileSystem(anon=False)

# List the contents of a bucket
contents = s3.ls('my-data-bucket/')
print(contents)
# Output: ['my-data-bucket/dataset.csv', 'my-data-bucket/images/']

3. Reading and Writing Files

Writing a File

You can use standard open syntax with a context manager to write files. s3fs buffers the write and handles the S3 Multipart Upload process automatically in the background.

# Open a file in write-binary mode ('wb')
with s3.open('my-data-bucket/output/hello.txt', 'wb') as f:
    f.write(b'Hello from s3fs!\n')
    f.write(b'This file was uploaded directly to S3.')

# The file is fully uploaded to S3 when the context manager exits (f.close() is called)

Reading a File

Reading is just as simple. s3fs implements intelligent caching (like readahead buffers) to optimize network requests.

# Open a file in read-binary mode ('rb')
with s3.open('my-data-bucket/output/hello.txt', 'rb') as f:
    content = f.read()
    print(content.decode('utf-8'))

4. Reading Specific Data Blocks

If you are dealing with massive files (like gigabyte-sized CSVs) and only need a specific chunk, you can use read_block. This method reads a specific byte range and ensures it stops cleanly on a specified delimiter (like a newline).

# Read roughly 100 bytes starting at offset 1000.
# It will read forward until it hits a newline character to ensure complete lines.
block = s3.read_block(
    'my-data-bucket/massive_dataset.csv', 
    offset=1000, 
    length=100, 
    delimiter=b'\n'
)
print(block)

5. Ecosystem Integration (Pandas Example)

s3fs really shines when combined with the PyData ecosystem. Libraries like Pandas and Dask recognize the s3:// prefix and automatically invoke s3fs under the hood.

import pandas as pd

# Pandas uses s3fs automatically to stream the file into a DataFrame.
# The storage_options dictionary is passed directly to the s3fs.S3FileSystem constructor.
df = pd.read_csv(
    "s3://my-data-bucket/reports/financials.csv",
    storage_options={"anon": False, "profile": "my-aws-profile"}
)

print(df.head())

Common Pitfalls for Beginners

  1. Text vs. Binary Mode: In older versions, s3fs strongly preferred binary modes ('rb', 'wb'). While text mode is now supported, it is generally safer and more performant to read/write in binary and decode/encode explicitly, or let a library like pandas handle the encoding.
  2. Missing Region Configuration: If you encounter botocore.exceptions.ClientError: Bad Request, it often means the AWS region is not configured correctly. Ensure your environment specifies a region, or pass it explicitly: client_kwargs={'region_name': 'us-west-2'}.
  3. Forgetting to Close Files: If you don't use a with statement context manager, you must call f.close(). For writes, s3fs does not finalize the multipart upload to S3 until close() is called. If your script crashes before close(), the file will not appear in the bucket.

Next, explore the Usage Guide to learn about async support, concurrency, and caching strategies.