Amazon S3 connection strings

S3 URIs describe the bucket, optional credentials, and object prefix. Connect understands native AWS endpoints and custom S3-compatible services such as MinIO.

Basic bucket access

• s3://my-bucket — bucket root with default authentication (AWS profile, IAM role, or alias-provided credentials).
• s3://my-bucket/data/ — prefix within the bucket; trailing slash is optional but clarifies folder intent.

Authentication priority

When credentials are not embedded in the URI, Connect walks the default AWS chain:

1. AWS_REGION (or AWS_DEFAULT_REGION) and credentials from environment variables.
2. Profiles defined in ~/.aws/credentials and ~/.aws/config (honors AWS_PROFILE).
3. IAM roles or container credentials exposed via the metadata service.
4. Fallbacks to us-east-1 if no region is discovered.

Inline credentials

• s3://ACCESS_KEY:SECRET_KEY:eu-central-1@my-bucket/data/2024/ — classic access key and secret with region.
• s3://ACCESS:SECRET:us-east-1@host.example.com:9000@minio-bucket/backups/ — adds a custom endpoint host and port for MinIO or other S3-compatible storage.

Secrets inside URIs are convenient for testing, but we recommend storing them in aliases or using the AWS credential chain in production.

Temporary sessions

• s3://AKIA...:SECRET:eu-west-1:SESSION_TOKEN@my-bucket/delta/ — append the session token after the region for STS credentials (wrap in quotes to avoid shell parsing issues).

Absolute vs relative keys

• s3://my-bucket//absolute/path/file.zip — double slash indicates that the key begins with / exactly as written.
• s3://my-bucket/archive/file.zip — standard object key without a leading slash.

Pattern support

• Globs: *.txt, logs/*.json, reports/file[0-9].csv.
• Regular expressions: prefix with regex: (regex:^2024-.*\\.parquet$).

Putting it together

$ connect sync -r --parallel 4 \
    /data/export/ \
    s3://AKIA...:SECRET:us-east-1@analytics-landing/incoming/

This sync mirrors /data/export/ to the incoming/ prefix of the analytics-landing bucket using inline credentials.

Additional examples:

• s3://analytics-landing/logs/*.gz — list or copy matching objects using default AWS credentials.
• s3://AKIA...:SECRET:eu-west-1@finance-bucket/regex:.*\\.csv$ — pull every CSV from finance-bucket in eu-west-1.
• s3://access:secret:us-east-1@s3.internal.local:9000@minio/backups/ — target a MinIO cluster with a non-standard endpoint.