Set custom endpoint uri

[lcaquot94]

I want to use custom endpoint URL to use s3pathlib with Minio.

From the documentation below, I can see it is easy to change many parameters, but not the endpoint_url which is defined inside the client. And in resolve_s3_client, context.s3_client returns a new client so it is not possible to change its endpoint.

from s3pathlib import context
context.attach_boto_session(
     boto3.session.Session(
          region_name="us-east-1",
          profile_name="my_aws_profile",
     )
)

I would like to change the endpoint so I can use a self hosted minio instead of default AWS endpoint.

[MacHu-GWU]

I think you can do that. Firstly the s3_client is cached by default, which means that it is a multable object, and you can custom it's endpoint after creation. Another method would be, explicitly create the boto session object using boto3.session.Session with the custom endpoint, and explicitly give it to bsm and s3pathlib.context.

from s3pathlib import context
import boto3
boto_ses = boto3.session.Session(...) # customize the endpoint
context.attach_boto_ses(boto_ses)

[alspadoni]

The custom endpoint belongs to the client, not the session. Is it troublesome to add support for custom endpoint in context?

[MacHu-GWU]

The custom endpoint belongs to the client, not the session. Is it troublesome to add support for custom endpoint in context?

@alspadoni add support to this feature in 2.3.2. Now you can do context.attach_s3_client(your_custom_s3_client)

[MacHu-GWU]

MARK THIS ISSUE CLOSED

[sergun]

Just vote for this feature

[MacHu-GWU]

@sergun THIS FEATURE IS ALREADY SUPPORTED FROM 1.X.Y. Here's an sample code to show how to use s3pathlib to connect to cloudflare R2, using custom endpoint.

Create API Token
step1.png
step2.png
Sample Code


# -*- coding: utf-8 -*-
import json
from pathlib import Path
import boto3
from s3pathlib import S3Path, context
path = Path.home().joinpath(
    ".cloudflare",
    "r2-account-token.json",
)
cred = json.loads(path.read_text())
token = cred["token"]
access_key = cred["access_key"]
secret_key = cred["secret_key"]
endpoint = cred["endpoint"]
s3_client = boto3.client(
    service_name="s3",
    endpoint_url=endpoint,
    aws_access_key_id=access_key,
    aws_secret_access_key=secret_key,
    region_name="auto",
)
context._s3_client = s3_client
s3path = S3Path(f"s3://my-bucketdev/test.txt")
s3path.write_text("hello world")
print(s3path.read_text())

# or you can use 
s3path.write_text("hello world", bsm=s3_client)