Missing AWS Profile Support in PyIceberg

[thomas-pfeiffer]

Feature Request / Improvement

Feature: Missing AWS Profile Support in PyIceberg / PyIceberg should support AWS profiles

Description:
When working with multiple AWS configs / credentials in parallel, AWS profiles are a convenient way to achieve this. Ideally, PyIceberg should therefore also support AWS profiles, which it currently does not.

Current state (as of writing - pyIceberg v0.10.0):

• The Glue part of the GlueCatalog can be configured to use the profile by specifying the Glue client explicitly in the Glue Catalog or via glue.profile-name config parameter:

from boto3 import Session
...
catalog = GlueCatalog(name="your_glue_catalog",client=Session(profile_name="your_aws_profile").client("glue"),...)

or

catalog = GlueCatalog(
    name="your_glue_catalog",
    **{ 
        "glue.profile-name": "your_aws_profile",
        ...
    },
)

• For fsspec backends, AWS profile support is generally available (see https://s3fs.readthedocs.io/en/latest/api.html#s3fs.core.S3FileSystem and https://github.com/fsspec/s3fs/blob/aceea3a4985f667979e4d8a5a5b8eeddaf23b7be/s3fs/core.py#L229), but it's not implemented in PyIceberg (see

  iceberg-python/pyiceberg/io/fsspec.py

  Line 199 in e07296e

  fs = S3FileSystem(anon=anon, client_kwargs=client_kwargs, config_kwargs=config_kwargs)

  ). To change that we would need to set the session parameter of the S3FileSystem explicitly:

from s3fs import S3FileSystem
from aiobotocore.session import AioSession
...
fs = S3FileSystem(session=AioSession(profile="your_aws_profile"),...)

• For PyArrow backend, the AWS profile support is not yet available, but they do have an enhancement ticket for it (see [Python][C++] Add Profile support to S3FileSystem arrow#47880). Once AWS profile is supported in PyArrow it can be implemented in PyIceberg as well, I assume.

Workaround for this feature gap:

session = Session(profile_name="your_aws_profile")
credentials = session.get_credentials()  
if credentials is None:
    raise ValueError("Could not retrieve credentials for profile")
catalog = GlueCatalog(
    name="your_glue_catalog",
    **{ 
        "client.access-key-id": credentials.access_key,
        "client.secret-access-key": credentials.secret_key,
        "client.session-token": credentials.token,
        ...
    },
)

To-Be / Expected Behavior:

1. PyIceberg should have a new client.profile-name and s3.profile-name configuration parameter (next to existing glue.profile-name.
2. New client.profile-name should also set glue.profile-name (same behaviour as for all the other unified AWS credentials).
3. For now, AWS profile support should be implemented for fsspec backend and client.profile-name and s3.profile-name should only be supported when using fsspec backend ("py-io-impl": "pyiceberg.io.fsspec.FsspecFileIO").
4. Once PyArrow supports AWS profile names (see [Python][C++] Add Profile support to S3FileSystem arrow#47880), AWS profile support should be implemented for PyArrow backend as well and client.profile-name and s3.profile-name should be fully supported.

Remark: I found this feature gap with the GlueCatalog; it might be that the RestCatalog is equally affected, but not sure.
Issues possibly related to this issue: #570, #1207, #2657