Custom Authentication & External File Systems

[andygrove]

What is the problem the feature request solves?

We need to support custom authentication with external file systems

(Access hdfs/hadoop-aws via JNI)

1. HDFS support via fs-hdfs

• Fork into Comet
• PRs open in fs-hdfs
• One draft PR
• Update Comet to use internal fork

2. OpenDAL

• Investigate OpenDAL features #2367

Describe the potential solution

No response

Additional context

No response

[Xuanwo]

Hi, could I ask for a bit more context? Maybe I can take a look and help.

[parthchandra]

The AWS SDKs (both V1 and V2) provide a basic set of authentication providers which are used by the (Java) S3Client to authenticate a request for any AWS resource. (https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-explicit.html).

Hadoop-aws extends these to provide a Hadoop specific method of accessing S3 (https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Authenticating_with_S3) which allows Hadoop applications to access S3 via the HDFS client API (as opposed to the S3 client API).

Almost all Java and Hadoop based engines use the HDFS client APIs (via Hadoop-aws) to access S3. Spark, for instance, uses the HDFS APIs. Some implementations like Iceberg (java) can use both methods to access S3 (and allow users to choose which API should be used).

The Hadoop client APIs also allow end users to specify their own credentials providers. An example use case is a credentials provider that will first authenticate a user against a proprietary authentication system, apply any organization specific access control, and then interface with AWS to get AWS credentials. The application then uses these credentials with the HDFS client which in turn uses them with the AWS SDK.

Some credentials providers will also implement credentials caching and credentials refreshing which is useful for long running jobs that are typical of batch workloads in Spark.

Almost all such custom credentials providers are written in Java, and because they are organization specific, cannot be implemented by us (Comet/OpenDAL) in native code.

[Xuanwo]

Thank you @parthchandra for the context. I didn't have much background on Hadoop, Spark, or Comet, so please correct me if I'm misunderstanding anything.

So our current blocker is that OpenDAL/Requisiing doesn't support passing custom credential providers written in Java. And once we (or Comet) build such a bridge, we can switch to using OpenDAL?

[mbutrovich]

So our current blocker is that OpenDAL/Requisiing doesn't support passing custom credential providers written in Java. And once we (or Comet) build such a bridge, we can switch to using OpenDAL?

I think if we can come up with an API that makes sense for arbitrary authentication providers to be passed in like the EncryptionFactory in DataFusion...

https://github.com/apache/datafusion/blob/20870c18a418ec081d44ecf8a90a30a95aa53138/datafusion-examples/examples/data_io/parquet_encrypted_with_kms.rs#L217

...then we might be in business. Basically, Parquet Modular Encryption allows for arbitrary key provider implementations, so you can define a class that unwraps keys with whatever code it sees fit. In Comet, we use that API to build a JNI bridge to any Java/Scala PME key providers a Spark user might have.

https://github.com/apache/datafusion-comet/pull/2447/files

I'm imagining a similar design for authentication, but I'm not sure it just handles signing requests or works at a different abstraction.

[parthchandra]

It isn't just custom authentication. Hadoop's FileSystem is an abstract class and users can extend the concrete implementations available or implement their own. One use case is to provide an implementation to a custom object store. Another use case is to extended an existing implementation to provide stricter access to individual files. Both Spark and Hadoop allow an end user to use such a custom implementation.

To support this we will end up having to implement a JNI layer to access the entire FileSystem API.

Neither this, nor custom auth is a hard requirement for OpenDAL to be supported by Comet. We can have both fs-hdfs and OpenDAL as options to access HDFS. OpenDAL provides us with the ability to access storage systems beyond HDFS while fs-hdfs provides us with more specialized Hadoop specific functionality.

[Xuanwo]

I see. Thank you @parthchandra for the extra context.

To support this we will end up having to implement a JNI layer to access the entire FileSystem API.

Btw, OpenDAL has its own Java binding. Would it be useful for the OpenDAL community to build a Hadoop FileSystem implementation using that Java binding?

Neither this, nor custom auth is a hard requirement for OpenDAL to be supported by Comet.

Happy to know this. So if we want to move forward, what steps can we take?

[parthchandra]

I see. Thank you @parthchandra for the extra context.

To support this we will end up having to implement a JNI layer to access the entire FileSystem API.

Btw, OpenDAL has its own Java binding. Would it be useful for the OpenDAL community to build a Hadoop FileSystem implementation using that Java binding?

I believe so, because we could potentially eliminate fs-hdfs.

Neither this, nor custom auth is a hard requirement for OpenDAL to be supported by Comet.

Happy to know this. So if we want to move forward, what steps can we take?

We have support for hdfs via fs-hdfs and OpenDAL but I'm not sure we have adequately defined what more we need from OpenDAL. I would think that at a minimum the tests defined in ParquetReadFromFakeHadoopFsSuite and ParquetReadFromS3Suite should pass for both the implementations.

I also feel that support for custom auth and custom file systems is required in the longer term.

Also, the implementations are behind a feature flag, but it might be easier to test/compare if we could choose the implementation based on a configuration flag.