org.apache.hadoop.fs.s3a.DefaultS3ClientFactory Java Examples

The following examples show how to use org.apache.hadoop.fs.s3a.DefaultS3ClientFactory. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.

Example #1

Source File: S3FileSystem.java From dremio-oss with Apache License 2.0

6 votes

@Override
public AmazonS3 load(S3ClientKey clientKey) throws Exception {
  logger.debug("Opening S3 client connection for {}", clientKey);
  DefaultS3ClientFactory clientFactory = new DefaultS3ClientFactory();
  clientFactory.setConf(clientKey.s3Config);
  final AWSCredentialProviderList credentialsProvider = S3AUtils.createAWSCredentialProviderSet(S3_URI, clientKey.s3Config);
  final AmazonS3 s3Client = clientFactory.createS3Client(S3_URI, "", credentialsProvider);

  return registerReference(AmazonS3.class, s3Client, client -> {
    client.shutdown();

    try {
      // Note that AWS SDKv1 client will NOT close the credentials provider when being closed so it has to be done ourselves
      // Because client still holds a reference to credentials provider, it won't be garbage collected until the client is garbage collected itself
      if (credentialsProvider instanceof AutoCloseable) {
        ((AutoCloseable) credentialsProvider).close();
      }
    } catch (Exception e) {
        logger.warn("Failed to close AWS credentials provider", e);
    }
  });
}

Example #2

Source File: S3SelectAccessor.java From pxf with Apache License 2.0

5 votes

/**
 * Returns a new AmazonS3 client with credentials from
 * the configuration file
 */
private AmazonS3 initS3Client() {
    try {
        DefaultS3ClientFactory factory = new DefaultS3ClientFactory();
        factory.setConf(configuration);
        return factory.createS3Client(name);
    } catch (IOException e) {
        throw new RuntimeException("Unable to create S3 Client connection", e);
    }
}