S3 Connections
S3 Connections are used to migrate data from a remote S3 bucket to Amorphic Data Cloud. Using these connections data can be written directly into an Amorphic Dataset. There are two types of S3 Connections available, using - Bucket Policy - Access Keys
Create connection
Bucket Policy
To create an S3 Connection using a bucket policy, user has to first select the bucket from which the data needs to be migrated. Once the connection is created, a bucket policy will be available for User to download. Bucket policy generated should be attached to the source bucket which was added to connection during creation. Below image shows how the connection with bucket policy is created
Access Keys
To create an S3 Connection using access keys, user has to select the bucket from which the data has to be migrated and provide access key and secret access key of the user who has permissions to read the data from the bucket.
Below image shows how the connection with access keys is created
| Attribute | Description |
|---|---|
| Connection Name | Name of the connection Amorphic |
| Connection Type | Type of connection. In this case it is S3 |
| Description | Connection related information user wants to store |
| Authorized Users | Amorphic users to whom user wants to have access to this connection |
| S3 Bucket | Name of the bucket from which the dataset files have to be imported. |
| Connection Access Type | Access type for the connection, can be one of Access Keys and Bucket Policy |
| Version | Enables the user to select what version of ingestion scripts to use (Amorphic specific). For any new feature/Glue version that gets added to the underlying ingestion script, new version will be added to the Amorphic. |
| S3 Bucket Region | Region where source S3 bucket is created. If the source bucket is in one of the regions (eu-south-1, af-south-1, me-south-1,ap-east-1) then this property needs to be provided and the region needs to be enabled in Amorphic else ingestion fails. |
User cannot specify any of the Amorphic S3 buckets. For ex: <projectshortname-region-accountid-env-dlz>
Edit connection
There is an option to edit an S3 Connection. To edit an S3 Connection, click the edit button which on the right corner.
Description, Authorized users, S3Bucket, Keywords and S3 Bucket Region of an S3 Connection can be changed.
Below image shows how to edit an S3 connection
Upgrade connection
User can upgrade a connection (if available) which will be shown in the options. Upgrade option is visible only if there is any new version available else it is not shown.
Connection upgrade upgrades the underlying Glue version and the data ingestion script with new features.
Below image shows how to upgrade a connection
Downgrade connection
User can downgrade a connection to previous version if he thinks the upgrade is not serving the purpose. A connection can only be downgraded if it has been upgraded. For a connection that has been created we cant downgrade it. The option to downgrade is available on the top right corner if the connection is downgrade compatible.
Delete connection
On the top right corner, there is a button with trash can on it. Click on it to delete
Below image shows how to delete a connection
Connection Versions
1.6
In this version of s3 connection, the data ingestion happens by considering and comparing ETag of the files in source and target.
First we check the file name, if file exists, size is same then we get the ETags of the source and current file in dataset. If they match then we do the ingestion. This is because an ETag of a file doesn't change when the file name changes and if the user intends to duplicate the files by changing the names then he won't be able to if only ETag is considered.
In this version, only files stored in S3 Standard class are supported for S3 data ingestion. If there exist files from other storage classes, the ingestion process will fail.
1.7
In this version of s3 connection, the storage classes of files do not affect the flow of ingestion.
That means, the ingestion process will not be terminated even if there exist files from storage classes other than S3 Standard. We just skip that files from ingestion, then show the details of skipped files and successfully complete the ingestion without any failure.
Only files that are stored in S3 Standard classes are selected, and all other files are skipped from S3 data ingestion.