Schedules
Amorphic Schedule is for automating data ingestion, you can schedule batch and streaming data ingestion on a regular basis. This eliminates the need for manual intervention and ensures that data is always up-to-date. You set up custom schedules based on your specific needs.
How to create a Schedule?
Click on + New Schedule to create schedule and fill in the information shown below.
| Type | Description |
|---|---|
| Schedule Name | A unique name that identifies schedules' specific purpose |
| Job Type | You can pick a specific type type from the dropdown list (Details given in the Job type table below) |
| Schedule Type | There are are two schedule types
|
| Schedule expression | Time based schedules require a schedule expression. i.e., Every 15 min, daily, etc. |
If the schedule job type is 'Data Ingestion' and the dataset is of 'reload' type then schedule execution will load the data and also reload the data automatically.
| Type | Description |
|---|---|
| ETL Job | This option is used to schedule an ETL job. |
| JDBC CDC | This option is used to sync data between a data warehouse and S3 for change data capture (CDC) related tasks. Please note that only tasks with the "SyncToS3" option setting as "yes" will be shown and can be scheduled. |
| Data Ingestion | This option is used to schedule a data ingestion job for normal JDBC, S3 and external API connections. |
| JDBC FullLoad | This option is used to schedule a JDBC Bulk Data Load full-load task. |
| Forecast Predictors | This option is used to schedule a forecast predictor. |
| Forecast Reports | This option is used to schedule a forecast report. |
| Workflows | This option is used to schedule a workflow. |
| HCLS-Store | This option is used to schedule an import job for Omics Storage: Sequence Store, Omics Analytics: Variant Store, Annotation Store. |
Schedule details
Once you have created a schedule, you can view it on the schedules listing page, and perform various actions on it, such as running, disabling, enabling, editing, cloning, or deleting the schedule.
Run Schedule
To schedule a job, you can use the Run Schedule option located at the top right corner of the page. Once you run the schedule, you can check its status under the Execution Status tab. This will tell you if the job is currently running, if it was successful, or if it failed.
Schedule execution will error out if the related S3 connection is using any of Amorphic S3 buckets as source. For ex: <projectshortname-region-accountid-env-dlz>
Schedule use case
When the schedule execution is completed, an email notification will be sent out, based on the notification setting and schedule execution status. You can also view the execution logs of each schedule run, which include Output Logs, Output Logs (Full), and Error Logs.
For example, if you need to create a schedule that runs an ETL job and sends out important emails every 4 hours, you can create a workflow with an ETL Job Node followed by a Mail Node. This workflow can then be scheduled to run every 4 hours, every day.
