Schedules
Amorphic Schedule provides a self-service data ingestion mechanism which provides a mechanism for batch and streaming data ingestion on a scheduled basis.
Create Schedule
A Schedule can be created using the + icon on the top right of page
under 'Schedules' section.
In order to create a schedule, User needs to specify all the mandatory requirements.
Schedule Name: A Unique name of schedule that user wants to create.
Description: Brief description of the schedule.
Type of Job: User needs to choose the type of Job from the dropdown list. Below are the list of options available for schedule.
| Type | Description |
|---|---|
| ETL Job | This option is used to schedule an ETL job |
| JDBC CDC | This option is used to Sync data between DWH to S3 for CDC related tasks, Please note that only tasks which have SyncToS3 option setting as yes will be shown and can be scheduled. |
| Data Ingestion | Data ingestion job for normal JDBC, S3 and Ext-API connections. |
| JDBC FullLoad | This option is used to schedule a JDBC BulkDataLoad full-load task. |
| Forecast Predictors | This option is used to schedule a Forecast predictor. |
| Forecast Reports | This option is used to schedule a Forecast report. |
| Workflows | This option is used to schedule a workflow. |
Notification settings has been moved to Settings page and using that user can choose the type of notification setting for the schedule. Please check the documentation on Notification settings.
Max Capacity: User can specify max capacity based on the data load. For JDBC type datasets, must be an integer value between 2 and 100, else either 0.0625 or 1. Allocated Capacity earlier used has been deprecated.
Schedule Type: The user can choose the schedule type whether to be On-Time or On-Demand.
On-Demand schedules can be run as per the need.
User can execute the schedules based on specific time as per requirement. Time based schedules require a schedule expression.
If the schedule job type is 'Data Ingestion' and the dataset is of 'reload' type then schedule execution will load the data and also reload the data automatically.
Schedule details
To view the details of a schedule, click the View Details link for the respective schedule. in the schedules listing page.
There are multiple options available to perform on the scheduled job:
- Run Schedule
- Disable Schedule
- Enable Schedule
- Edit Schedule
- Clone Schedule
- Delete Schedule
Run Schedule
Once a schedule is created for a specific job, the schedule can be using the Run Schedule Option available at the top right icon on the page.
Once the user runs the schedule then the user can check the status of the execution through 'Execution Status' tab.
Execution status indicates the current status of the scheduled job whether is it running, successful or failed.
Once the schedule execution is completed, Email notification will be sent based on the notification setting and schedule execution status.
Schedule execution will error out if the related S3 connection is using any of Amorphic S3 buckets as source. For ex: <projectshortname-region-accountid-env-dlz>
From Amorphic 1.13 Onwards, ExecutionTime key is changed to StartTime which is relevant for the execution. For executions, it'll display ExecutionTime if StartTime is not available.
User can stop/cancel the schedule execution if the execution is in running state.
Download logs
Once a schedule status is updated, user can download the output (if any) and error logs (if any) through more (3 dots) option. The logs option is of 3 types:
- Output Logs (latest 1 MB): The latest 1 MB of the output logs for the schedule execution. This option will download the latest 1 MB of output logs immediately. If logs are not available, It'll display that 'No output logs available for the execution' message.
- Output Logs (All): All of the output logs for the schedule execution. This option will initiate log file creation. User will be notified through email once the log file is created. If there are no output logs then the log file will be empty. This option is applicable only for schedules with 'ETL job' type.
- Error Logs: Error logs for the schedule execution. If logs are not available, It'll display that 'No error logs available for the execution' message.
If the user opts to download the full logs then it initiates the log file creation and the status will be 'triggered'. Status will be changed to 'available' and user will receive the email once log file is created. User can download the full logs using the same 'Output Logs (All)' option.
Disable Schedule
To disable the schedule, click the Disable Schedule (- minus) button on the right side options of the details page.
If the schedule is disabled successfully, then it will not run until it is enabled again.
Enable Schedule
To enable the schedule, click the Enable Schedule (tick) button on the right side options of the details page.
Once the schedule is enabled successfully, then it will resume the executions based on the time specified if it is of type 'time-based' or user can trigger it manually anytime if it is of type 'on-demand'.