Scheduler & Advanced Settings
Managing data replication and transformation workflows requires precise execution timing, automated fail-safes, and proactive resource monitoring to ensure data integrity and operational efficiency.
Edilitics provides flexible scheduling options and advanced settings that allow users to:
✔ Automate replication and transformation workflows with fine-tuned scheduling controls.
✔ Prevent workflow failures by enabling auto-pause triggers for schema changes and execution failures.
✔ Proactively manage compute consumption through admin-level usage monitoring and notifications.
This guide details how to configure and optimize scheduling, auto-pause mechanisms, and enterprise-level monitoring for seamless data operations.
Automated Validation: Ensuring Workflow Readiness
Before executing any replication or transformation workflow, Edilitics automatically validates database connections and table accessibility as the first step of execution.
✔ Ensures all required connections are active and tables are accessible before processing begins.
✔ Prevents failures caused by missing access permissions, unavailable resources, or updated/out-of-date credentials.
✔ If validation fails, the workflow stops immediately, preventing incomplete or incorrect data processing.
💡 Best Practice: This built-in validation ensures that users only run workflows on properly configured and accessible databases, reducing troubleshooting time and compute wastage.
Workflow Conflict Detection: Preventing Scheduling Overlaps
To maintain data consistency and avoid conflicts between dependent workflows, Edilitics automatically detects scheduling conflicts when users attempt to save a new replication or transformation workflow.
✔ Automated Conflict Detection – When scheduling a new workflow, the system scans all existing scheduled workflows to check whether:
- The same table(s) being used as a source in the new workflow are already being used as a destination table in an existing replication or transformation workflow.
✔ Prevents Data Overwrites – If a conflict is detected, users receive a notification prompting them to adjust the schedule so that the existing workflow has time to complete before the new one starts.
✔ Optimized Execution Timing – Ensures that each workflow processes the most recent, fully updated dataset, reducing the risk of incomplete or inconsistent data processing.
💡 Best Practice: When scheduling multiple dependent workflows, stagger execution times strategically to ensure each process has adequate time to complete before the next one starts.
Scheduling Workflows: Automating Data Processing
Edilitics’ Scheduler allows users to configure custom execution schedules for both replication and transformation workflows.
Scheduling Options
| Type | Description |
|---|---|
| One-Time Execution | Runs a single workflow instance, ideal for ad-hoc data processing. |
| Daily Scheduling | Automates daily updates at specific times to keep data synchronized with business operations. |
| Custom CRON Expressions | Allows granular scheduling control using CRON syntax for complex execution patterns (e.g., every 2 hours on weekdays: 0 0 */2 * 1-5). |
✔ Both Replications & Transformations can be scheduled using the same framework to ensure consistent workflow execution.
Auto-Pause Mechanisms: Safeguarding Data Integrity
To ensure data consistency, Edilitics includes automated pause triggers that halt execution when inconsistencies are detected.
Auto-Pause on Schema Change
✔ Automatic Suspension – If schema modifications are detected in the source database, the replication job is paused immediately.
✔ Comprehensive Schema Monitoring – The system tracks:
-
Column additions or deletions
-
Data type modifications
-
Column renaming
✔ Prevents Data Corruption – Ensures schema changes do not propagate errors into downstream workflows.
✔ Immediate Notifications – Alerts are sent via in-app notifications, Slack, and email for quick intervention and resolution.
Auto-Pause on Failures
✔ Failure Detection & Prevention – Automatically halts workflows if multiple consecutive failures occur.
✔ Customizable Failure Thresholds – Users can choose between 0, 1, 2, or 3 consecutive failures before the system pauses execution.
-
If set to 0, the workflow will not auto-pause, and it will continue running on its next scheduled execution regardless of failures.
-
Fallback Mechanism: If a workflow experiences 3 consecutive failures, it will automatically pause, and the user will be notified even if auto-pause was set to 0.
✔ Real-Time Alerts – Admins & workflow owners receive notifications via in-app alerts, Slack, and email for immediate corrective action.
💡 Best Practice: Use higher failure thresholds (e.g., 2-3 failures) for temporary issues and lower thresholds (1-2 failures) for business-critical workflows to prevent extended downtime.
Enterprise Admin Access: Monitoring Execution & Compute Usage
For Enterprise accounts, Admins have exclusive access to workspace-wide scheduling, compute usage, and execution tracking.
Admin-Level Visibility into Run History & Compute Usage
✔ Workspace-Level Execution Monitoring – Admins can track all replication & transformation jobs under the Usage Summary section.
✔ Compute Consumption Insights – Admins can analyze compute usage trends across teams to ensure efficient allocation of resources.
Top 10 Longest-Running Workflows for Optimization
✔ Admin Dashboard for High-Compute Jobs – Enterprise admins receive a dedicated view of the top 10 longest-running replication and transformation workflows.
✔ Optimization Insights – Helps identify inefficient workflows that may require scheduling adjustments or configuration optimizations.
Automated Admin Notifications for Compute Depletion
Admins receive automated alerts when compute resources approach depletion thresholds:
| Notification Trigger | Alert Type |
|---|---|
| Compute balance ≤ 75% | In-app notification & Slack alert |
| Compute balance ≤ 90% | In-app notification, Slack, and email notification |
| Compute fully depleted (0%) | Immediate in-app alert, Slack, and email |
✔ Ensures uninterrupted workflow execution by allowing Admins to purchase additional compute before depletion impacts scheduled jobs.
Optimizing Workflow Execution with Edilitics
Edilitics’ Scheduler & Advanced Settings provide users with:
✔ Automated execution scheduling to ensure replication and transformation workflows run at optimal times.
✔ Built-in conflict detection to prevent scheduling overlaps and ensure seamless data processing.
✔ Auto-pause safeguards that proactively halt workflows on schema changes and repeated failures, ensuring data consistency.
✔ Enterprise-grade monitoring that gives admins full visibility into compute usage, execution history, and workflow optimization.
By leveraging these advanced controls, organizations can ensure data accuracy, streamline operations, and maintain a reliable, high-performance analytics pipeline. 🚀
Need Assistance? Edilitics Support is Here for You!