Seamless Data Replication with Edilitics | Features & Steps#

Edilitics is a sophisticated data analytics platform designed to empower enterprises with precise and efficient data replication capabilities. Our Replicate module ensures that your datasets remain synchronized and up-to-date, whether through full loads, incremental updates, or Change Data Capture (CDC) mechanisms. Leveraging our intuitive interface and advanced replication features, you can effortlessly synchronize data across diverse sources and destinations, streamlining data governance and enhancing operational efficiency.

Implementing Data Replication in Edilitics#

Steps to Configure Data Replication#

1. Access the Replicate Module#

• Navigate to the "Replication" section within the Edilitics platform's sidebar.

2. Initiate a New Replication Workflow#

• Click the “New Replication” button to begin the configuration of your data replication workflow.

3. Select Source and Destination Integrations#

• Identify your data source from the available integrations, then select the destination where the data will be replicated.

Note: Source and destination integrations must be distinct. Only integrations that are compatible for use as destinations will be displayed.

4. Specify Load Type#

• Choose the appropriate load type: Full Load, Incremental Load, or Change Data Capture (CDC).

Note: The available load types will be dynamically adjusted based on the database type of the selected source.

5. Load Historical Data (If Applicable)#

• For Incremental or CDC load types, you may opt to load Historical Data to ensure comprehensive data capture.

6. Leverage Auto-Schema Mapping#

• Enable the Auto-Schema Mapper to automatically map data types from the source tables to those supported by the destination database.

  • If this option is disabled, manual mapping of data types for each table will be required.

7. Select Tables for Replication#

• Select the table(s) you wish to replicate from the source to the destination, then proceed by clicking "Continue."

8. Configure Destination Settings#

Refine the destination settings, including table names, column mappings, and data type configurations, to align with your replication strategy.

• Destination Table Names: Customize destination table names as needed. If a table with the same name already exists in the destination, Edilitics will suggest alternative names (e.g., table_name_v1, table_name_v2).

• Column to Increment On: For Incremental Load replications, specify the column used to track incremental changes.

Note: This column must be a Primary Key or Datetime column. Tables lacking such columns cannot be included in an Incremental Load replication.

• Map Data Types: If Auto-Schema Mapper is disabled, manually map the source data types to those supported by the destination using the provided dropdown menu.

• Reorder Columns: Adjust the column order to meet your specific data architecture requirements.

• Drop Columns: Exclude any columns that are not needed in the destination database.

Note: Columns utilized for tracking incremental loads cannot be dropped.

9. Define Replication Title and Description#

• Assign a unique and descriptive title to your replication workflow to ensure clarity within your organization. Additionally, provide a concise description that encapsulates the nature and purpose of the replicated data.

10. Schedule Replication Runs#

Configure the scheduling for your replication workflow:

• Once: The replication will run a single time, with manual triggering available if necessary.

• Daily: Automate the replication to run at a specific time each day.

• Custom Cron: Use custom cron expressions for fine-grained control over the replication schedule.

Example: A cron expression of "0 0 * * 1,4" will schedule the replication to run at midnight on Mondays and Thursdays.

11. Configure Advanced Settings#

• Table Availability Check: Enable this feature to automatically pause the replication if any tables are unavailable, optimizing compute resource usage.

• Pause & Notify on Schema Change: Configure this setting to pause the replication upon detecting schema changes in the source table. If disabled, schema changes will be automatically replicated.

• Pause After Failed Runs: Specify the threshold for consecutive failed runs that will trigger a pause in the workflow, minimizing resource wastage and prompting notifications for user intervention.

12. Save the Replication Setup#

• Finalize the configuration by saving the replication workflow.

Monitoring and Managing Data Replication#

• Edilitics offers robust tools to monitor and manage your data replication workflows, ensuring comprehensive oversight and control:

Detailed Run History#

• Access a thorough log of all replication activities, including success rates, failures, execution times, and user interactions (e.g., pause, resume, update, trigger), all with precise timestamps. Additionally, table-level details offer granular insights into data processing stages, facilitating efficient diagnostics and troubleshooting.

View Configuration#

• Maintain a consolidated overview of all tables involved in a replication workflow, including their configurations (schedule, load type, incremental update columns, and alerts). This centralized interface streamlines workflow management and supports strategic decision-making.

On-Demand Triggering#

• Address urgent data analysis needs by initiating a replication run outside of its scheduled time. Upon completion, the workflow automatically resumes its predefined schedule, ensuring flexibility without disrupting ongoing processes.

Pause/Resume Control#

• Exercise full control over your data replication workflows by pausing or resuming them as needed. This feature is particularly valuable for conserving resources or avoiding errors during scheduled maintenance periods for the source database.

Granular Sharing with Team Members#

• Enhance collaboration by sharing replication workflows with other workspace members. Shared users can monitor runs, view configurations, and trigger, pause, or resume workflows as necessary. Importantly, data security and ownership are preserved, as shared members cannot edit, delete, or further distribute the workflow.

Managing Replication Workflows Throughout Their Lifecycle#

Edilitics empowers you to efficiently manage replication workflows throughout their lifecycle:

Update Replication Titles and Descriptions#

• Maintain clarity and streamline workflow identification within your workspace by updating titles and descriptions directly from the Home Screen.

Modify Schedules#

• Easily adjust the schedule of your replication workflow from the Home Screen, enabling quick and efficient management of replication timing.