Understanding Load Types in Edilitics#

Efficient data replication requires selecting the right load type based on data volume, frequency of updates, system performance, and data consistency requirements.

Edilitics supports four distinct load types, each designed for specific replication scenarios:

✔ Full Load – Transfers the entire dataset in each replication cycle.

✔ Incremental Load – Captures only new or modified records since the last replication.

✔ Change Data Capture (CDC) – Tracks and replicates insertions, updates, and deletions using database logs.

✔ Historical Load – Loads historical data before initiating Incremental or CDC flows.

This guide explains the functionality, use cases, and performance implications of each load type, helping users choose the most efficient strategy for their workflows.

Full Load: Complete Data Transfer#

How It Works#

✔ Overwrites the destination database with the entire dataset during each replication cycle.

✔ Does not track previous loads—every run transfers all source data from scratch.

When to Use#

✅ Initial Data Migration – First-time setup before transitioning to Incremental or CDC.

✅ Dataset Refresh – When a complete overwrite is required.

✅ Small to Moderate Datasets – Suitable when full replication does not cause performance issues.

Performance Implications#

❌ High Resource Utilization – Requires substantial CPU, memory, and network bandwidth, especially for large datasets.

❌ Extended Processing Time – Transfer time scales proportionally with dataset size.

❌ System Impact – Can affect performance of source and destination databases if executed frequently.

💡 Best Practice: Use Full Load only for initial migrations or when an entire dataset refresh is unavoidable.

Incremental Load: Optimized Data Updates#

How It Works#

✔ Transfers only new or modified records since the last replication.

✔ Requires a Primary Key or Datetime/Timestamp column to track changes.

✔ Edilitics automatically identifies tables that support Incremental Load and segregates them in the UI.

When to Use#

✅ Frequent Updates – Ensures datasets remain current without redundant transfers.

✅ Data Warehousing – Reduces replication workload for analytical processing.

✅ Large Datasets – More efficient than Full Load for ongoing updates.

Performance Implications#

✔ Lower Resource Consumption – Only a subset of data is transferred, minimizing CPU, memory, and network usage.

✔ Faster Processing – Significantly shorter execution time compared to Full Load.

✔ Minimal System Impact – Reduces load on source & destination databases, preventing performance degradation.

💡 Best Practice: Use Incremental Load whenever possible to improve replication efficiency and system stability.

Change Data Capture (CDC): Real-Time Data Synchronization#

How It Works#

✔ Captures insertions, updates, and deletions from database logs, ensuring only actual changes are replicated.

✔ Supported for:

• MongoDB, MySQL, PostgreSQL (including cloud versions).

✔ Edilitics automatically verifies log access during flow setup and notifies users if CDC is feasible.

When to Use#

✅ High-Volume Data Environments – Prevents unnecessary replication of static records.

✅ Real-Time Analytics – Provides near-instant updates for dashboards and operational monitoring.

✅ Compliance & Auditing – Tracks historical changes for regulatory reporting.

Performance Implications#

✔ Highly Resource-Efficient – Only modified data is replicated, reducing workload.

✔ Minimal Processing Time – Updates flow with minimal latency.

✔ Negligible System Impact – Utilizes database logs instead of full-table queries, improving performance.

💡 Best Practice: Use CDC when log-based tracking is available for maximum replication efficiency.

Historical Load: Backfilling Data for Incremental & CDC Flows#

How It Works#

✔ Transfers all existing records before the first Incremental or CDC run.

✔ Executes only once before resuming scheduled updates.

✔ Pauses all scheduled runs until the Historical Load completes.

When to Use#

✅ Data Backfilling – Populates historical records before enabling Incremental or CDC updates.

✅ Archival Migrations – Moves entire datasets for historical reference.

✅ Workflow Initialization – Ensures past data is available before real-time updates begin.

Performance Implications#

❌ Resource-Intensive – Transfers large volumes of data in a single run.

❌ Long Processing Time – Duration depends on dataset size and system capacity.

❌ Temporary System Impact – Can increase database load, but affects only the initial replication.

💡 Best Practice: Schedule Historical Loads during off-peak hours to prevent performance bottlenecks.

Choosing the Right Load Strategy#

Selecting the optimal load type depends on data volume, frequency of updates, system performance considerations, and consistency requirements.

💡 Best Practice: Choose Incremental Load or CDC for routine updates, and reserve Full Load or Historical Load for initial migrations.

Optimizing Replication Performance with Edilitics#

Edilitics’ flexible load strategies empower users to:

✔ Maximize efficiency – Replicate data without unnecessary resource consumption.

✔ Ensure data consistency – Select the right load type to match update frequency & accuracy needs.

✔ Scale with enterprise demands – Handle large datasets & real-time changes without performance trade-offs.

By leveraging incremental, CDC, and historical load types strategically, organizations can enhance replication workflows, maintain high availability, and optimize data pipeline performance. 🚀