DQ Refresh & Schema Drift
How Edilitics keeps DQ and AIR scores current, detects schema changes, and protects validated descriptions from being overwritten.
Your DQ Score and AIR Score reflect the state of your data at a point in time. As your data grows and your schema evolves, those scores need to stay current. DQ Refresh re-profiles your connected source, recalculates every score, and detects any columns that have been added or removed since the last run.
What a Refresh Does
Every refresh run, regardless of what triggered it, does three things in sequence:
Recalculates every score. Edilitics draws a fresh sample from your source, recomputes all per-column statistics (null count, distinct count, noncompliant count, min, max, and top values), and updates every Column DQ Score, Table DQ Score, Integration DQ Score, and AIR Score. The scores on your integration card update automatically when the run completes.
Checks for schema changes. The refresh compares the current structure of your source against the stored record. If columns have been added or removed since the last run, the stored schema is updated to match. See Schema Drift below.
Protects validated descriptions. If AI Column Insights are enabled on the integration, any column description that has been approved or edited by a person is never touched by a refresh. Only AI-generated descriptions that have not yet been validated are eligible to be updated during a refresh run.
When Refresh Runs
A full profile runs automatically the moment a new integration is saved. No action required. This is always the first run and establishes the baseline DQ and AIR scores shown on the integration card.
Editing an integration triggers a re-profile only if the connection identity has changed: the hostname, database name, or port. Renaming the integration, updating its description, or changing privacy mode settings does not trigger a re-profile. A re-profile also runs if the integration previously had no DQ score, or if AI Column Insights are being enabled for the first time on an existing integration.
Every time a Transform flow runs successfully, DQ scores are recalculated for the destination table and the integration's overall DQ and AIR scores update automatically. This happens on every run — the first run establishes the baseline, and every subsequent run keeps the scores current as data changes.
Column descriptions are not regenerated during a flow run. If new columns appear in the destination table, use Generate AI Insights from the integration hover menu to produce descriptions for them.
A daily background job checks all live database integrations and targets those that meet both of the following conditions:
- Not profiled in the last 7 or more days
- Underlying data has grown by more than 5% since the last profiling run
If the data volume has not grown by more than 5%, the integration is skipped even if it is stale. Schema changes alone do not trigger the background refresh.
File-based sources are excluded. CSV, Excel, PDF, JSON, Parquet, Avro, Feather, Pickle, and similar file integrations are not targeted by the daily background job. Their scores update on connection or when the integration is edited.
Use Refresh DQ & AIR Scores in the integration hover menu to trigger a refresh immediately. This option is available to the integration owner and workspace admins. Shared users cannot trigger a manual refresh.
Manual refresh is rate-limited to once per 24 hours per integration. If you attempt a refresh within 24 hours of the last run, the request is rejected and a message explains when the next refresh will be available.
What Happens While a Refresh Runs
When a manual or background refresh is running, the integration card shows a blue Refreshing pulse badge. All card actions are disabled during this time. When the refresh completes, the badge clears and the scores on the card update automatically. You receive a notification when it is done.
Schema Drift
Every refresh re-examines the structure of your connected source and compares it to what Edilitics has stored. This comparison detects three types of change:
New columns. If your source has gained columns since the last refresh, they are added to the stored schema and appear in View Tables with their DQ metrics. If AI Column Insights are enabled on the integration, descriptions are automatically generated for new columns as part of the refresh. If AI Column Insights are not enabled, new columns appear with no description until you enable them.
Removed columns. If columns that were previously present are no longer in the source, the change is detected. The DQ scores for the integration are recalculated to reflect the current state of the source. Descriptions for columns that no longer exist are retained in the stored record but are no longer active.
Type changes. If a column's data type has changed in the source, the change is detected. The column's DQ metrics are recalculated against the new type. Existing descriptions for that column are not automatically updated: if the type change affects what the column represents, review and update the description manually in the Metadata Viewer.
Schema drift is detected during every refresh run, regardless of what triggered it. You do not need to trigger a manual refresh specifically to detect schema changes.
How Descriptions Are Protected
Not all descriptions are treated equally during a refresh. The rule is straightforward:
- Human-validated descriptions — columns where a person has approved or edited the description — are never touched by any refresh. Their descriptions, validation status, and AIR Score contribution are unchanged regardless of what else updates.
- AI-generated descriptions that have not been validated — columns where the description was written by the AI and has not yet been reviewed — are eligible to be updated when a refresh runs and AI Column Insights are enabled.
This means you never lose curation work you have already done. The only descriptions that can change are the ones your team has not yet confirmed.
New columns added by schema drift have no descriptions and are not validated. They contribute 0 to the AIR Score until descriptions are generated and validated. If AI Column Insights are already enabled on the integration, descriptions are generated automatically during the refresh. If not, use Generate AI Insights from the integration hover menu to document them.
Frequently Asked Questions
Next Steps
Data Quality Profiling (DQ)
How DQ scores are calculated and what each dimension measures.
AI Readiness (AIR) Score
How the AIR Score combines DQ with schema documentation, and how refresh affects it.
AI Column Insights
How to generate and validate column descriptions, including for new columns detected by schema drift.
Need help? Email support@edilitics.com with your workspace, job ID, and context. We reply within one business day.
Last updated on
AI Readiness (AIR)
What the AI Readiness Score measures, how to read and act on it in the Metadata Viewer, and how AIR affects AskEdi and Auto Generate Charts.
Sharing & Access Control
How to share an integration with workspace members and what happens when you revoke access from a user who has built work on top of it.