DQ Refresh & Schema Drift
How Edilitics keeps DQ and AIR scores current, detects schema changes, and protects validated descriptions from being overwritten.
Your DQ Score and AIR Score reflect the state of your data at a point in time. As your data grows and your schema evolves, those scores need to stay current. DQ Refresh re-profiles your connected source, recalculates every score, and detects any columns that have been added or removed since the last run.
What a Refresh Does
Every refresh run, regardless of what triggered it, does three things in sequence:
Recalculates every score. Edilitics queries exact table population metrics (nulls, distincts, min, max) via engine pushdown and draws a fresh 16,600-row sample for compliance checks, recomputes all per-column statistics, and updates every Column DQ Score, Table DQ Score, Integration DQ Score, and AIR Score. The scores on your integration card update automatically when the run completes.
Checks for schema changes. The refresh compares the current structure of your source against the stored record. If columns have been added or removed since the last run, the stored schema is updated to match. See Schema Drift below.
Protects validated descriptions, with one exception. If AI Column Insights are enabled on the integration, a refresh only regenerates descriptions when a table has gained a new column or has no AI metadata at all. When that happens, every unvalidated description in that same table is resent to the AI and can be overwritten, not just the new column, because generation runs once per table rather than once per column. A description that has been approved or edited by a person is never touched, regardless of what else updates in the same table.
Refreshing costs resources every time it runs, whether or not any description is regenerated. The DQ profiling and scoring pass is billed as compute on every refresh. AI credits are only billed on top of that when the table-level condition above is met and the AI is actually called.
When Refresh Runs
A full profile runs automatically the moment a new integration is saved. No action required. This is always the first run and establishes the baseline DQ and AIR scores shown on the integration card.
Editing an integration triggers a re-profile only if the connection identity has changed: the hostname, database name, or port. Renaming the integration, updating its description, or changing privacy mode settings does not trigger a re-profile. A re-profile also runs if the integration previously had no DQ score, or if AI Column Insights are being enabled for the first time on an existing integration.
Every connection form except file uploads and Redis includes an Auto schedule setting with four options: Auto (the default 7-day/5%-growth rule described under Daily Background), 12 hours, 24 hours, or Manual. Setting a connection to 12 or 24 hours profiles it on that fixed interval instead of waiting for the Auto rule's conditions to be met.
This setting is available at connection time and can be changed later by editing the integration.
Every time a Transform flow runs successfully, DQ scores are recalculated for the destination table and the integration's overall DQ and AIR scores update automatically. This happens on every run - the first run establishes the baseline, and every subsequent run keeps the scores current as data changes.
Column descriptions are not regenerated during a flow run. If new columns appear in the destination table, use Generate AI Insights from the integration hover menu to produce descriptions for them.
A daily background job checks integrations set to the Auto schedule and targets those that meet both of the following conditions:
- Not profiled in the last 7 or more days
- Underlying data has grown by more than 5% since the last profiling run
If the data volume has not grown by more than 5%, the integration is skipped even if it is stale. Schema changes alone do not trigger the background refresh. Integrations set to a 12-hour or 24-hour schedule instead profile on that fixed interval, regardless of these two conditions.
File-based sources are excluded. CSV, Excel, PDF, JSON, Parquet, Avro, Feather, Pickle, and similar file integrations are not targeted by the daily background job. Their scores update on connection or when the integration is edited.
Use Refresh Schema & Insights in the integration hover menu to trigger a refresh immediately. This option is available to the integration owner and workspace admins. Shared users cannot trigger a manual refresh.
Manual refresh is rate-limited to once per 24 hours per integration. If you attempt a refresh within 24 hours of the last run, the request is rejected and a message explains when the next refresh will be available.
What Happens While a Refresh Runs
When a manual or background refresh is running, the integration card shows a blue Refreshing pulse badge. All card actions are disabled during this time. When the refresh completes, the badge clears and the scores on the card update automatically. You receive a notification when it is done.
Schema Drift
Every refresh re-examines the structure of your connected source and compares it to what Edilitics has stored. This comparison detects three types of change:
New columns. If your source has gained columns since the last refresh, they are added to the stored schema and appear in View Tables with their DQ metrics. If AI Column Insights are enabled on the integration, descriptions are automatically generated for the new columns as part of the refresh, and any other unvalidated description already in that table is regenerated alongside them. If AI Column Insights are not enabled, new columns appear with no description until you enable them.
Removed columns. If columns that were previously present are no longer in the source, the change is detected. The DQ scores for the integration are recalculated to reflect the current state of the source. Descriptions for columns that no longer exist are retained in the stored record but are no longer active.
Type changes. If a column's data type has changed in the source, the change is detected. The column's DQ metrics are recalculated against the new type. Existing descriptions for that column are not automatically updated: if the type change affects what the column represents, review and update the description manually in the Metadata Viewer.
Schema drift is detected during every refresh run, regardless of what triggered it. You do not need to trigger a manual refresh specifically to detect schema changes.
How Descriptions Are Protected
Not all descriptions are treated equally during a refresh. The rule is straightforward:
- Human-validated descriptions - columns where a person has approved or edited the description - are never touched by any refresh. Their descriptions, validation status, and AIR Score contribution are unchanged regardless of what else updates.
- AI-generated descriptions that have not been validated - columns where the description was written by the AI and has not yet been reviewed - are only regenerated when the table they belong to gains a new column or has no AI metadata yet. A table where nothing has changed since the last run keeps its existing AI-generated descriptions exactly as they are, even if none of them have been validated.
This means you never lose curation work you have already done. The only descriptions that can change are the ones your team has not yet confirmed, and only when something in that specific table has actually changed.
New columns added by schema drift have no descriptions and are not validated. They contribute 0 to the AIR Score until descriptions are generated and validated. If AI Column Insights are already enabled on the integration, descriptions are generated automatically during the refresh. If not, use Generate AI Insights from the integration hover menu to document them.
Frequently Asked Questions
Next Steps
Data Quality Profiling (DQ)
How DQ scores are calculated and what each dimension measures.
AI Readiness (AIR) Score
How the AIR Score combines DQ with schema documentation, and how refresh affects it.
AI Column Insights
How to generate and validate column descriptions, including for new columns detected by schema drift.
Need help? Email support@edilitics.com with your workspace, job ID, and context. We reply within one business day.
Last updated on
AI Readiness (AIR)
What the AI Readiness Score measures, how to read and act on it in the Metadata Viewer, and how AIR affects AskEdi and Auto Generate Charts.
Sharing & Access Control
How to share an integration with workspace members and what happens when you revoke access from a user who has built work on top of it.