Checking for changes

Check for changes tells you which components in your workspace no longer match a database, without extracting anything. Connect to DEV, UAT or production, check, and the Changes column says, component by component, whether the database still holds what your workspace holds.

It works from a checksum. When a component is extracted, the database computes a checksum of its rows and DataStar records it in the component's data document. A check asks the connected database for the same checksum and compares the two. Only the checksum crosses the network, so checking thousands of components takes seconds.

Turning it on

Checking for changes is per template. Set TrackChanges="true" on a data template that stores its components as documents:

<DataComponent Category="currency-lists" Name="${filename}"
               OutputFormat="json" TrackChanges="true">

Then extract the template's components once, from the database you treat as the source, and commit the documents. Each document now carries a checksum block (see Tracking changes), and from then on any database of the same vendor can be checked against it.

TrackChanges needs OutputFormat="json", and SQL Server 2016 or Oracle 12c or later. A database older than that still extracts, without a checksum, and DataStar warns that the component cannot be checked.

Checking a tab

On a category tab whose template tracks changes, the toolbar shows Changes, after Pull to Draft. On other tabs it is hidden.

• With rows selected, Changes checks those rows afresh, whatever they showed before.
• With nothing selected, it checks every row not yet checked. Once every row has a verdict, it checks them all again.

Right-click and choose Check for Changes to do the same from the context menu.

A strip above the grid shows the progress, with Cancel. A check you cancel is not lost: the rows already checked keep their verdicts, and the next check carries on with the rest. A check of rows that already have verdicts keeps each one on show until its new verdict arrives, so the column never goes blank while you wait.

Only components that are both in the workspace and in the database are checked. A component the database does not have shows NOT FOUND in the Database column, as before.

What the column says

Changes	Meaning	What to do
UP TO DATE	The database holds exactly the rows the document holds.	Nothing.
CHANGED	The database's rows differ from the document's. Someone changed the data there, or a deployment has not reached it yet.	Compare With → Compare with Current Connection shows the difference. DB Pull takes the database's version.
RE-EXTRACT	Nothing was compared, because the document cannot vouch for itself: it has no checksum, its rows have changed since its checksum was recorded (a merge, a clone or a hand edit), or its template has changed shape since. DataStar decides this on its own, without asking the database.	If the document's rows are what you want, Update Checksum records theirs and keeps them. If you want the database's rows, extract the component. If the template changed, extract it.
NOT CHECKABLE	The database could not compute the checksum, or is of another vendor than the document was extracted from. Hover the badge for the reason.	Usually nothing to do; the reason says.

Hover a badge for its reason, for example Rows differ or The template changed since the document was extracted. The Changes column has a filter like the others, so you can show only the CHANGED rows.

Extracting a component that showed CHANGED or RE-EXTRACT marks it UP TO DATE, since its document now holds that database's rows. One that showed NOT CHECKABLE loses that verdict, which was about the document the extract replaced.

Update Checksum

A document's checksum belongs to the rows it was recorded for. When something other than an extract changes those rows (a merge brings in a colleague's, you edit the file, an older DataStar wrote it without one), the document reads RE-EXTRACT. Extracting fixes that by replacing the rows with the database's, which is not always what you want: on a branch, the document's rows are the ones you mean to deploy, and for a template that deploys changesets an extract deletes the work item's changeset.

Update Checksum keeps the rows. Right-click the components and choose Update Checksum, or choose it with nothing selected to update every row that reads RE-EXTRACT. The connected database computes the checksum of the rows each document holds, running only what a check of the template runs (its build-time SetUps among them), and DataStar writes it into the document. The rows stay exactly as they are. The file is written the way the template writes documents now, so if the template's layout has changed since (from inline to shared definitions, say), the file follows it as an extract would. Its progress shows where a check's does, and Cancel stops before the next component, keeping those already done. The components are then checked again, and read UP TO DATE or CHANGED against that database like any other. It is not offered while a check is running, and Changes waits until it has finished. It works one document at a time (on Oracle, 500 rows to a statement), so updating thousands of large documents takes a while: select the ones you need rather than every RE-EXTRACT row.

Any database of the same vendor gives the same checksum, so it does not matter which environment you are connected to. A document whose template has changed since it was extracted is not updated, even when the checksum it holds is its rows': its rows are another query's, so the message says to extract it instead. A document whose checksum is already its rows' is left as it is.

A few Oracle values cannot be hashed from a document, because the document does not hold all of them or one statement cannot carry them: a TIMESTAMP WITH TIME ZONE value (the document keeps the time but not its zone; a column holding only NULLs is fine), a TIMESTAMP(8) or TIMESTAMP(9) value (the document keeps seven fractional digits), a BLOB of more than 2,000 bytes, and text of more than 4,000 bytes (2,000 characters for NVARCHAR2 and NCHAR), which a database with extended string sizes can hold. A component with one of those keeps its RE-EXTRACT, and the message says why.

Changesets keep their checksum

Pull My Changes, Apply Draft and Delete Changeset rewrite a component's document without extracting it, so they record its checksum as they write it:

• when you take every row a pull offers, or a delete puts the file back as it was committed, the document usually holds exactly the rows another document recorded a checksum for, and then that checksum is copied across;
• otherwise the database the rows came from (or, for a branch or a draft, the connected database) computes the checksum of the rows the document now holds.

Either way the document reads UP TO DATE or CHANGED at the next check rather than RE-EXTRACT. The exception is a template changed since the workspace file was extracted: its rows are another query's, so no checksum is recorded, and only an extract settles it. Against the environment you pulled from it usually reads CHANGED: you took your rows, not everyone's. Against an environment the work item has been deployed to, it reads UP TO DATE. If no checksum could be recorded, for example because no connection was open, the pull's results say so, and Update Checksum records one later. Cancel on the overlay while the checksum is computed stops only that: the rows you ticked are still written, without a checksum.

What a check compares

The checksum covers what a deployment of the document takes from its rows. So a document compares with every environment it has been deployed to, whatever ids and audit values each environment holds of its own. It leaves out:

• keys the target generates, and columns set at deploy time (CurrentDate, a %GETDATE()% value, Increment);
• columns the template excludes (Include="false");
• constants (Value="N"), which the script writes in place of the row's value;
• columns a deployment leaves alone on a row that already exists (UpdateExclude="true"), such as who created it;
• a lookup column's id, wherever its lookup finds the row it points to. A deployment sets that id from the target's own lookup table, so the lookup's criteria, such as a view's GUID, are compared instead.

Leaving these out costs nothing: none of them is something a deployment would change.

Partly checked components

Some columns the checksum cannot cover:

• XML columns with lookups, whose stored ids differ by environment;
• on SQL Server, sql_variant columns.

On Oracle, a large object is compared by its first part and its length.

A component with such a column reads as partly checked, and its badge's tooltip says so. UP TO DATE then means every compared column matches, not that every byte does.

The Changes tab

Workspace → Changes opens one tab for every component, across every category, whose template tracks changes. Use it to check a whole workspace against an environment in one go.

• The template picker at the left chooses which templates take part (all of them, to begin with).
• Check checks the listed rows against the connected database, with progress and Cancel. Opening the tab lists the rows and finds which the database has, but checks nothing until you press Check.
• Only changes (on by default) shows only the rows that need your attention: changed, to re-extract, not checkable, or not in the database. Rows that are up to date, or not checked yet, are hidden.
• Only basket lists only the components in your basket.
• The summary at the right counts what the check found: changed, to re-extract, not in the database, not checkable, up to date and not yet checked.

The rows are ordinary component rows. DB Pull, Pull to Draft, Compare With, Add to Basket and the rest work on them as on a category tab.

Verdicts and connections

A verdict belongs to the database it was reached against. When you switch connection, the Changes column shows the verdicts you reached against the new one, or nothing if you have not checked it. Switch back, and the earlier verdicts return, with the time they were reached (as checked at 09:42).

Verdicts last while DataStar runs. They are not saved, so after a restart every row starts unchecked: a verdict from yesterday says nothing about the database today.

A change to a component's document, such as an edit, a pull or a merge, drops its verdicts against every connection, even one made while no tab listed the component.

Files under an older naming

DataStar names each component's file from the component's name, using the workspace's file naming rules (Workspace Settings, the Formatting tab: File Name Formatting and Search & Replace Rules). If those rules change, say to turn * into _, a file written before the change is still on disk under its old name.

When DataStar next opens the workspace, it notices the change and remembers each component's old file name. A file under a name it remembers is never shown as a component of its own. When a tab finds one, its banner says so and offers to tidy it:

• A file that holds what its component's current file holds is deleted.
• A file that is its component's only file is renamed to the name the rules give it now.
• A file that holds other rows than its component's current file is left alone, for you to compare with that file and delete.

The button says what it does (Delete the file, Rename the 2 files, Tidy the 3 files) and asks first, listing every file and what happens to it. Each file is looked at again when you confirm, so one that has changed since the banner appeared is kept. A deleted file can be recovered from version control if it was committed.

A copy or a move you make yourself is not an old file: it is listed under its own name, as always, even while the name inside it is still the component it was copied from.

From an agent or a pipeline

• An MCP agent can ask the same question with check_for_changes, for example "which components differ from PRD?". Its verdicts appear in the Changes column and the Changes tab as it runs, just as if you had pressed Check.
• A pipeline can run datastar --command-name changes to report drift, fail a build on it, re-extract what changed, or with --update-checksums record the checksum of documents a merge changed.

How hard it works the database

Components are checked 25 to a statement, several statements at a time. Every check DataStar runs, from any tab or agent, shares one limit on how many statements it has at a database at once: the MaxDegreeOfParallelism setting in DataStar's appsettings.json (5 as shipped), which also limits extraction. A tab and an agent checking at the same time cannot flood the server between them.

A statement the database cannot run is asked again one component at a time, so one component's problem does not hide the others' verdicts. When the components fail just as the statement did, the problem is the template's (a column the database lacks, a permission it denies): the rest of the template's components read NOT CHECKABLE with the reason, without being asked.

Against SQL Server LocalDB, a thousand components check in about a second.

Troubleshooting

You see	It means	What to do
No Changes button on the toolbar	No template of the tab's category sets TrackChanges="true".	Set it on the template and extract the components once.
Every row reads RE-EXTRACT	The documents were extracted before the template tracked changes, or the template has changed since.	Extract the components once more.
A component reads RE-EXTRACT after a merge	The merge changed the document's rows, so its checksum is not theirs.	Update Checksum keeps the merged rows and records their checksum.
NOT CHECKABLE, is of another vendor	A document extracted from Oracle checked against SQL Server, or the other way round.	Check against a database of the vendor the document came from.
Rows read UP TO DATE although you know a value differs	The value is in a column the checksum leaves out. The badge's tooltip says the component was partly checked.	Use Compare With for a full comparison.
A banner about files written before the file naming rules changed	Files under an older naming are on disk beside, or instead of, their components' current files.	Use the banner's button to tidy them, then commit. A file the banner leaves alone holds other rows: compare it with the current file and delete the one you do not want.