Using DataStar With Octopus

DataStar integrates with Octopus Deploy via packaged database scripts (zip or nupkg), a CLI tool that runs them, and a set of templates the CLI uses to generate reversal scripts. The CI server (Azure DevOps, GitLab, Jenkins, Bitbucket Pipelines, etc.) builds and pushes the package. Octopus pulls it at release time and runs the deployment against the target database.

If the CI is Git-based, GitSources.Tools produces the deployment package directly from a DataStar deployment file.

Dynamic package selection

Octopus projects usually deploy many versions of the same artefact: the package id is stable, only the version changes. DataStar is the opposite. A deployment is scoped to a work item, so a sensible naming convention puts the work-item reference in the package id: DAT-34343.001.nupkg, DAT-34344.001.nupkg, and so on.

Octopus supports this through dynamic package selection: a project variable (typically a regex over the release number) resolves to the real package id at deployment time. The trade-off is that Octopus can't know ahead of time which packages a release will touch, so built-in package-retention policies can't clean up old dynamic packages the way they do static ones. Plan a housekeeping step.

Two projects: releases and reversals

Most teams run two Octopus projects:

Releases. A standard lifecycle, promoting through environments.
Reversals. A flexible lifecycle, so a reversal package can be applied to one specific environment. Only relevant if you use DataStar's reversal mode to generate rollback scripts.

The supporting packages

A release needs three packages at the Tentacle:

The scripts package produced by the CI build.
DataStar.Tools, the CLI that runs the deployment.
DataStar templates, so reversal scripts can be generated.

Installing the CLI directly onto the Tentacle isn't recommended: it creates a maintenance burden, and older releases may depend on older versions. Instead, package DataStar.Tools as a NuGet package, push it to the library feed, and let Octopus pull the correct version at deployment time.

Packaging DataStar.Tools

An Azure DevOps step that packs DataStar.Tools with a custom nuspec:

- task: NuGetCommand@2
  displayName: "nuget pack"
  inputs:
    command: 'custom'
    arguments: 'pack $(build.sourcesdirectory)/src/DataStar.Tools/DataStar.Tools.nuspec -BasePath "$(build.artifactstagingdirectory)/output/src/DataStar.Tools" -Version "1.0.0" -OutputDirectory $(build.artifactstagingdirectory)/nuget -Verbosity Detailed'

The nuspec:

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
  <metadata>
    <id>DataStar.Tools</id>
    <version>$version$</version>
    <description>DataStar Command Line Interface</description>
    <authors>Absolute Technology Limited</authors>
    <title>DataStar.Tools</title>
    <copyright>Copyright Absolute Technology Ltd: All rights reserved</copyright>
    <dependencies></dependencies>
  </metadata>
  <files>
    <file src="**\*.*" target="content"/>
  </files>
</package>

Packaging the templates

Templates define the components DataStar versions, and the reversal feature needs them to generate rollback scripts from the target database immediately before a deployment. Package and push them to the library feed the same way.

- task: NuGetCommand@2
  inputs:
    command: custom
    arguments: pack $(build.sourcesdirectory)/package/templates.nuspec -BasePath "$(build.sourcesdirectory)/templates" -Version $(Build.BuildNumber) -OutputDirectory $(build.artifactstagingdirectory)/nuget -Verbosity Detailed

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
  <metadata>
    <id>DataStar.Templates</id>
    <version>$version$</version>
    <description>DataStar Templates</description>
    <authors>Absolute Technology Limited</authors>
    <title>DataStar.Templates</title>
    <copyright>Copyright Absolute Technology Ltd: All rights reserved</copyright>
    <dependencies></dependencies>
  </metadata>
  <files>
    <file src="**\*.xml" target="templates"/>
    <file src="**\*.sql" target="templates"/>
  </files>
</package>

The step template

The DataStar Release step template wraps the CLI for Octopus. It exposes every flag as a parameter, reads a metadata.json from the deployment package for dynamic variables like #{WorkItem} and #{Version}, and writes the CLI log file as an Octopus artefact.

note

The full step-template JSON is in step-templates/datastar-release.json. Import it into Octopus via Library -> Step templates -> Import.

The shape of the step template, trimmed for illustration:

{
  "Id": "3080d706-1b1d-47d6-bcea-ff59a96a3cd9",
  "Name": "DataStar Release",
  "Description": "Execute the DataStar Release Process",
  "ActionType": "Octopus.Script",
  "Version": 1,
  "Packages": [
    {
      "Name": "DataStarPackage",
      "FeedId": "Feeds-1061",
      "AcquisitionLocation": "Server",
      "Properties": {
        "SelectionMode": "deferred",
        "PackageParameterName": "dsr-datastar-release"
      }
    }
  ],
  "Properties": {
    "Octopus.Action.Script.ScriptSource": "Inline",
    "Octopus.Action.Script.Syntax": "PowerShell",
    "Octopus.Action.Script.ScriptBody": "... invokes DataStar.Tools.exe with parameters ..."
  },
  "Parameters": [
    { "Name": "dsr-database", "Label": "Target database (Oracle or Microsoft)" },
    { "Name": "dsr-licenseKey", "Label": "License Key" },
    { "Name": "dsr-workItem", "Label": "Work Item / User Story" },
    "...and one parameter per DataStar.Tools flag"
  ]
}

Expand the full step-template JSON inline

step-templates/datastar-release.json
{
  "Id": "3080d706-1b1d-47d6-bcea-ff59a96a3cd9",
  "Name": "DataStar Release",
  "Description": "Execute the DataStar Release Process",
  "ActionType": "Octopus.Script",
  "Version": 1,
  "CommunityActionTemplateId": null,
  "Packages": [
    {
      "Id": "0b9459b0-0394-4a45-a0ef-1e1b8b573bb6",
      "Name": "DataStarPackage",
      "PackageId": null,
      "FeedId": "Feeds-1061",
      "AcquisitionLocation": "Server",
      "Properties": {
        "Extract": "True",
        "SelectionMode": "deferred",
        "PackageParameterName": "dsr-datastar-release",
        "Purpose": ""
      }
    }
  ],
  "Properties": {
    "Octopus.Action.Script.ScriptSource": "Inline",
    "Octopus.Action.Script.Syntax": "PowerShell",
    "Octopus.Action.Script.ScriptBody": "function Apply-CmdLineOptions ([System.Diagnostics.ProcessStartInfo]$startInfo, [string]$switchName, [string]$variableName, $variableValue)\n{\n    if (![string]::IsNullOrEmpty($variableValue)) \n\t{\n    \tif ($variableValue -eq $true)\n        {\t\n        \tWrite-Host \"Parameter [$variableName]: $variableValue\"\n        \t$startInfo.Arguments = $startInfo.Arguments + \" $switchName\"\n        }\n    \telseif ($variableValue -ne $false)\n    \t{\n        \tif($variablesTable.ContainsKey($variableValue))\n            {\n            \t$variableValue = $variablesTable[$variableValue]\n            }\n            Write-Host \"Parameter [$variableName]: $variableValue\"\n\t\t\t$startInfo.Arguments = $startInfo.Arguments + \" $switchName \"\"$variableValue\"\"\"\n    \t} \n    }\n}\n\n$failOnStderr = $true\n$WorkingDirectory = $OctopusParameters[\"dsr-workingDirectory\"]\n\n$variablesTable = @{}\n\n$variablesFile = $OctopusParameters[\"dsr-variables-file\"]\nif (![string]::IsNullOrEmpty($variablesFile)) \n{\n\t$JsonFileLocation = Join-Path -Path $WorkingDirectory -ChildPath $variablesFile\n\t$json = Get-Content $jsonFileLocation -Raw | ConvertFrom-Json\n\t[string[]]$variables = (($json | get-member -Name * -MemberType NoteProperty).Name)\n\tforeach ($v in $variables)\n\t{\n    \t$value = ($json.$v).Trim()\n    \tWrite-Host \"Setting up variable: $v : $value\"\n  \t\tSet-Variable -name $v -value $value -Force\n        Set-OctopusVariable -name $v -value ($json.$v)\n        $variablesTable.Add(\"#{$v}\", $value)\n\t}\n}\n\n$BasePath = $OctopusParameters[\"dsr-path\"]\nWrite-Host \"DataStar Release Base Path: $BasePath\"\n\n... (truncated; see step-templates/datastar-release.json for the full script body) ...\n"
  },
  "Parameters": [
    "... one entry per DataStar.Tools parameter; see the linked file ..."
  ],
  "StepPackageId": "Octopus.Script",
  "$Meta": {
    "ExportedAt": "2022-02-23T22:00:22.239Z",
    "OctopusVersion": "2022.1.552",
    "Type": "ActionTemplate"
  },
  "LastModifiedBy": "Your GitHub Username",
  "Category": "other"
}

Next steps

Create Release: how to build Octopus releases that use dynamic package selection.
Deploy Release: a worked Octopus pipeline that deploys a DataStar package.
Apply Reversal: the pipeline for applying a generated reversal package.

Dynamic package selection​

Two projects: releases and reversals​

The supporting packages​

Packaging DataStar.Tools​

Packaging the templates​

The step template​

Next steps​