Skip to main content Link Search Menu Expand Document (external link)
Azure Data Factory CI/CD tools

Welcome to Azure Data Factory CI/CD tools

Introduction

Hi, and welcome to my Azure Data Factory CI/CD tools Extension documentation page.

This is a tool that I'm maintaining in my personal time, although I would like to thank my employeer twoday Kapacity for allowing me to build this based on knowledge and experience I have gained while working as a Principal Architect there for the past many years.

Why create a deployment tool

Until recently, the best practice from Microsoft to deploy Data Factory resources involved a manual step in the UI, which IMHO is a complete no-go in a CI/CD scenario.
They have since created a new way of doing this, that requires a build step that builds the json files into ARM templates.
I still feel that this process should be simpler and more user friendly.

In twoday Kapacity we have been deploying Data Factory resources for several years through a massive PowerShell script that essentially used the Data Factory REST APIs to deploy everything.

This extension builds upon the experience we have gathered doing this, and simplifies the process a lot also.

Prerequisites

Validation task

There are no prerequisites to start using the validation task.
Just install the extension and setup branch policies with build validation for any relevant branches. Head over to the Validation task documentation for more info
(* please note that this task has not yet been released)

Deployment task - Azure

Before you can start using this task to deploy your Data Factory resources, a few things needs to be in place first:

  • Your 3 (or more environments) with a Data Factory resource in each
  • Any global parameters needed should already have been configured
  • Any KeyVault linked services should have been configured and deployed (the extension will take care of all other Linked Services)
  • If you need to use one (or more) Self Hosted Integration Runtimes, they should already have been configured.

In my opinion, these prerequisites belongs as part of your infrastructure setup (preferably using Infrastructure as Code), and not as part of deploying Data Factory resources.

Deployment task - Azure DevOps

You need to have a service connection configured for each environment, that have at least contributor rights on the Data Factory resource.
In most cases, you should already have one, that have contributor rights to the entire resource group, that you use to deploy other resources such as IaC, Database or the likes.

Install and configure extension

It's as simple as going to Azure Data Factory CI/CD tools on Visual Studio Marketplace and click [Get it free].
You will need to have the right permission setup, in order to do this. Please see the Microsoft documentation for details.

Troubleshooting

If you are having issues using this extension, please first look at the troubleshooting page.
If you do not find the solution there, please feel free to create an issue (see next topic)

Issue tracker on GitHub

You will find the issue tracker here.
Please do a search before creating an issue, to see if it has already been reported.

Feature requests

Feature requests are very welcome.
Please use the issue tracker mentioned in the previous section to report them.


I do hope that you enjoy my work.
Best regards

David