Azure Data Factory cloud ETL — canonical Git-published artifact layout

Context

Azure Data Factory pipelines authored only in the GUI carry a hidden cost: the workspace becomes the source of truth, and code review, rollback, and multi-environment promotion all become harder than they need to be. The discipline of authoring against the canonical Git-published artifact layout solves it; the project is the proof of that discipline applied end-to-end.

Problem

“It works in the GUI” and “we can roll this back if it breaks” are not the same statement. Without a Git-tracked artifact layout, pipeline changes have no diff, no review, and no clean rollback path. ADF emits the canonical layout when source-control configuration is enabled, but only if practitioners author against it consistently.

Approach

Author every ADF resource as a version-controlled JSON artifact in its canonical subfolder: Linked Service, Dataset, Pipeline, Trigger each in their own top-level directory, plus the publish_config.json that wires the workspace to the publish/collaboration branch contract. Treat the layout as the authoritative state of the pipeline; the GUI is a renderer.

Solution

• ADF artifacts authored in the canonical Git-published layout: dataset/, factory/, linkedService/, pipeline/, trigger/ subfolders.
• publish_config.json configured for the publish/collaboration branch contract, so the workspace stays in lockstep with Git.
• Every ADF resource as a one-JSON-per-resource definition: code-reviewable, branchable, promotable through standard Git tooling.
• Linked services, datasets, pipelines, and triggers all version-controlled rather than GUI-only.

Outcome

Above. The artifact-layout discipline is the load-bearing engineering outcome: it is what separates a team that can roll back a broken pipeline from one that cannot. Operational specifics (pipeline counts, source/sink systems, schedule cadence) live in the private repo and are shared in client conversations rather than on the public surface.