Declarative Data Orchestration

I implemented the TaskSets contract, client, and runtime compiler, including reference and cycle validation before Airflow dependencies are generated. The compiler can express explicit fan-out and fan-in while reusing the existing operator-building path.

The platform serves engineering teams through shared production workflows. That platform context is separate from personal ownership; my contribution is the opt-in graph path and its compatibility with existing serial workflows.

A TaskSets definition passes through the client to the runtime compiler. The compiler validates references and set-level cycles, builds a graph of serial or parallel stages, and only then wires the corresponding Airflow dependencies.

A pipeline that does not declare TaskSets still returns to the established serial path. That two-path behavior lets a team adopt the graph where it helps without altering a stable workflow that has no parallel requirement.

I chose an opt-in contract instead of converting every generic pipeline to a parallel abstraction. A team declares a TaskSet when its work has real branches or joins; otherwise, the runtime keeps the serial chain it already understands.

Validation happens before dependency wiring so an unknown reference or set-level cycle fails as a graph problem rather than a half-running DAG. The cost is maintaining two code paths and a richer contract, but it gives teams a gradual adoption path.

The compiler rejects unresolved task references and set-level cycles before it generates Airflow dependencies. The workflow owner gets a bounded configuration problem to fix instead of discovering a broken relationship after tasks have started.

Once the definition is corrected, the compiler rebuilds only that graph. Existing serial pipelines remain on their established route, which keeps a TaskSets error from changing unrelated workflows.