Welcome to DryPipe’s documentation¶
A Python DSL for pipelines¶
Don’t Repeat Yourself (D.R.Y) while writing pipelines, and stay away from leaky abstractions, use DryPipe !
What is a pipeline ?¶
A pipeline could be described as “a bunch of programs feeding data to one another”.
Programs within a pipeline tend to:
run for a long time
need large amounts of resources (cpu, memory, disk space, etc), requiring clusters to run (Slurm, Torque,etc)
be written in different languaged, have different CLI interfaces, file formats, etc.
long CODE->RUN->DEBUG->CODE cycles
The length of the debug cycle make pipelines difficult to debug.