Reproducible Computational Pipelines with Docker and Nextflow
6 likes2,679 views
This document summarizes a presentation about using Docker and Nextflow to create reproducible computational pipelines. It discusses two major challenges in computational biology being reproducibility and complexity. Containers like Docker help address these challenges by creating portable and standardized environments. Nextflow is introduced as a workflow framework that allows pipelines to run across platforms and isolates dependencies using containers, enabling fast prototyping. Examples are given of using Nextflow with Docker to run pipelines on different systems like HPC clusters in a scalable and reproducible way.
![$ nextflow run nextflow-io/rnatoy -with-docker
N E X T F L O W ~ version 0.14.3
Pulling nextflow-io/rnatoy ...
downloaded from https://github.com/nextflow-io/rnatoy.git
Launching 'nextflow-io/rnatoy' - revision: 9c61bf5ac5 [master]
R N A T O Y P I P E L I N E
=================================
genome : /User/../data/ggal_1_4885000_49020000.Ggal71.500bp.fa
annotat : /User/../data/ggal_1_4885000_49020000.bed.gff
pair1 : /User/../data/*_1.fq
pair2 : /User/../data/*_2.fq
[warm up] executor > local
[02/b08c28] Submitted process > buildIndex (ggal_1_4885000_49020000.Ggal71)
[ea/97d004] Submitted process > mapping (ggal_gut)
[98/16c9e5] Submitted process > mapping (ggal_liver)
[b5/38a0c7] Submitted process > makeTranscript (ggal_gut)
[00/e5efd6] Submitted process > makeTranscript (ggal_liver)
Saving: transcript_ggal_gut.gtf
Saving: transcript_ggal_liver.gtf](https://image.slidesharecdn.com/paoloditommaso-160404213941/85/Reproducible-Computational-Pipelines-with-Docker-and-Nextflow-39-320.jpg)
![$ nextflow run nextflow-io/rnatoy -revision v1.0
N E X T F L O W ~ version 0.14.3
Launching 'nextflow-io/rnatoy' - revision: 0d0443d8f7 [v1.0]
R N A T O Y P I P E L I N E
=================================
[35/cb611b] Submitted process > prepareTranscriptome (1)
[cd/239926] Submitted process > buildIndex (1)
[c6/f6488d] Submitted process > mapping (2)
[bc/b3ea76] Submitted process > mapping (1)
[f4/8d4628] Submitted process > makeTranscript (1)
[eb/92db7f] Submitted process > makeTranscript (2)
Saving: transcript_ggal_alpha.gtf
Saving: transcript_ggal_beta.gtf](https://image.slidesharecdn.com/paoloditommaso-160404213941/85/Reproducible-Computational-Pipelines-with-Docker-and-Nextflow-40-320.jpg)
Reproducible Computational Pipelines with Docker and Nextflow
- 1. Paolo DiTommaso - Notredame Lab Center for Genomic Regulation (CRG) HPC Advisory Council - 22 March 2016, Lugano Reproducible computational pipelines with Docker and Nextflow
- 2. @PaoloDiTommaso Research software engineer Comparative Bioinformatics, Notredame Lab Center for Genomic Regulation (CRG)
- 3. TWO MAJOR CHALLENGES IN COMPUTATIONAL BIOLOGY
- 7. To replicate the result of a typical computational biology paper requires 280 hours!
- 9. COMPLEXITY • Dozens of dependencies (binary tools, compilers, libraries, system tools, etc) • Experimental nature of academic SW tends to be difficult to install, configure and deploy • Heterogeneous executing platforms and system architecture (laptop→supercomputer)
- 11. CONTAINERS ARE THETHIRD BIG WAVE INVIRTUALISATION TECHNOLOGY
- 12. BENEFITS • Smaller images (~100MB) • Fast instantiation time (~1sec) • Almost native performance • Easy to build, publish, share and deploy • Transparent build process
- 14. Host NAIVE APPROACH Docker image User application Binary tools Workflow scripts Config files Libraries Environment Operating System Docker engine
- 15. SCALING OUT . . . .
- 16. CONTAINERS ORCHESTRATION • Swarm • Fleet • Kubernetes • Marathon
- 17. NOTTHE RIGHT ANSWER FOR COMPUTATIONAL PIPELINES
- 19. OUR SOLUTION Nextflow Host file system Registry
- 20. • A workflow framework that allows the same pipeline to run across different platforms • Provides a high level parallelisation model • Isolates task dependencies using containers • It enables fast prototyping reusing any existing piece of software
- 22. REACTIVE NETWORK
- 23. DATAFLOW • Declarative computational model for concurrent processes • Processes wait for data, when an input set is ready the process is executed • They communicate by using dataflow variables i.e. async stream of data called channels • Parallelisation and tasks dependencies are implicitly defined by process in/out declarations
- 24. PLATFORM AGNOSTIC Dataflow Task dispatcher Executors POSIX processes qsub/bsub/... tasks DSL interpreter nextflow
- 26. BATCH SCHEDULER nextflow login node NFS cluster node cluster node cluster node cluster node batch scheduler submit tasks cluster node
- 27. DISTRIBUTED MODE Login node NFS/Lustre Job request cluster node cluster node Job wrapper #!/bin/bash #$ -q <queue> #$ -pe ompi <nodes> #$ -l virtual_free=<mem> mpirun nextflow run <your-pipeline> -with-mpi HPC cluster nextflow cluster nextflow driver nextflow worker nextflow worker nextflow worker
- 28. USE CASE • Deploying phylogenetic pipeline in BSC MareNostrum • 500 lines of Nextflow scripting • ~ 400k jobs • 512 cores - 32 nodes • ~ 50k cpu/h
- 30. DOCKER AT CRG Nextflow Config file Pipeline script docker registry head node Univa grid engine
- 31. PROS • Dead easy deployment procedure • Self-contained and precise controlled runtime • Rapidly reproduce any former configuration • Consistent results over time and across different platforms
- 32. CONS • Requires a modern Linux kernel (≥3.10) • Security concerns • Containers/images cleanup
- 33. SHIFTER • Container technology developed at NERSC • Nextflow has built-in support for Shifter • Experimental feature, under test • It only requires an extra setting in the configuration file
- 35. BENCHMARK* * DiTommaso P, Palumbo E, Chatzou M, Prieto P, Heuer ML, Notredame C. (2015) The impact of Docker containers on the performance of genomic pipelines. PeerJ 3:e1273 https://dx.doi.org/10.7717/peerj.1273
- 36. DEMO
- 38. $ nextflow run nextflow-io/rnatoy -with-docker
- 39. $ nextflow run nextflow-io/rnatoy -with-docker N E X T F L O W ~ version 0.14.3 Pulling nextflow-io/rnatoy ... downloaded from https://github.com/nextflow-io/rnatoy.git Launching 'nextflow-io/rnatoy' - revision: 9c61bf5ac5 [master] R N A T O Y P I P E L I N E ================================= genome : /User/../data/ggal_1_4885000_49020000.Ggal71.500bp.fa annotat : /User/../data/ggal_1_4885000_49020000.bed.gff pair1 : /User/../data/*_1.fq pair2 : /User/../data/*_2.fq [warm up] executor > local [02/b08c28] Submitted process > buildIndex (ggal_1_4885000_49020000.Ggal71) [ea/97d004] Submitted process > mapping (ggal_gut) [98/16c9e5] Submitted process > mapping (ggal_liver) [b5/38a0c7] Submitted process > makeTranscript (ggal_gut) [00/e5efd6] Submitted process > makeTranscript (ggal_liver) Saving: transcript_ggal_gut.gtf Saving: transcript_ggal_liver.gtf
- 40. $ nextflow run nextflow-io/rnatoy -revision v1.0 N E X T F L O W ~ version 0.14.3 Launching 'nextflow-io/rnatoy' - revision: 0d0443d8f7 [v1.0] R N A T O Y P I P E L I N E ================================= [35/cb611b] Submitted process > prepareTranscriptome (1) [cd/239926] Submitted process > buildIndex (1) [c6/f6488d] Submitted process > mapping (2) [bc/b3ea76] Submitted process > mapping (1) [f4/8d4628] Submitted process > makeTranscript (1) [eb/92db7f] Submitted process > makeTranscript (2) Saving: transcript_ggal_alpha.gtf Saving: transcript_ggal_beta.gtf
- 41. $ vim nextflow.config process { executor = 'slurm' memory = 10.GB cpus = 32 }
- 42. WHO IS USING NEXTFLOW?
- 43. WHO IS USING NEXTFLOW? International Agency for Research on Cancer Lyon, France Writing reproducible and scalable bioinformatics pipelines using nextflow, docker and github Matthieu Foll Nov. 12th 2015
- 44. CONCLUSION • Containers are a game-changer for computational workflows packaging and deployment • Nextflow is a reactive/functional framework for computational workflows. • Docker + Nextflow = Reproducible self- contained pipelines.
- 45. ACKNOWLEDGMENT Evan Floden, CRG Emilio Palumbo, CRG Maria Chatzou, CRG Cedric Notredame, CRG
- 46. THANKS
- 47. LINKS project home http://nextflow.io GitHub repository http://github.com/nextflow-io/nextflow Docker benchmark https://peerj.com/articles/1273/ Docker-Univa white paper http://www.nextflow.io/misc/Univa-Docker-Whitepaper_FINAL.pdf