![Motivation Solution Implementation Demonstration
Edit /export/data/galaxy/universe_wsgi.ini
configuration file to add internal host name.
[galaxy_amqp]
host = ip-10-125-10-182.ec2.internal
port = 5672
userid = biouser
password = tester](https://image.slidesharecdn.com/exomeamazonchapman-110915054115-phpapp02/85/Developing-distributed-analysis-pipelines-with-shared-community-resources-using-CloudBioLinux-and-CloudMan-51-320.jpg)
![Motivation Solution Implementation Demonstration
Run level: YAML Configuration
$ cat /export/data/exome_example/config/run_info.yaml
---
fc_date: ’100326’
fc_name: FC6107FAAXX
details:
- files: [7_100326_FC6107FAAXX_1-chr22.fastq,
7_100326_FC6107FAAXX_2-chr22.fastq]
lane: 7
description: Test replicate 1
analysis: SNP calling
genome_build: hg19
algorithm:
quality_format: Standard
hybrid_bait: hybrid_selection/baits.bed
hybrid_target: hybrid_selection/targets.bed](https://image.slidesharecdn.com/exomeamazonchapman-110915054115-phpapp02/85/Developing-distributed-analysis-pipelines-with-shared-community-resources-using-CloudBioLinux-and-CloudMan-55-320.jpg)
![Motivation Solution Implementation Demonstration
Monitoring: Log files
$ less nextgen_analysis_server.py.o10
INFO: nextgen_pipeline: Processing sample: Test replicate 2;
lane 8; reference genome hg19; researcher ;
analysis method SNP calling
INFO: nextgen_pipeline:
Aligning lane 8_100326_FC6107FAAXX with bwa aligner
INFO: nextgen_pipeline:
Combining and preparing wig file [u’’, u’Test replicate 2’]
INFO: nextgen_pipeline:
Recalibrating [u’’, u’Test replicate 2’] with GATK](https://image.slidesharecdn.com/exomeamazonchapman-110915054115-phpapp02/85/Developing-distributed-analysis-pipelines-with-shared-community-resources-using-CloudBioLinux-and-CloudMan-61-320.jpg)
Developing distributed analysis pipelines with shared community resources using CloudBioLinux and CloudMan
- 1. Motivation Solution Implementation Demonstration Developing distributed analysis pipelines with shared community resources using CloudBioLinux and CloudMan Brad Chapman Bioinformatics Core Harvard School of Public Health 22 September 2011
- 2. Motivation Solution Implementation Demonstration Acknowledgements CloudBioLinux – Ntino Krampis, Tim Booth, Dawn Field, Pjotr Prins and CloudBioLinux community CloudMan – Enis Afgan, James Taylor Exome pipeline – HSPH, MGH, Win Hide, Oliver Hofmann
- 3. Motivation Solution Implementation Demonstration Follow along http://www.slideshare.net/chapmanb
- 4. Motivation Solution Implementation Demonstration Cue the “lots of data” slide ls -lh fastq/ 24G 1_110907_AD08A5ACXX_1_fastq.txt 21G 1_110907_AD08A5ACXX_2_fastq.txt 24G 2_110907_AD08A5ACXX_1_fastq.txt 20G 2_110907_AD08A5ACXX_2_fastq.txt
- 5. Motivation Solution Implementation Demonstration Rapidly changing tools
- 6. Motivation Solution Implementation Demonstration Science – fundamental challenge 75% one-off experimental 25% reused code
- 7. Motivation Solution Implementation Demonstration Unfortunate result http://news.ycombinator.com/item?id=2735537
- 8. Motivation Solution Implementation Demonstration Hard choices Computation Demands flexible, well-architected, scalable code Science Requires rapid turn around and experimentation
- 9. Motivation Solution Implementation Demonstration 2 solutions (at least) 1 Improve your programming skills 2 Utilize community resources
- 10. Motivation Solution Implementation Demonstration Become a better coder http://software-carpentry.org/
- 11. Motivation Solution Implementation Demonstration Community resources Share painful parts Base of well-written, scalable code Start each problem from a higher level of abstraction
- 12. Motivation Solution Implementation Demonstration Community components CloudBioLinux – install software CloudMan – manage cluster Exome analysis pipeline – do science
- 13. Motivation Solution Implementation Demonstration CloudBioLinux Amazon image with bioinformatics software and libraries Automated build framework Community effort to maintain and extend http://cloudbiolinux.org
- 14. Motivation Solution Implementation Demonstration CloudMan SGE cluster plus automation Web interface and monitoring Persistence and sharing Powers the Galaxy Cloud offering http://wiki.g2.bx.psu.edu/Admin/Cloud
- 15. Motivation Solution Implementation Demonstration Exome analysis pipeline Existing algorithms Aligners – Bowtie, BWA Variation – GATK Quality assessment – FastQC, Picard Messaging system – AMQP https://github.com/chapmanb/bcbb/ tree/master/nextgen
- 16. Motivation Solution Implementation Demonstration Fastq lane processing
- 17. Motivation Solution Implementation Demonstration Sample processing
- 18. Motivation Solution Implementation Demonstration Variant calling
- 19. Motivation Solution Implementation Demonstration Parallelization
- 20. Motivation Solution Implementation Demonstration
- 21. Motivation Solution Implementation Demonstration Amazon Virtual machines Share Reproduce Coordinate Accessibility
- 22. Motivation Solution Implementation Demonstration What are we going to do? Use AWS console to boot CloudBioLinux Setup CloudMan in AWS console Boot CloudMan instance with demo data
- 23. Motivation Solution Implementation Demonstration What are we going to do? continued Manage cluster with CloudMan interface Setup messaging queue Run pipeline, examine results Share cluster
- 24. Motivation Solution Implementation Demonstration CloudBioLinux Select and launch CloudBioLinux AMI from AWS console Connect FreeNX graphical client ssh Full tutorial PDF: http://j.mp/nnh5TE
- 25. Motivation Solution Implementation Demonstration Prep work Signup for AWS account: http://aws.amazon.com/ Create login key pair in AWS Console Install NX client: http://www.nomachine.com/select-package-client.php
- 27. Select CloudBioLinux image from Community AMIs
- 28. enter NX password in user-data (freenxpass: secret)
- 30. Get external hostname from Instances page
- 31. Connect using NX client, with ubuntu user and secret password
- 33. Connect with ssh, using private ssh key-pair
- 34. Terminate the server when finished
- 35. Motivation Solution Implementation Demonstration Setup CloudMan in AWS console Create a custom security group Full tutorial: http://wiki.g2.bx.psu.edu/Admin/Cloud
- 36. Create security group rules following wiki instructions
- 37. Final security group specifications
- 38. Motivation Solution Implementation Demonstration Boot CloudMan instance with demo data Start server Pass in CloudMan user data Load shared CloudMan image
- 39. Follow same procedure as CloudBioLinux
- 40. Create CloudMan user-data file cluster_name: cbldemo password: cbl access_key: your_access_key secret_key: your_long_AWS_secret_key
- 41. Provide user-data from file
- 42. Choose created security group
- 43. Login to instance with password from user-data
- 44. Motivation Solution Implementation Demonstration CloudMan share-an-instance Persist data in a CloudMan cluster Easily sharable For this demo cm-b53c6f1223f966914df347687f6fc818/shared/2011-10-07–14-00
- 45. Import shared instance with demo data
- 46. Motivation Solution Implementation Demonstration Manage cluster with CloudMan Web-based console Monitor running processes Add nodes to cluster as needed
- 47. CloudMan console to interact with cluster
- 48. Add node to cluster
- 49. Motivation Solution Implementation Demonstration Setup messaging communication Command line access to server Adjust RabbitMQ configuration Setup messaging queue
- 50. Motivation Solution Implementation Demonstration Command line access to server ssh -i ~/.ec2/id-kunkel.keypair [email protected] Follow approach used to connect to CloudBioLinux cluster; can also connect via NX
- 51. Motivation Solution Implementation Demonstration Edit /export/data/galaxy/universe_wsgi.ini configuration file to add internal host name. [galaxy_amqp] host = ip-10-125-10-182.ec2.internal port = 5672 userid = biouser password = tester
- 52. Motivation Solution Implementation Demonstration Setup messaging queue $ sudo rabbitmqctl add_user biouser tester creating user ’biouser’ ... ...done. $ sudo rabbitmqctl add_vhost bionextgen creating vhost ’bionextgen’ ... ...done. $ sudo rabbitmqctl set_permissions -p bionextgen biouser ".*" ".*" ".*" setting permissions for user ’biouser’ in vhost ’bionextgen’ .. ...done.
- 53. Motivation Solution Implementation Demonstration Run pipeline, examine results Ready to run distributed pipeline Demo data – two paired end fastq lanes Variant calling workflow
- 54. Motivation Solution Implementation Demonstration Input sequence data $ ls -1 /export/data/exome_example/fastq/ 7_100326_FC6107FAAXX_1-chr22.fastq 7_100326_FC6107FAAXX_2-chr22.fastq 8_100326_FC6107FAAXX_1-chr22.fastq 8_100326_FC6107FAAXX_2-chr22.fastq
- 55. Motivation Solution Implementation Demonstration Run level: YAML Configuration $ cat /export/data/exome_example/config/run_info.yaml --- fc_date: ’100326’ fc_name: FC6107FAAXX details: - files: [7_100326_FC6107FAAXX_1-chr22.fastq, 7_100326_FC6107FAAXX_2-chr22.fastq] lane: 7 description: Test replicate 1 analysis: SNP calling genome_build: hg19 algorithm: quality_format: Standard hybrid_bait: hybrid_selection/baits.bed hybrid_target: hybrid_selection/targets.bed
- 56. Motivation Solution Implementation Demonstration System level: YAML Configuration $ cat /export/data/galaxy/post_process.yaml --- program: bowtie: bowtie bwa: bwa ucsc_bigwig: wigToBigWig picard: /usr/share/java/picard gatk: /usr/share/java/gatk snpEff: /usr/share/java/snpeff fastqc: fastqc distributed: cluster_platform: sge platform_args: ’-q all.q’ cores_per_host: 1 rabbitmq_vhost: bionextgen
- 57. Motivation Solution Implementation Demonstration Run exome pipeline $ cd /export/data/work $ distributed_nextgen_pipeline.py /export/data/galaxy/post_process.yaml /export/data/exome_example/fastq /export/data/exome_example/config/run_info.yaml
- 58. Motivation Solution Implementation Demonstration What just happened?
- 59. Motivation Solution Implementation Demonstration Monitoring: SGE queues $ qstat ob-ID prior name state submit/start at queue -------------------------------------------------------------- 1 0.55500 nextgen_an r 18:16:32 [email protected] 2 0.55500 nextgen_an r 18:16:32 [email protected] 3 0.55500 automated_ r 18:16:47 [email protected]
- 60. Motivation Solution Implementation Demonstration Monitoring: Analysis directory $ cd /export/data/work $ ls -lh drwxr-xr-x 4.0 alignments -rw-r--r-- 2.0K automated_initial_analysis.py.o11 drwxr-xr-x 33 log -rw-r--r-- 15K nextgen_analysis_server.py.o10 -rw-r--r-- 15K nextgen_analysis_server.py.o9 drwxr-xr-x 102 tmp
- 61. Motivation Solution Implementation Demonstration Monitoring: Log files $ less nextgen_analysis_server.py.o10 INFO: nextgen_pipeline: Processing sample: Test replicate 2; lane 8; reference genome hg19; researcher ; analysis method SNP calling INFO: nextgen_pipeline: Aligning lane 8_100326_FC6107FAAXX with bwa aligner INFO: nextgen_pipeline: Combining and preparing wig file [u’’, u’Test replicate 2’] INFO: nextgen_pipeline: Recalibrating [u’’, u’Test replicate 2’] with GATK
- 62. Motivation Solution Implementation Demonstration Retrieve results: Copy files $ upload_to_galaxy.py /export/data/galaxy/post_process.yaml /export/data/exome_example/fastq /export/data/work /export/data/exome_example/config/run_info.yaml Final files copied into new directory; allows cleanup of analysis directory
- 63. Motivation Solution Implementation Demonstration Retrieve results: Output directory $ ls -lh /export/data/galaxy/storage/100326_FC6107FAAXX/7 -rw-r--r-- 38M 7_100326_FC6107FAAXX.bam -rw-r--r-- 22M 7_100326_FC6107FAAXX-coverage.bigwig -rw-r--r-- 72M 7_100326_FC6107FAAXX-gatkrecal.bam -rw-r--r-- 109K 7_100326_FC6107FAAXX-snp-effects.tsv -rw-r--r-- 827K 7_100326_FC6107FAAXX-snp-filter.vcf -rw-r--r-- 1.6M 7_100326_FC6107FAAXX-summary.pdf
- 64. Motivation Solution Implementation Demonstration Share results Share-an-instance Uses CloudMan web interface Reproducible research CloudBioLinux AMI – software CloudMan – data and configuration
- 65. CloudMan console enables push button sharing
- 66. Can make public or available to specific collaborators
- 67. When finished, turn everything off through CloudMan
- 68. Motivation Solution Implementation Demonstration Summary CloudBioLinux Shared machine image of biological software Boot from AWS console Connect with NX graphical client and ssh
- 69. Motivation Solution Implementation Demonstration Summary CloudMan Cluster setup and management Boot from share-an-instance Manage cluster through web interface Share final results
- 70. Motivation Solution Implementation Demonstration Summary Exome pipeline Parallel framework for running analyses Run using automated scripts Extract alignments, variant calls and summary information
- 71. Motivation Solution Implementation Demonstration Future: interfaces make it easier https://bitbucket.org/hbc/galaxy-central-hbc
- 72. Motivation Solution Implementation Demonstration Future: Simplified file selection
- 73. Motivation Solution Implementation Demonstration Future: Top level parameters
- 74. Motivation Solution Implementation Demonstration Future: Galaxy data libraries
- 75. Motivation Solution Implementation Demonstration Future: Galaxy analysis
- 76. Motivation Solution Implementation Demonstration Future: External UCSC visualization
- 77. Motivation Solution Implementation Demonstration Read more Step-by-step instructions http://j.mp/rp69nx Approaches to parallelism http://j.mp/nPQHcm Future work http://bcbio.wordpress.com