Ingestion file copy using apex
2 likes484 views
The document discusses using Apache Apex for big data file ingestion. It describes how Apex uses a directed acyclic graph (DAG) model with operators and streams to ingest data in parallel. The DAG includes components for reading data using a file splitter, block reader, and block writer. It also includes a synchronizer and file merger for output. The Apex ingestion handles large files, many small files, and offers features like compression, encryption, and processing during ingestion.
![Apache Apex Meetup
Input DAG - FileSplitter
● For each file in the directory:
■ [output] FileMetaData - file information
● Name
● Size
● Relative path
● Block IDs into which the file is virtually split
■ [output] BlockMetaData - block information
● BlockID
● Start position
● End position
● File URL
InputFile.txt
1073741824 (1GB)
input/data/InputFile.txt
[0,1,2,3,4,5,6,7,8]
1
134217728
268435456 (128MB)
hdfs://node18:8020/user/sandeep/input](https://image.slidesharecdn.com/ingestion-filecopyusingapex-160421140349/85/Ingestion-file-copy-using-apex-13-320.jpg)
Ingestion file copy using apex
- 1. Apache Apex Meetup Big Data File Ingestion using Apex Sandeep Deshmukh, PhD [email protected]
- 2. Apache Apex Meetup Contents ● What is Big Data Ingestion ● Challenges in File copy @ scale ● Ingestion using Apex ○ Input ○ Output ○ Key features ● Demo ● Summary
- 3. Apache Apex Meetup Directed Acyclic Graph (DAG) •A Stream is a sequence of data tuples •An Operator takes one or more input streams, performs computations & emits one or more output streams • Each Operator is YOUR custom business logic in java, or built-in operator from our open source library • Operator has many instances that run in parallel and each instance in single-threaded •Directed Acyclic Graph (DAG) is made up of operators and streams Apex: Application Programming Model Output StreamTuple Tuple FilteredStream Enriched Stream Enriched Stream er Operator er Operator er Operator er Operator Filtered Stream
- 4. Apache Apex Meetup What is Ingestion Data ingestion ● process of obtaining, importing, and processing data for later use or storage in a database Big Data Ingestion ● discovering the data sources ● importing the data ● processing data to produce intermediate data ● Send data out to durable data stores
- 5. Apache Apex Meetup Challenges in File copy @ scale ● Failure Recovery ● Copying big files in parallel ● Copying large number of small files ● Processing ○ Encryption ○ Compression ○ Compaction
- 6. Apache Apex Meetup DAG - Components Read Data Write Data Process
- 7. Apache Apex Meetup DAG - Read Data : Requirements ● Independent of input file type ○ HDFS ○ S3 ○ FTP ○ NFS ● Scale to large data ○ Large files ○ Large number of small files ● Configurable Bandwidth usage
- 8. Apache Apex Meetup DAG - Read Data Break the whole task into smaller sub-tasks Connect to input and scan for available data Assign smaller tasks for downstream operators StepsPurposeName Work on the sub-tasks given by Operator 1, one at a time Connect to source and read data as smaller tasks one-by-one Pass on the read data to downstream operator Write File Save the data read by Operator 2 File Splitter Block Reader File Writer
- 9. Apache Apex Meetup DAG - Simple Design File Splitter Block Reader File WriterBlockMetaData Data Challenges ● Reading files in parallel is not possible ○ Can have multiple Block Readers and File Writers reading multiple files in parallel but single file can’t be read by two Block Readers ● Failure recovery is hard
- 10. Apache Apex Meetup DAG - Read Data Break the whole task into smaller sub-tasks Connect to input and scan for available data Assign smaller tasks for downstream operators StepsPurposeName Work on the sub-tasks given by Operator 1, one at a time Connect to source and read data as smaller tasks one-by-one Pass on the read data to downstream operator Write File Save the data read by Operator 2 File Splitter Block Reader File Writer Check for completeness Make sure all smaller tasks for a file are completed by upstream operators & send file merger trigger Synchronizer
- 11. Apache Apex Meetup DAG - Input File Splitter Block Reader Block Writer BlockMetaData Data Block Reader Block Writer Synchronizer BlockMetaData FileMetaData BlockMetaData BlockMetaData Data
- 12. Apache Apex Meetup Input DAG - FileSplitter Scan input files/ directories Create smaller sub-tasks FileMetaData BlockMetaData File Splitter ● Parameters ○ input files/directories to copy data from ○ recursive - Yes / No ○ polling - Yes / No ○ bandwidth - MB / sec
- 13. Apache Apex Meetup Input DAG - FileSplitter ● For each file in the directory: ■ [output] FileMetaData - file information ● Name ● Size ● Relative path ● Block IDs into which the file is virtually split ■ [output] BlockMetaData - block information ● BlockID ● Start position ● End position ● File URL InputFile.txt 1073741824 (1GB) input/data/InputFile.txt [0,1,2,3,4,5,6,7,8] 1 134217728 268435456 (128MB) hdfs://node18:8020/user/sandeep/input
- 14. Apache Apex Meetup Input DAG - BlockReader Block Reader Read block from remote location and emit Data Data BlockMetaData ● Parameters Input URL: E.g.: hdfs://node18:8020/user/hduser/input BlockMetaData
- 15. Apache Apex Meetup Input DAG - BlockWriter Block Writer Write block data on local HDFS BlockMetaData BlockMetaData Data Saves data in apps directory
- 16. Apache Apex Meetup Input DAG - Synchronizer Track blocks for each file and send trigger once all the block for that file are available FileMetaDataSynchronizer FileMetaData BlockMetaData
- 17. Apache Apex Meetup DAG - Input File Splitter Block Reader Block Writer Synchronizer BlockMetaData Data FileMetaData BlockMetaData BlockMetaData FileMetaData
- 18. Apache Apex Meetup Output DAG - FileMerger Merge blocks to recreate original file FileMerger ● Parameters ○ Output directory to copy data to ○ Overwrite - Yes/No FileMetaData
- 19. Apache Apex Meetup Output DAG - FileMerger - FastMerge Magic Different Blocks: File : B1 DataNode1DataNode2 DataNode3DataNode4 B2 B1 B1 B2 B2 Bn Bn Bn BnB1 B2 1 2 1 1 2 2 n n n 1 2 n
- 20. Apache Apex Meetup ● Same replication factor ● On same HDFS cluster ● Same block size for all files ● Size of all files (except last) : multiple of block size Output DAG - FileMerger - FastMerge Magic
- 21. Apache Apex Meetup DAG - Complete File Splitter Block Reader Block Writer Synchronizer BlockMetaData BlockMetaData Data BlockMetaData FileMetaData FileMerger FileMetaData
- 22. Apache Apex Meetup Other features: Optional processing ● Compression ○ Gzip and lzo ● Encryption ○ PKI & AES ● Compaction ○ Size based ● Dedup ● Dimension Computation & Aggregation
- 24. Apache Apex Meetup Summary ● Easy to use ○ Configure and run ● Unified for batch and continuous ingestion ● Handles ○ Large files ○ Large number of small files
- 25. 25
- 26. Resources Apache Apex Meetup • Apache Apex website - http://apex.incubator.apache.org/ • Subscribe - http://apex.incubator.apache.org/community.html • Download - http://apex.incubator.apache.org/downloads.html • Twitter - @ApacheApex; Follow - https://twitter.com/apacheapex • Facebook - https://www.facebook.com/ApacheApex/ • Meetup - http://www.meetup.com/topics/apache-apex • Startup Program – Free Enterprise License for startups, Universities, Non-Profits