SlideShare a Scribd company logo

Internal Hive

Download as PPTX, PDF
Recruit Technologies, profile picture
Recruit Technologies

The document provides an overview of Hive architecture and workflow. It discusses how Hive converts HiveQL queries to MapReduce jobs through its compiler. The compiler includes components like the parser, semantic analyzer, logical and physical plan generators, and logical and physical optimizers. It analyzes sample HiveQL queries and shows the transformations done at each compiler stage to generate logical and physical execution plans consisting of operators and tasks.

Technology
1 of 41
Downloaded 959 times
1
2
3
4
5
6
7
Most read
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
Inside Hive (for beginners)1Takeshi NAKANO / Recruit Co. Ltd.
Why?Hive is good tool for non-specialist!The number of M/R controls the Hive processing time.↓How can we reduce the number?What can we do for this on writing HiveQL?↓How does Hive convert HiveQLto M/R jobs?On this, what optimizing processes are adopted?7/6/2011HIVE - A warehouse solution over Map Reduce Framework2
Don’t you have..This fb’s paper has a lot of information!But this is a little old..7/6/2011HIVE - A warehouse solution over Map Reduce Framework3
Component Level Analysis7/6/2011HIVE - A warehouse solution over Map Reduce Framework4
Hive Architecture / Exec Flow7/6/2011HIVE - A warehouse solution over Map Reduce Framework5ClientHadoopMetastoreDriverCompiler
ClientHadoopDriverCompilerHive WorkflowHive has the operators which are minimum processing units.The process of each operator is done with HDFS operation or M/R jobs.The compiler converts HiveQL to the sets of operators.7/6/2011HIVE - A warehouse solution over Map Reduce Framework6Metastore
Hive WorkflowOperators7/6/2011HIVE - A warehouse solution over Map Reduce Framework7
ClientHadoopMetastoreDriverCompilerHive WorkflowFor M/R processing, Hiveuses ExecMaper and ExecReducer.On processing, we have 2 modes.Local processing modeDistributed processing mode7/6/2011HIVE - A warehouse solution over Map Reduce Framework8
ClientHadoopMetastoreDriverCompilerHive WorkflowOn 1(Local mode)Hive fork the process with hadoop command.The plan.xml is made just on 1 and the single node processes this.On 2(Distributed mode).Hive send the process to exsistingJobTracker.The information is housed on DistributedCacheand processed on multi nodes.7/6/2011HIVE - A warehouse solution over Map Reduce Framework9
Compiler : How to Process HiveQL7/6/2011HIVE - A warehouse solution over Map Reduce Framework10ClientHadoopMetastoreDriverCompiler
“Plumbing” of HIVE compiler7/6/201111HIVE - A warehouse solution over Map Reduce Framework
“Plumbing” of HIVE compiler7/6/201112HIVE - A warehouse solution over Map Reduce Framework
Compiler Overview13ParserSemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizer
Compiler Overview14HiveQLParserASTSemanticAnalyzerQBLogicalPlan Gen.Operator TreeLogicalOptimizerOperator TreePhysicalPlan Gen.Task TreePhysicalOptimizerTask Tree
ParserHiveQLASTINSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono);HiveQLTOK_QUERY + TOK_FROM + TOK_JOIN + TOK_TABREF + TOK_TABNAME + "access_log_hbase" + a + TOK_TABREF + TOK_TABNAME + "product_hbase" + "p" + "=" + "." + TOK_TABLE_OR_COL + "a" + "access_log_hbase" + "." + TOK_TABLE_OR_COL + "p" + "prono“AST + TOK_INSERT + TOK_DESTINATION + TOK_TAB + TOK_TABNAME + "access_log_temp2" + TOK_SELECT + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "a" + "user" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "a" + "prono" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "p" + "maker" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "p" + "price"SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser
ParserSQLASTINSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono);SQLTOK_QUERY + TOK_FROM + TOK_JOIN + TOK_TABREF + TOK_TABNAME + "access_log_hbase" + a + TOK_TABREF + TOK_TABNAME + "product_hbase" + "p" + "=" + "." + TOK_TABLE_OR_COL + "a" + "access_log_hbase" + "." + TOK_TABLE_OR_COL + "p" + "prono“ + TOK_INSERT + TOK_DESTINATION + TOK_TAB + TOK_TABNAME + "access_log_temp2" + TOK_SELECT + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "a" + "user" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "a" + "prono" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "p" + "maker" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "p" + "price"AST123SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser
17Semantic Analyzer (1/2)ASTQB+ TOK_FROM + TOK_JOIN + TOK_TABREF + TOK_TABNAME + "access_log_hbase" + a + TOK_TABREF + TOK_TABNAME + "product_hbase" + "p" + "=" + "." + TOK_TABLE_OR_COL + "a" + "access_log_hbase" + "." + TOK_TABLE_OR_COL + "p" + "prono“AST1QBMetaDataAliasTo Table Info“a”=Table Info(“access_log_hbase”)“p”=Table Info(“product_hbase”)ParseInfoJoin Node+ TOK_JOIN + TOK_TABREF … + TOK_TABREF … + “=” …SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser17
18Semantic Analyzer (2/2)ASTQB + TOK_DESTINATION + TOK_TAB + TOK_TABNAME + "access_log_temp2”AST2QBParseInfoNameTo Destination Node+ TOK_TAB + TOK_TABNAME +"access_log_temp2”SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser1818
19Semantic Analyzer (2/2)ASTQB + TOK_SELECT + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "a" + "user" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "a" + "prono" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "p" + "maker" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "p" + "price"ASTQBParseInfo3Name To Select Node+ TOK_SELECT + TOK_SELEXPR … + TOK_SELEXPR … + TOK_SELEXPR … + TOK_SELEXPR …SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser1919
20Logical Plan Generator (1/4)QBOPTreeQBMetaDataAliasTo Table Info“a”=Table Info(“access_log_hbase”)“p”=Table Info(“product_hbase”)OPTreeTableScanOperator(“access_log_hbase”)TableScanOperator(“product_hbase”)SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser2020
21Logical Plan Generator (2/4)QBOPTreeQBParseInfo + TOK_JOIN + TOK_TABREF + TOK_TABNAME + "access_log_hbase" + a + TOK_TABREF + TOK_TABNAME + "product_hbase" + "p" + "=" + "." + TOK_TABLE_OR_COL + "a" + "access_log_hbase" + "." + TOK_TABLE_OR_COL + "p" + "prono“ReduceSinkOperator(“access_log_hbase”)ReduceSinkOperator(“product_hbase”)OPTreeJoinOperatorSemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser
22Logical Plan Generator (3/4)QBOPTreeQBParseInfoName To Select Node+ TOK_SELECT + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "a" + "user" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "a" + "prono" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "p" + "maker" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "p" + "price"OPTreeSelectOperatorSemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser
23Logical Plan Generator (4/4)QBOPTreeQBMetaDataName To Destination Table Info“insclause-0”= Table Info(“access_log_temp2”)OPTreeFileSinkOperatorSemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser
Logical Plan Generator (result)24LCF OPTreeTableScanOperatorTS_1TableScanOperatorTS_0ReduceSinkOperatorRS_2ReduceSinkOperatorRS_3JoinOperatorJOIN_4SelectOperatorSEL_5FileSinkOperatorFS_6SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser
Logical OptimizerSemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser252525
Logical Optimizer (Predicate Push Down)INSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono);INSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono) WHERE p.maker = 'honda';SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser2626
Logical Optimizer (Predicate Push Down)TableScanOperatorTS_1TableScanOperatorTS_0INSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono);ReduceSinkOperatorRS_3ReduceSinkOperatorRS_2JoinOperatorJOIN_4INSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono) WHERE p.maker = 'honda';SelectOperatorSEL_6FileSinkOperatorFS_7SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser2727
INSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono);INSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono) WHERE p.maker = 'honda';Logical Optimizer (Predicate Push Down)TableScanOperatorTS_1TableScanOperatorTS_0ReduceSinkOperatorRS_3ReduceSinkOperatorRS_2JoinOperatorJOIN_4FilterOperatorFIL_5(_col8 = 'honda')SelectOperatorSEL_6FileSinkOperatorFS_7SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser2828
Logical Optimizer (Predicate Push Down)TableScanOperatorTS_1TableScanOperatorTS_0INSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono);FilterOperatorFIL_8(maker = 'honda')ReduceSinkOperatorRS_2ReduceSinkOperatorRS_3JoinOperatorJOIN_4INSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono) WHERE p.maker = 'honda';FilterOperatorFIL_5(_col8 = 'honda')SelectOperatorSEL_6FileSinkOperatorFS_7SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser2929
30Physical Plan GeneratorOPTreeTaskTreeMoveTask(Stage-0)OpeTreeLoadTableDescTableScanOperator(TS_0)TableScanOperator(TS_1)ReduceSinkOperator(RS_2)MapRedTask(Stage-1/root)ReduceSinkOperator(RS_3)JoinOperator(JOIN_4)SelectOperator(SEL_5)FileSinkOperator(FS_6) StatsTask(Stage-2)SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser3030
OPTreeTaskTreeMapRedTask (Stage-1/root)TableScanOperator(TS_0)Physical Plan Generator (result)31LCF MapperTableScanOperatorTS_1TableScanOperatorTS_0TableScanOperator(TS_1)ReduceSinkOperatorRS_2ReduceSinkOperatorRS_3ReduceSinkOperator(RS_2)MapRedTask(Stage-1/root)ReduceSinkOperator(RS_3)ReducerJoinOperatorJOIN_4JoinOperator(JOIN_4)SelectOperatorSEL_5SelectOperator(SEL_5)FileSinkOperatorFS_6SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser313131
32Physical OptimizerTaskTreeTaskTreejava/org/apache/hadoop/hive/ql/optimizer/physical/以下SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser
33Physical Optimizer (MapJoinResolver)TaskTreeTaskTreeMapRedTask (Stage-1)MapperTableScanOperatorTS_1TableScanOperatorTS_0MapJoinOperatorMAPJOIN_7SelectOperatorSEL_8SelectOperatorSEL_5FileSinkOperatorFS_6SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser33
34Physical Optimizer (MapJoinResolver)TaskTreeTaskTreeMapredLocalTask(Stage-7)MapRedTask (Stage-1)TableScanOperatorTS_0MapperTableScanOperatorTS_1TableScanOperatorTS_0HashTableSinkOperatorHASHTABLESINK_11MapJoinOperatorMAPJOIN_7MapRedTask (Stage-1)SelectOperatorSEL_8MapperTableScanOperatorTS_1SelectOperatorSEL_5MapJoinOperatorMAPJOIN_7FileSinkOperatorFS_6SelectOperatorSEL_8SelectOperatorSEL_5FileSinkOperatorFS_6SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser34
In the end7/6/2011HIVE - A warehouse solution over Map Reduce Framework35ClientHadoopMetastoreDriverCompiler
In the end36HiveQLParserASTSemanticAnalyzerQBLogicalPlan Gen.Operator TreeLogicalOptimizerOperator TreePhysicalPlan Gen.Task TreePhysicalOptimizerTask Tree
End7/6/201137
Appendix: What does Explain show?7/6/2011HIVE - A warehouse solution over Map Reduce Framework38
Appendix: What does Explain show?hive> explain INSERT OVERWRITE TABLE access_log_temp2 > SELECT a.user, a.prono, p.maker, p.price > FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono);OKABSTRACT SYNTAX TREE: (TOK_QUERY (TOK_FROM (TOK_JOIN (TOK_TABREF (TOK_TABNAME access_log_hbase) a) (TOK_TABREF (TOK_TABNAME product_hbase) p) (= (. (TOK_TABLE_OR_COL a) prono) (. (TOK_TABLE_OR_COL p) prono)))) (TOK_INSERT (TOK_DESTINATION (TOK_TAB (TOK_TABNAME access_log_temp2))) (TOK_SELECT (TOK_SELEXPR (. (TOK_TABLE_OR_COL a) user)) (TOK_SELEXPR (. (TOK_TABLE_OR_COL a) prono)) (TOK_SELEXPR (. (TOK_TABLE_OR_COL p) maker)) (TOK_SELEXPR (. (TOK_TABLE_OR_COL p) price)))))STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 Stage-2 depends on stages: Stage-0STAGE PLANS: Stage: Stage-1 Map Reduce Alias -> Map Operator Tree: aTableScan alias: a Reduce Output Operator key expressions:expr: prono type: int sort order: + Map-reduce partition columns:expr: prono type: int tag: 0 value expressions:expr: user type: stringexpr: prono type: int pTableScan alias: p Reduce Output Operator key expressions:expr: prono type: int sort order: + Map-reduce partition columns:expr: prono type: int tag: 1 value expressions:expr: maker type: stringexpr: price type: intReduce Operator Tree: Join Operator condition map: Inner Join 0 to 1 condition expressions: 0 {VALUE._col0} {VALUE._col2} 1 {VALUE._col1} {VALUE._col2}handleSkewJoin: falseoutputColumnNames: _col0, _col2, _col6, _col7 Select Operator expressions:expr: _col0 type: stringexpr: _col2 type: intexpr: _col6 type: stringexpr: _col7 type: intoutputColumnNames: _col0, _col1, _col2, _col3 File Output Operator compressed: falseGlobalTableId: 1 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormatserde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.access_log_temp2 Stage: Stage-0 Move Operator tables: replace: true table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormatserde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.access_log_temp2 Stage: Stage-2 Stats-Aggr OperatorTime taken: 0.1 secondshive>
Appendix: What does Explain show?hive> explain INSERT OVERWRITE TABLE access_log_temp2 > SELECT a.user, a.prono, p.maker, p.price > FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono);OKABSTRACT SYNTAX TREE: (TOK_QUERY (TOK_FROM (TOK_JOIN (TOK_TABREF (TOK_TABNAME access_log_hbase) a) (TOK_TABREF (TOK_TABNAME product_hbase) p) (= (. (TOK_TABLE_OR_COL a) prono) (. (TOK_TABLE_OR_COL p) prono)))) (TOK_INSERT (TOK_DESTINATION (TOK_TAB (TOK_TABNAME access_log_temp2))) (TOK_SELECT (TOK_SELEXPR (. (TOK_TABLE_OR_COL a) user)) (TOK_SELEXPR (. (TOK_TABLE_OR_COL a) prono)) (TOK_SELEXPR (. (TOK_TABLE_OR_COL p) maker)) (TOK_SELEXPR (. (TOK_TABLE_OR_COL p) price)))))STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 Stage-2 depends on stages: Stage-0STAGE PLANS: Stage: Stage-1 Map Reduce Alias -> Map Operator Tree: aTableScan alias: aReduce Output Operator key expressions:expr: prono type: int sort order: + Map-reduce partition columns:expr: prono type: int tag: 0 value expressions:expr: user type: stringexpr: prono type: int pTableScan alias: pReduce Output Operator key expressions:expr: prono type: int sort order: + Map-reduce partition columns:expr: prono type: int tag: 1 value expressions:expr: maker type: stringexpr: price type: intABSTRACT SYNTAX TREE:STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 Stage-2 depends on stages: Stage-0STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree:TableScan Reduce Output OperatorTableScan Reduce Output Operator Reduce Operator Tree: Join Operator Select Operator File Output Operator Stage: Stage-0 Move Operator Stage: Stage-2 Stats-Aggr OperatorReduce Operator Tree: Join Operator condition map: Inner Join 0 to 1 condition expressions: 0 {VALUE._col0} {VALUE._col2} 1 {VALUE._col1} {VALUE._col2}handleSkewJoin: falseoutputColumnNames: _col0, _col2, _col6, _col7 Select Operator expressions:expr: _col0 type: stringexpr: _col2 type: intexpr: _col6 type: stringexpr: _col7 type: intoutputColumnNames: _col0, _col1, _col2, _col3File Output Operator compressed: falseGlobalTableId: 1 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormatserde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.access_log_temp2 Stage: Stage-0 Move Operator tables: replace: true table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormatserde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.access_log_temp2 Stage: Stage-2 Stats-Aggr OperatorTime taken: 0.1 secondshive>
Appendix: What does Explain show?ABSTRACT SYNTAX TREE:STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 Stage-2 depends on stages: Stage-0STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree:TableScan Reduce Output OperatorTableScan Reduce Output Operator Reduce Operator Tree: Join Operator Select Operator File Output Operator Stage: Stage-0 Move Operator Stage: Stage-2 Stats-Aggr OperatorMapRedTask (Stage-1/root)MapperTableScanOperatorTS_1TableScanOperatorTS_0ReduceSinkOperatorRS_2ReduceSinkOperatorRS_3ReducerJoinOperatorJOIN_4≒SelectOperatorSEL_5FileSinkOperatorFS_6MoveTask (Stage-0)Stats Task (Stage-2)

More Related Content

PDF
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Noritaka Sekiyama
 
PDF
What is new in Apache Hive 3.0?
DataWorks Summit
 
PDF
Hive tuning
Michael Zhang
 
PPTX
Using Apache Hive with High Performance
Inderaj (Raj) Bains
 
PPTX
Introduction to HiveQL
kristinferrier
 
PPT
Zookeeper Introduce
jhao niu
 
PDF
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Databricks
 
PPTX
Apache hive introduction
Mahmood Reza Esmaili Zand
 
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Noritaka Sekiyama
 
What is new in Apache Hive 3.0?
DataWorks Summit
 
Hive tuning
Michael Zhang
 
Using Apache Hive with High Performance
Inderaj (Raj) Bains
 
Introduction to HiveQL
kristinferrier
 
Zookeeper Introduce
jhao niu
 
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Databricks
 
Apache hive introduction
Mahmood Reza Esmaili Zand
 

What's hot (20)

PPTX
Hive + Tez: A Performance Deep Dive
DataWorks Summit
 
PPTX
Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...
StampedeCon
 
PDF
Understanding Query Plans and Spark UIs
Databricks
 
PPTX
Apache Hive Tutorial
Sandeep Patil
 
PDF
A Thorough Comparison of Delta Lake, Iceberg and Hudi
Databricks
 
PDF
Aurora MySQL Backtrack을 이용한 빠른 복구 방법 - 진교선 :: AWS Database Modernization Day 온라인
Amazon Web Services Korea
 
PPTX
Session 14 - Hive
AnandMHadoop
 
PDF
MySQL Database Architectures - InnoDB ReplicaSet & Cluster
Kenny Gryp
 
PPTX
Kudu Deep-Dive
Supriya Sahay
 
PDF
HBase Advanced - Lars George
JAX London
 
PDF
The Parquet Format and Performance Optimization Opportunities
Databricks
 
PDF
Deep Dive: Memory Management in Apache Spark
Databricks
 
PDF
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안
SANG WON PARK
 
PDF
Redshift VS BigQuery
Kostas Pardalis
 
PDF
CDC Stream Processing with Apache Flink
Timo Walther
 
PPTX
HBase Low Latency
DataWorks Summit
 
PDF
Amazon OpenSearch Deep dive - 내부구조, 성능최적화 그리고 스케일링
Amazon Web Services Korea
 
PPTX
MySQL_MariaDB-성능개선-202201.pptx
NeoClova
 
PDF
효율적인 빅데이터 분석 및 처리를 위한 Glue, EMR 활용 - 김태현 솔루션즈 아키텍트, AWS :: AWS Summit Seoul 2019
Amazon Web Services Korea
 
PDF
AWS EMR Cost optimization
SANG WON PARK
 
Hive + Tez: A Performance Deep Dive
DataWorks Summit
 
Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...
StampedeCon
 
Understanding Query Plans and Spark UIs
Databricks
 
Apache Hive Tutorial
Sandeep Patil
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
Databricks
 
Aurora MySQL Backtrack을 이용한 빠른 복구 방법 - 진교선 :: AWS Database Modernization Day 온라인
Amazon Web Services Korea
 
Session 14 - Hive
AnandMHadoop
 
MySQL Database Architectures - InnoDB ReplicaSet & Cluster
Kenny Gryp
 
Kudu Deep-Dive
Supriya Sahay
 
HBase Advanced - Lars George
JAX London
 
The Parquet Format and Performance Optimization Opportunities
Databricks
 
Deep Dive: Memory Management in Apache Spark
Databricks
 
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안
SANG WON PARK
 
Redshift VS BigQuery
Kostas Pardalis
 
CDC Stream Processing with Apache Flink
Timo Walther
 
HBase Low Latency
DataWorks Summit
 
Amazon OpenSearch Deep dive - 내부구조, 성능최적화 그리고 스케일링
Amazon Web Services Korea
 
MySQL_MariaDB-성능개선-202201.pptx
NeoClova
 
효율적인 빅데이터 분석 및 처리를 위한 Glue, EMR 활용 - 김태현 솔루션즈 아키텍트, AWS :: AWS Summit Seoul 2019
Amazon Web Services Korea
 
AWS EMR Cost optimization
SANG WON PARK
 
Ad

Similar to Internal Hive (20)

PDF
Pdxpugday2010 pg90
Selena Deckelmann
 
PDF
Hive_p
Samchu Li
 
PPT
Python And GIS - Beyond Modelbuilder And Pythonwin
Chad Cooper
 
PPTX
Lean & Mean Tokyo Cabinet Recipes (with Lua) - FutureRuby '09
Ilya Grigorik
 
ODP
Python 3000
Bob Chao
 
PPT
Developing A Real World Logistic Application With Oracle Application - UKOUG ...
Roel Hartman
 
DOCX
Computer science project work
rahulchamp2345
 
PPT
Migration testing framework
IndicThreads
 
PDF
Code Management
Y. Thong Kuah
 
PDF
Танки_в_Лунапарке: нагрузочное_тестирование_в_Яндексе
Yandex
 
PPTX
Jquery mobile
Eric Turcotte
 
PPTX
Wellington APAC Groundbreakers tour - Upgrading to the 12c Optimizer
Connor McDonald
 
PPT
Introduction to Assembly Language
Motaz Saad
 
KEY
CloudKit
Jon Crosby
 
PPT
JDBC Java Database Connectivity
Ranjan Kumar
 
PDF
VoCamp Seoul2009 Sparql
kwangsub kim
 
PPT
What's new in Rails 2?
brynary
 
PPTX
Html5
dotNETUserGroupDnipro
 
PPT
Php
Mindtree
 
PDF
TYPO3 Extension development using new Extbase framework
Christian Trabold
 
Pdxpugday2010 pg90
Selena Deckelmann
 
Hive_p
Samchu Li
 
Python And GIS - Beyond Modelbuilder And Pythonwin
Chad Cooper
 
Lean & Mean Tokyo Cabinet Recipes (with Lua) - FutureRuby '09
Ilya Grigorik
 
Python 3000
Bob Chao
 
Developing A Real World Logistic Application With Oracle Application - UKOUG ...
Roel Hartman
 
Computer science project work
rahulchamp2345
 
Migration testing framework
IndicThreads
 
Code Management
Y. Thong Kuah
 
Танки_в_Лунапарке: нагрузочное_тестирование_в_Яндексе
Yandex
 
Jquery mobile
Eric Turcotte
 
Wellington APAC Groundbreakers tour - Upgrading to the 12c Optimizer
Connor McDonald
 
Introduction to Assembly Language
Motaz Saad
 
CloudKit
Jon Crosby
 
JDBC Java Database Connectivity
Ranjan Kumar
 
VoCamp Seoul2009 Sparql
kwangsub kim
 
What's new in Rails 2?
brynary
 
Html5
dotNETUserGroupDnipro
 
Php
Mindtree
 
TYPO3 Extension development using new Extbase framework
Christian Trabold
 
Ad

More from Recruit Technologies (20)

PDF
新卒2年目が鍛えられたコードレビュー道場
Recruit Technologies
 
PDF
カーセンサーで深層学習を使ってUX改善を行った事例とそこからの学び
Recruit Technologies
 
PDF
Rancherを活用した開発事例の紹介　～Rancherのメリットと辛いところ～
Recruit Technologies
 
PDF
Tableau活用4年の軌跡
Recruit Technologies
 
PDF
HadoopをBQにマイグレしようとしてる話
Recruit Technologies
 
PDF
LT（自由）
Recruit Technologies
 
PDF
リクルートグループの現場事例から見る　AI/ディープラーニング ビジネス活用の勘所
Recruit Technologies
 
PDF
Company Recommendation for New Graduates via Implicit Feedback Multiple Matri...
Recruit Technologies
 
PDF
リクルート式AIの活用法
Recruit Technologies
 
PDF
銀行ロビーアシスタント
Recruit Technologies
 
PDF
リクルートにおけるマルチモーダル Deep Learning Web API 開発事例
Recruit Technologies
 
PDF
ユーザー企業内製CSIRTにおける対応のポイント
Recruit Technologies
 
PDF
ユーザーからみたre:Inventのこれまでと今後
Recruit Technologies
 
PDF
Struggling with BIGDATA　－リクルートおけるデータサイエンス/エンジニアリング－
Recruit Technologies
 
PDF
EMRでスポットインスタンスの自動入札ツールを作成する
Recruit Technologies
 
PDF
RANCHERを使ったDev(Ops)
Recruit Technologies
 
PDF
リクルートにおけるセキュリティ施策方針とCSIRT組織運営のポイント
Recruit Technologies
 
PDF
ユーザー企業内製CSIRTにおける対応のポイント
Recruit Technologies
 
PDF
リクルートテクノロジーズが語る 企業における、「AI／ディープラーニング」活用のリアル
Recruit Technologies
 
PDF
「リクルートデータセット」 ～公開までの道のりとこれから～
Recruit Technologies
 
新卒2年目が鍛えられたコードレビュー道場
Recruit Technologies
 
カーセンサーで深層学習を使ってUX改善を行った事例とそこからの学び
Recruit Technologies
 
Rancherを活用した開発事例の紹介　～Rancherのメリットと辛いところ～
Recruit Technologies
 
Tableau活用4年の軌跡
Recruit Technologies
 
HadoopをBQにマイグレしようとしてる話
Recruit Technologies
 
LT（自由）
Recruit Technologies
 
リクルートグループの現場事例から見る　AI/ディープラーニング ビジネス活用の勘所
Recruit Technologies
 
Company Recommendation for New Graduates via Implicit Feedback Multiple Matri...
Recruit Technologies
 
リクルート式AIの活用法
Recruit Technologies
 
銀行ロビーアシスタント
Recruit Technologies
 
リクルートにおけるマルチモーダル Deep Learning Web API 開発事例
Recruit Technologies
 
ユーザー企業内製CSIRTにおける対応のポイント
Recruit Technologies
 
ユーザーからみたre:Inventのこれまでと今後
Recruit Technologies
 
Struggling with BIGDATA　－リクルートおけるデータサイエンス/エンジニアリング－
Recruit Technologies
 
EMRでスポットインスタンスの自動入札ツールを作成する
Recruit Technologies
 
RANCHERを使ったDev(Ops)
Recruit Technologies
 
リクルートにおけるセキュリティ施策方針とCSIRT組織運営のポイント
Recruit Technologies
 
ユーザー企業内製CSIRTにおける対応のポイント
Recruit Technologies
 
リクルートテクノロジーズが語る 企業における、「AI／ディープラーニング」活用のリアル
Recruit Technologies
 
「リクルートデータセット」 ～公開までの道のりとこれから～
Recruit Technologies
 

Recently uploaded (20)

PDF
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
PPTX
A Presentation on Artificial Intelligence
memembruh336
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
PDF
Getting Started with Data Integration: FME Form 101
Safe Software
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PDF
Accuracy of neural networks in brain wave diagnosis of schizophrenia
IAESIJAI
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PPTX
Machine Learning_overview_presentation.pptx
kondavamsidharreddy2
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PPTX
Spectroscopy.pptx food analysis technology
nawahassan45
 
PPTX
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
PPTX
OMC Textile Division Presentation 2021.pptx
JahangirAlam816069
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PPTX
Tartificialntelligence_presentation.pptx
ry7r52mzb4
 
PDF
Encapsulation theory and applications.pdf
gurumoop
 
PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
A Presentation on Artificial Intelligence
memembruh336
 
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
Getting Started with Data Integration: FME Form 101
Safe Software
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
Accuracy of neural networks in brain wave diagnosis of schizophrenia
IAESIJAI
 
Approach and Philosophy of On baking technology
30anthuong17
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
Machine Learning_overview_presentation.pptx
kondavamsidharreddy2
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Spectroscopy.pptx food analysis technology
nawahassan45
 
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
OMC Textile Division Presentation 2021.pptx
JahangirAlam816069
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
Tartificialntelligence_presentation.pptx
ry7r52mzb4
 
Encapsulation theory and applications.pdf
gurumoop
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 

Internal Hive

  • 1. Inside Hive (for beginners)1Takeshi NAKANO / Recruit Co. Ltd.
  • 2. Why?Hive is good tool for non-specialist!The number of M/R controls the Hive processing time.↓How can we reduce the number?What can we do for this on writing HiveQL?↓How does Hive convert HiveQLto M/R jobs?On this, what optimizing processes are adopted?7/6/2011HIVE - A warehouse solution over Map Reduce Framework2
  • 3. Don’t you have..This fb’s paper has a lot of information!But this is a little old..7/6/2011HIVE - A warehouse solution over Map Reduce Framework3
  • 4. Component Level Analysis7/6/2011HIVE - A warehouse solution over Map Reduce Framework4
  • 5. Hive Architecture / Exec Flow7/6/2011HIVE - A warehouse solution over Map Reduce Framework5ClientHadoopMetastoreDriverCompiler
  • 6. ClientHadoopDriverCompilerHive WorkflowHive has the operators which are minimum processing units.The process of each operator is done with HDFS operation or M/R jobs.The compiler converts HiveQL to the sets of operators.7/6/2011HIVE - A warehouse solution over Map Reduce Framework6Metastore
  • 7. Hive WorkflowOperators7/6/2011HIVE - A warehouse solution over Map Reduce Framework7
  • 8. ClientHadoopMetastoreDriverCompilerHive WorkflowFor M/R processing, Hiveuses ExecMaper and ExecReducer.On processing, we have 2 modes.Local processing modeDistributed processing mode7/6/2011HIVE - A warehouse solution over Map Reduce Framework8
  • 9. ClientHadoopMetastoreDriverCompilerHive WorkflowOn 1(Local mode)Hive fork the process with hadoop command.The plan.xml is made just on 1 and the single node processes this.On 2(Distributed mode).Hive send the process to exsistingJobTracker.The information is housed on DistributedCacheand processed on multi nodes.7/6/2011HIVE - A warehouse solution over Map Reduce Framework9
  • 10. Compiler : How to Process HiveQL7/6/2011HIVE - A warehouse solution over Map Reduce Framework10ClientHadoopMetastoreDriverCompiler
  • 11. “Plumbing” of HIVE compiler7/6/201111HIVE - A warehouse solution over Map Reduce Framework
  • 12. “Plumbing” of HIVE compiler7/6/201112HIVE - A warehouse solution over Map Reduce Framework
  • 14. Compiler Overview14HiveQLParserASTSemanticAnalyzerQBLogicalPlan Gen.Operator TreeLogicalOptimizerOperator TreePhysicalPlan Gen.Task TreePhysicalOptimizerTask Tree
  • 15. ParserHiveQLASTINSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono);HiveQLTOK_QUERY + TOK_FROM + TOK_JOIN + TOK_TABREF + TOK_TABNAME + "access_log_hbase" + a + TOK_TABREF + TOK_TABNAME + "product_hbase" + "p" + "=" + "." + TOK_TABLE_OR_COL + "a" + "access_log_hbase" + "." + TOK_TABLE_OR_COL + "p" + "prono“AST + TOK_INSERT + TOK_DESTINATION + TOK_TAB + TOK_TABNAME + "access_log_temp2" + TOK_SELECT + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "a" + "user" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "a" + "prono" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "p" + "maker" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "p" + "price"SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser
  • 16. ParserSQLASTINSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono);SQLTOK_QUERY + TOK_FROM + TOK_JOIN + TOK_TABREF + TOK_TABNAME + "access_log_hbase" + a + TOK_TABREF + TOK_TABNAME + "product_hbase" + "p" + "=" + "." + TOK_TABLE_OR_COL + "a" + "access_log_hbase" + "." + TOK_TABLE_OR_COL + "p" + "prono“ + TOK_INSERT + TOK_DESTINATION + TOK_TAB + TOK_TABNAME + "access_log_temp2" + TOK_SELECT + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "a" + "user" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "a" + "prono" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "p" + "maker" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "p" + "price"AST123SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser
  • 17. 17Semantic Analyzer (1/2)ASTQB+ TOK_FROM + TOK_JOIN + TOK_TABREF + TOK_TABNAME + "access_log_hbase" + a + TOK_TABREF + TOK_TABNAME + "product_hbase" + "p" + "=" + "." + TOK_TABLE_OR_COL + "a" + "access_log_hbase" + "." + TOK_TABLE_OR_COL + "p" + "prono“AST1QBMetaDataAliasTo Table Info“a”=Table Info(“access_log_hbase”)“p”=Table Info(“product_hbase”)ParseInfoJoin Node+ TOK_JOIN + TOK_TABREF … + TOK_TABREF … + “=” …SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser17
  • 18. 18Semantic Analyzer (2/2)ASTQB + TOK_DESTINATION + TOK_TAB + TOK_TABNAME + "access_log_temp2”AST2QBParseInfoNameTo Destination Node+ TOK_TAB + TOK_TABNAME +"access_log_temp2”SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser1818
  • 19. 19Semantic Analyzer (2/2)ASTQB + TOK_SELECT + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "a" + "user" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "a" + "prono" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "p" + "maker" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "p" + "price"ASTQBParseInfo3Name To Select Node+ TOK_SELECT + TOK_SELEXPR … + TOK_SELEXPR … + TOK_SELEXPR … + TOK_SELEXPR …SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser1919
  • 20. 20Logical Plan Generator (1/4)QBOPTreeQBMetaDataAliasTo Table Info“a”=Table Info(“access_log_hbase”)“p”=Table Info(“product_hbase”)OPTreeTableScanOperator(“access_log_hbase”)TableScanOperator(“product_hbase”)SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser2020
  • 21. 21Logical Plan Generator (2/4)QBOPTreeQBParseInfo + TOK_JOIN + TOK_TABREF + TOK_TABNAME + "access_log_hbase" + a + TOK_TABREF + TOK_TABNAME + "product_hbase" + "p" + "=" + "." + TOK_TABLE_OR_COL + "a" + "access_log_hbase" + "." + TOK_TABLE_OR_COL + "p" + "prono“ReduceSinkOperator(“access_log_hbase”)ReduceSinkOperator(“product_hbase”)OPTreeJoinOperatorSemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser
  • 22. 22Logical Plan Generator (3/4)QBOPTreeQBParseInfoName To Select Node+ TOK_SELECT + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "a" + "user" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "a" + "prono" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "p" + "maker" + TOK_SELEXPR + "." + TOK_TABLE_OR_COL + "p" + "price"OPTreeSelectOperatorSemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser
  • 23. 23Logical Plan Generator (4/4)QBOPTreeQBMetaDataName To Destination Table Info“insclause-0”= Table Info(“access_log_temp2”)OPTreeFileSinkOperatorSemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser
  • 24. Logical Plan Generator (result)24LCF OPTreeTableScanOperatorTS_1TableScanOperatorTS_0ReduceSinkOperatorRS_2ReduceSinkOperatorRS_3JoinOperatorJOIN_4SelectOperatorSEL_5FileSinkOperatorFS_6SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser
  • 26. Logical Optimizer (Predicate Push Down)INSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono);INSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono) WHERE p.maker = 'honda';SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser2626
  • 27. Logical Optimizer (Predicate Push Down)TableScanOperatorTS_1TableScanOperatorTS_0INSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono);ReduceSinkOperatorRS_3ReduceSinkOperatorRS_2JoinOperatorJOIN_4INSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono) WHERE p.maker = 'honda';SelectOperatorSEL_6FileSinkOperatorFS_7SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser2727
  • 28. INSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono);INSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono) WHERE p.maker = 'honda';Logical Optimizer (Predicate Push Down)TableScanOperatorTS_1TableScanOperatorTS_0ReduceSinkOperatorRS_3ReduceSinkOperatorRS_2JoinOperatorJOIN_4FilterOperatorFIL_5(_col8 = 'honda')SelectOperatorSEL_6FileSinkOperatorFS_7SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser2828
  • 29. Logical Optimizer (Predicate Push Down)TableScanOperatorTS_1TableScanOperatorTS_0INSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono);FilterOperatorFIL_8(maker = 'honda')ReduceSinkOperatorRS_2ReduceSinkOperatorRS_3JoinOperatorJOIN_4INSERT OVERWRITE TABLE access_log_temp2 SELECT a.user, a.prono, p.maker, p.price FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono) WHERE p.maker = 'honda';FilterOperatorFIL_5(_col8 = 'honda')SelectOperatorSEL_6FileSinkOperatorFS_7SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser2929
  • 31. OPTreeTaskTreeMapRedTask (Stage-1/root)TableScanOperator(TS_0)Physical Plan Generator (result)31LCF MapperTableScanOperatorTS_1TableScanOperatorTS_0TableScanOperator(TS_1)ReduceSinkOperatorRS_2ReduceSinkOperatorRS_3ReduceSinkOperator(RS_2)MapRedTask(Stage-1/root)ReduceSinkOperator(RS_3)ReducerJoinOperatorJOIN_4JoinOperator(JOIN_4)SelectOperatorSEL_5SelectOperator(SEL_5)FileSinkOperatorFS_6SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser313131
  • 33. 33Physical Optimizer (MapJoinResolver)TaskTreeTaskTreeMapRedTask (Stage-1)MapperTableScanOperatorTS_1TableScanOperatorTS_0MapJoinOperatorMAPJOIN_7SelectOperatorSEL_8SelectOperatorSEL_5FileSinkOperatorFS_6SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser33
  • 34. 34Physical Optimizer (MapJoinResolver)TaskTreeTaskTreeMapredLocalTask(Stage-7)MapRedTask (Stage-1)TableScanOperatorTS_0MapperTableScanOperatorTS_1TableScanOperatorTS_0HashTableSinkOperatorHASHTABLESINK_11MapJoinOperatorMAPJOIN_7MapRedTask (Stage-1)SelectOperatorSEL_8MapperTableScanOperatorTS_1SelectOperatorSEL_5MapJoinOperatorMAPJOIN_7FileSinkOperatorFS_6SelectOperatorSEL_8SelectOperatorSEL_5FileSinkOperatorFS_6SemanticAnalyzerLogicalPlan Gen.LogicalOptimizerPhysicalPlan Gen.PhysicalOptimizerParser34
  • 35. In the end7/6/2011HIVE - A warehouse solution over Map Reduce Framework35ClientHadoopMetastoreDriverCompiler
  • 36. In the end36HiveQLParserASTSemanticAnalyzerQBLogicalPlan Gen.Operator TreeLogicalOptimizerOperator TreePhysicalPlan Gen.Task TreePhysicalOptimizerTask Tree
  • 38. Appendix: What does Explain show?7/6/2011HIVE - A warehouse solution over Map Reduce Framework38
  • 39. Appendix: What does Explain show?hive> explain INSERT OVERWRITE TABLE access_log_temp2 > SELECT a.user, a.prono, p.maker, p.price > FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono);OKABSTRACT SYNTAX TREE: (TOK_QUERY (TOK_FROM (TOK_JOIN (TOK_TABREF (TOK_TABNAME access_log_hbase) a) (TOK_TABREF (TOK_TABNAME product_hbase) p) (= (. (TOK_TABLE_OR_COL a) prono) (. (TOK_TABLE_OR_COL p) prono)))) (TOK_INSERT (TOK_DESTINATION (TOK_TAB (TOK_TABNAME access_log_temp2))) (TOK_SELECT (TOK_SELEXPR (. (TOK_TABLE_OR_COL a) user)) (TOK_SELEXPR (. (TOK_TABLE_OR_COL a) prono)) (TOK_SELEXPR (. (TOK_TABLE_OR_COL p) maker)) (TOK_SELEXPR (. (TOK_TABLE_OR_COL p) price)))))STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 Stage-2 depends on stages: Stage-0STAGE PLANS: Stage: Stage-1 Map Reduce Alias -> Map Operator Tree: aTableScan alias: a Reduce Output Operator key expressions:expr: prono type: int sort order: + Map-reduce partition columns:expr: prono type: int tag: 0 value expressions:expr: user type: stringexpr: prono type: int pTableScan alias: p Reduce Output Operator key expressions:expr: prono type: int sort order: + Map-reduce partition columns:expr: prono type: int tag: 1 value expressions:expr: maker type: stringexpr: price type: intReduce Operator Tree: Join Operator condition map: Inner Join 0 to 1 condition expressions: 0 {VALUE._col0} {VALUE._col2} 1 {VALUE._col1} {VALUE._col2}handleSkewJoin: falseoutputColumnNames: _col0, _col2, _col6, _col7 Select Operator expressions:expr: _col0 type: stringexpr: _col2 type: intexpr: _col6 type: stringexpr: _col7 type: intoutputColumnNames: _col0, _col1, _col2, _col3 File Output Operator compressed: falseGlobalTableId: 1 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormatserde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.access_log_temp2 Stage: Stage-0 Move Operator tables: replace: true table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormatserde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.access_log_temp2 Stage: Stage-2 Stats-Aggr OperatorTime taken: 0.1 secondshive>
  • 40. Appendix: What does Explain show?hive> explain INSERT OVERWRITE TABLE access_log_temp2 > SELECT a.user, a.prono, p.maker, p.price > FROM access_log_hbase a JOIN product_hbase p ON (a.prono = p.prono);OKABSTRACT SYNTAX TREE: (TOK_QUERY (TOK_FROM (TOK_JOIN (TOK_TABREF (TOK_TABNAME access_log_hbase) a) (TOK_TABREF (TOK_TABNAME product_hbase) p) (= (. (TOK_TABLE_OR_COL a) prono) (. (TOK_TABLE_OR_COL p) prono)))) (TOK_INSERT (TOK_DESTINATION (TOK_TAB (TOK_TABNAME access_log_temp2))) (TOK_SELECT (TOK_SELEXPR (. (TOK_TABLE_OR_COL a) user)) (TOK_SELEXPR (. (TOK_TABLE_OR_COL a) prono)) (TOK_SELEXPR (. (TOK_TABLE_OR_COL p) maker)) (TOK_SELEXPR (. (TOK_TABLE_OR_COL p) price)))))STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 Stage-2 depends on stages: Stage-0STAGE PLANS: Stage: Stage-1 Map Reduce Alias -> Map Operator Tree: aTableScan alias: aReduce Output Operator key expressions:expr: prono type: int sort order: + Map-reduce partition columns:expr: prono type: int tag: 0 value expressions:expr: user type: stringexpr: prono type: int pTableScan alias: pReduce Output Operator key expressions:expr: prono type: int sort order: + Map-reduce partition columns:expr: prono type: int tag: 1 value expressions:expr: maker type: stringexpr: price type: intABSTRACT SYNTAX TREE:STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 Stage-2 depends on stages: Stage-0STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree:TableScan Reduce Output OperatorTableScan Reduce Output Operator Reduce Operator Tree: Join Operator Select Operator File Output Operator Stage: Stage-0 Move Operator Stage: Stage-2 Stats-Aggr OperatorReduce Operator Tree: Join Operator condition map: Inner Join 0 to 1 condition expressions: 0 {VALUE._col0} {VALUE._col2} 1 {VALUE._col1} {VALUE._col2}handleSkewJoin: falseoutputColumnNames: _col0, _col2, _col6, _col7 Select Operator expressions:expr: _col0 type: stringexpr: _col2 type: intexpr: _col6 type: stringexpr: _col7 type: intoutputColumnNames: _col0, _col1, _col2, _col3File Output Operator compressed: falseGlobalTableId: 1 table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormatserde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.access_log_temp2 Stage: Stage-0 Move Operator tables: replace: true table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormatserde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.access_log_temp2 Stage: Stage-2 Stats-Aggr OperatorTime taken: 0.1 secondshive>
  • 41. Appendix: What does Explain show?ABSTRACT SYNTAX TREE:STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 Stage-2 depends on stages: Stage-0STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree:TableScan Reduce Output OperatorTableScan Reduce Output Operator Reduce Operator Tree: Join Operator Select Operator File Output Operator Stage: Stage-0 Move Operator Stage: Stage-2 Stats-Aggr OperatorMapRedTask (Stage-1/root)MapperTableScanOperatorTS_1TableScanOperatorTS_0ReduceSinkOperatorRS_2ReduceSinkOperatorRS_3ReducerJoinOperatorJOIN_4≒SelectOperatorSEL_5FileSinkOperatorFS_6MoveTask (Stage-0)Stats Task (Stage-2)