SlideShare a Scribd company logo

Three key concepts for java batch

T
timfanelli

The document provides an overview of key concepts for understanding JSR-352, which defines standards for batch applications in Java. It discusses three main concepts: implementation, which provides programming models for developing batch application components; orchestration, which defines a language for organizing the execution of components within a job; and execution, which specifies a runtime environment for executing batch applications. The document uses these concepts to explain the anatomy of a JSR-352 batch application and provides examples of programming models like the chunk model for reading, processing, and writing data in batches.

Technology
1 of 95
Downloaded 34 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
Timothy C. Fanelli - Senior IT Specialist 23 September 2013 Three Key Concepts for Understanding JSR-352: Batch Applications for the Java Platform © 2013 IBM Corporation Sunday, September 22, 13
Important Disclaimers THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE DIFFERENCES. ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE. IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE. IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: - CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS 2 Sunday, September 22, 13 © 2013 IBM Corporation
About me  Timothy C. Fanelli  Senior IT Specialist, IBM - Mainframe Workload Modernization  Instructor of Software Engineering Clarkson University, Potsdam NY  tim@timfanelli.com  tfanelli@us.ibm.com, tfanelli@clarkson.edu  Visit the IBM booth #5112 and meet other IBM developers at JavaOne 2013 3 Sunday, September 22, 13 © 2013 IBM Corporation
Agenda  Background  Three Key Concepts – Implementation – Orchestration – Execution  An Example JSR 352 Application  Advanced Topics – Splits and Flows – Partitioning – Java EE  Conclusion and Thoughts on What’s Next... 4 Sunday, September 22, 13 © 2013 IBM Corporation
Background © 2013 IBM Corporation Sunday, September 22, 13
Batch Processing  One of the oldest processing paradigms  Historically associated with mainframe computing  Still incredibly relevant today, with fresh challenges in an OLTP driven world 6 Sunday, September 22, 13 © 2013 IBM Corporation
Batch Processing  One of the oldest processing paradigms  Historically associated with mainframe computing  Still incredibly relevant today, with fresh challenges in an OLTP driven world 6 Sunday, September 22, 13 © 2013 IBM Corporation
Java for Batch Processing?  Mainframe developers have shied away from Java – Performance concerns over native languages – Integration concerns for legacy data – Disparate developer skill set between System Z and Java 7 Sunday, September 22, 13 © 2013 IBM Corporation
Java for Batch Processing?  Mainframe developers have shied away from Java – Performance concerns over native languages – Integration concerns for legacy data – Disparate developer skill set between System Z and Java  Java and JavaEE have dominated the Online Transaction Processing world 7 Sunday, September 22, 13 © 2013 IBM Corporation
Java for Batch Processing?  Mainframe developers have shied away from Java – Performance concerns over native languages – Integration concerns for legacy data – Disparate developer skill set between System Z and Java  Java and JavaEE have dominated the Online Transaction Processing world  Time to bridge the two worlds together – IBM Java for zOS, IBM WebSphere, and Spring Batch paved new paths – Just-in-Time Compilation, Garbage Collection optimizations proved it out – Adoption is wide-spread! 7 Sunday, September 22, 13 © 2013 IBM Corporation
Java for Batch Processing?  Mainframe developers have shied away from Java – Performance concerns over native languages – Integration concerns for legacy data – Disparate developer skill set between System Z and Java  Java and JavaEE have dominated the Online Transaction Processing world  Time to bridge the two worlds together – IBM Java for zOS, IBM WebSphere, and Spring Batch paved new paths – Just-in-Time Compilation, Garbage Collection optimizations proved it out – Adoption is wide-spread!  Only remaining challenge was the lack of a standard – The need for JSR-352 was obvious 7 Sunday, September 22, 13 © 2013 IBM Corporation
JSR 352: Batch Applications for the Java Platform  Expert working group formed 29 November 2011 – IBM*, VMWare, RedHat, Oracle, Credit Suisse, Independent participants – Broad range of talent with deep batch experience 8 Sunday, September 22, 13 © 2013 IBM Corporation
JSR 352: Batch Applications for the Java Platform  Expert working group formed 29 November 2011 – IBM*, VMWare, RedHat, Oracle, Credit Suisse, Independent participants – Broad range of talent with deep batch experience  Final Release 24 May 2013 8 Sunday, September 22, 13 © 2013 IBM Corporation
JSR 352: Batch Applications for the Java Platform  Expert working group formed 29 November 2011 – IBM*, VMWare, RedHat, Oracle, Credit Suisse, Independent participants – Broad range of talent with deep batch experience  Final Release 24 May 2013  Included in Java EE 7! 8 Sunday, September 22, 13 © 2013 IBM Corporation
Three Key Concepts... © 2013 IBM Corporation Sunday, September 22, 13
Three Key Concepts ...  JSR 352 defines – Implementation: A programming model for implementing the artifacts – Orchestration: A Job Specification Language, which orchestrates the execution of a batch artifacts within a job. – Execution: A runtime environment for executing batch application, according to a defined lifecycle. Orchestrate Implement Execute 10 Sunday, September 22, 13 © 2013 IBM Corporation
Three Key Concepts ...  JSR 352 defines – Implementation: A programming model for implementing the artifacts – Orchestration: A Job Specification Language, which orchestrates the execution of a batch artifacts within a job. – Execution: A runtime environment for executing batch application, according to a defined lifecycle. Orchestrate Implement Execute  Note: “key” concepts, not “new” concepts! – Roles and abstractions should be familiar to SOA and JavaEE developers 10 Sunday, September 22, 13 © 2013 IBM Corporation
Anatomy of JSR352  Those concepts define the anatomy of JSR 352: Batch Applications for the Java Platform... 11 Sunday, September 22, 13 © 2013 IBM Corporation
Anatomy of JSR352  Those concepts define the anatomy of JSR 352: Batch Applications for the Java Platform... Orchestrate Implement Execute 11 Sunday, September 22, 13 © 2013 IBM Corporation
Anatomy of JSR352  Those concepts define the anatomy of JSR 352: Batch Applications for the Java Platform... Listeners Contexts Listeners Partitioning Batchlet Orchestrate Implement Job Operator Job Reader Step Processor Chunk Writer Execute Job Repository Chunk Chunk Chunk Listeners 11 Sunday, September 22, 13 © 2013 IBM Corporation
Implementation: The programming model  Chunk and Batchlet provide models for implementing a step. Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 12 Sunday, September 22, 13 © 2013 IBM Corporation
Implementation: The programming model  Chunk and Batchlet provide models for implementing a step.  Contexts provide Job- and Step- level runtime information, and provide interim data persistence. Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 12 Sunday, September 22, 13 © 2013 IBM Corporation
Implementation: The programming model  Chunk and Batchlet provide models for implementing a step.  Contexts provide Job- and Step- level runtime information, and provide interim data persistence.  Listeners provide callback hooks to respond to lifecycle events on batch artifacts. Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 12 Sunday, September 22, 13 © 2013 IBM Corporation
Implementation: The programming model  Chunk and Batchlet provide models for implementing a step.  Contexts provide Job- and Step- level runtime information, and provide interim data persistence.  Listeners provide callback hooks to respond to lifecycle events on batch artifacts. Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk  Partitioning provides a mechanism imposing parallel processing on jobs and steps Writer Chunk Chunk Chunk Listeners 12 Sunday, September 22, 13 © 2013 IBM Corporation
Implementation: The programming model Chunk vs Batchlet Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 13 Sunday, September 22, 13 © 2013 IBM Corporation
Implementation: The programming model Chunk vs Batchlet  Both are implementations of a step within a batch job Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 13 Sunday, September 22, 13 © 2013 IBM Corporation
Implementation: The programming model Chunk vs Batchlet  Both are implementations of a step within a batch job  The chunk model – Encapsulates a very common pattern: ETL – Single “reader”, “processor” and “writer” – Reader/Processor combination is invoked until an entire “chunk” of data is processed – Output “chunk” is written atomically Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 13 Sunday, September 22, 13 © 2013 IBM Corporation
Implementation: The programming model Chunk vs Batchlet  Both are implementations of a step within a batch job  The chunk model – Encapsulates a very common pattern: ETL – Single “reader”, “processor” and “writer” – Reader/Processor combination is invoked until an entire “chunk” of data is processed – Output “chunk” is written atomically  Batchlet provides a “roll your own” step type – Invoked and runs to completion, producing a return code upon exit. 13 Sunday, September 22, 13 Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners © 2013 IBM Corporation
Orchestration: The Job Specification Language (JSL)  The JSL defines a batch job as an XML document Job 14 Sunday, September 22, 13 Step © 2013 IBM Corporation
Orchestration: The Job Specification Language (JSL)  The JSL defines a batch job as an XML document  Describes a step as an assemblage of batch artifacts 14 Sunday, September 22, 13 Job Step © 2013 IBM Corporation
Orchestration: The Job Specification Language (JSL)  The JSL defines a batch job as an XML document  Describes a step as an assemblage of batch artifacts Job Step  Provides for the description of steps, step groupings, and execution sequencing 14 Sunday, September 22, 13 © 2013 IBM Corporation
Execution: The JobOperator and Repository  JobOperator is the runtime interface for job management, including start, stop, restart and job repository related commands Job Operator Job Repository 15 Sunday, September 22, 13 © 2013 IBM Corporation
Execution: The JobOperator and Repository  JobOperator is the runtime interface for job management, including start, stop, restart and job repository related commands  The Job Repository holds information about completed and executing jobs 15 Sunday, September 22, 13 Job Operator Job Repository © 2013 IBM Corporation
Execution: The JobOperator and Repository  JobOperator is the runtime interface for job management, including start, stop, restart and job repository related commands  The Job Repository holds information about completed and executing jobs Job Operator Job Repository  To start a batch job, get a JobOperator instance use it to start a job described (described by JSL). 15 Sunday, September 22, 13 © 2013 IBM Corporation
Execution: JobInstance, JobExecution, and StepExecution Job Step * JobInstance * JobExecution * * StepExecution Job Operator 16 Sunday, September 22, 13 Job Repository © 2013 IBM Corporation
Execution: JobInstance, JobExecution, and StepExecution  The state of a job is broken down into various parts, and persisted in the repository – Submitting a job creates a JobInstance, a logical representation of a particular “run” of a job. – A JobExecution is a single attempt to run a JobInstance. A restart attempt creates another JobExecution – Similarly, a StepExecution is a single attempt to run a step within a job. It is created when a step starts execution. Job Operator 16 Sunday, September 22, 13 Job Step * JobInstance * JobExecution * * StepExecution Job Repository © 2013 IBM Corporation
An Example JSR 352 Application © 2013 IBM Corporation Sunday, September 22, 13
The Application 18 Sunday, September 22, 13 © 2013 IBM Corporation
The Application ‣ A typical “batch hello world”: 18 Sunday, September 22, 13 © 2013 IBM Corporation
The Application ‣ A typical “batch hello world”: – Reads strings from an input file 18 Sunday, September 22, 13 © 2013 IBM Corporation
The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms 18 Sunday, September 22, 13 © 2013 IBM Corporation
The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file 18 Sunday, September 22, 13 © 2013 IBM Corporation
The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file ‣ Key capabilities 18 Sunday, September 22, 13 © 2013 IBM Corporation
The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file ‣ Key capabilities – If something goes wrong, we don’t want to discard all the prior work; and we want to pick up where we left off 18 Sunday, September 22, 13 © 2013 IBM Corporation
The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file ‣ Key capabilities – If something goes wrong, we don’t want to discard all the prior work; and we want to pick up where we left off – We want control over the transaction scoping so prevent lock contention in high volume periods 18 Sunday, September 22, 13 © 2013 IBM Corporation
The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file ‣ Key capabilities – If something goes wrong, we don’t want to discard all the prior work; and we want to pick up where we left off – We want control over the transaction scoping so prevent lock contention in high volume periods – We want flexibility to “plug and play” where our records come from 18 Sunday, September 22, 13 © 2013 IBM Corporation
The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file ‣ Key capabilities – If something goes wrong, we don’t want to discard all the prior work; and we want to pick up where we left off – We want control over the transaction scoping so prevent lock contention in high volume periods – We want flexibility to “plug and play” where our records come from – For unit testing, development testing, and QA testing records may come from a variety of sources 18 Sunday, September 22, 13 © 2013 IBM Corporation
The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern 19 Sunday, September 22, 13 © 2013 IBM Corporation
The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model 19 Sunday, September 22, 13 © 2013 IBM Corporation
The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model – Encapsulates the ETL pattern components as Reader, Processor, and Writer interfaces 19 Sunday, September 22, 13 © 2013 IBM Corporation
The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model – Encapsulates the ETL pattern components as Reader, Processor, and Writer interfaces – Loosely coupled artifacts will be orchestrated into a single-step job later 19 Sunday, September 22, 13 © 2013 IBM Corporation
The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model – Encapsulates the ETL pattern components as Reader, Processor, and Writer interfaces – Loosely coupled artifacts will be orchestrated into a single-step job later – “Free” checkpoint/restart capability 19 Sunday, September 22, 13 © 2013 IBM Corporation
The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model – Encapsulates the ETL pattern components as Reader, Processor, and Writer interfaces – Loosely coupled artifacts will be orchestrated into a single-step job later – “Free” checkpoint/restart capability – Transaction scoping imposed externally in the job descriptor 19 Sunday, September 22, 13 © 2013 IBM Corporation
The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model – Encapsulates the ETL pattern components as Reader, Processor, and Writer interfaces – Loosely coupled artifacts will be orchestrated into a single-step job later – “Free” checkpoint/restart capability – Transaction scoping imposed externally in the job descriptor ‣ Job will be executed as a Java SE command line batch application 19 Sunday, September 22, 13 © 2013 IBM Corporation
The Code Implement 20 Sunday, September 22, 13 © 2013 IBM Corporation
The Code Implement ‣ An ItemReader encapsulates the data access and deserialization of a record. 20 Sunday, September 22, 13 © 2013 IBM Corporation
The Code Implement ‣ An ItemReader encapsulates the data access and deserialization of a record. ‣ No restriction on data access paradigm: use DAO patterns, JDBC, JPA, Hibernate, Spring Data, etc! 20 Sunday, September 22, 13 © 2013 IBM Corporation
The Code Implement ‣ An ItemReader encapsulates the data access and deserialization of a record. ‣ No restriction on data access paradigm: use DAO patterns, JDBC, JPA, Hibernate, Spring Data, etc! ‣ Checkpoint/Restart data provided as Serializable argument to “open” and from “checkpointInfo” methods. 20 Sunday, September 22, 13 © 2013 IBM Corporation
The Code Implement 21 Sunday, September 22, 13 © 2013 IBM Corporation
The Code Implement  An ItemWriter is the output counterpart to ItemReader 21 Sunday, September 22, 13 © 2013 IBM Corporation
The Code Implement  An ItemWriter is the output counterpart to ItemReader  Primary difference is that writeItems accepts a “chunk” of output objects (as a list) to serialize. 21 Sunday, September 22, 13 © 2013 IBM Corporation
The Code Implement  An ItemWriter is the output counterpart to ItemReader  Primary difference is that writeItems accepts a “chunk” of output objects (as a list) to serialize.  Again, no restriction on data access paradigm! 21 Sunday, September 22, 13 © 2013 IBM Corporation
The Code Implement 22 Sunday, September 22, 13 © 2013 IBM Corporation
The Code Implement  An ItemProcessor encapsulates the business logic applied to each record 22 Sunday, September 22, 13 © 2013 IBM Corporation
The Code Implement  An ItemProcessor encapsulates the business logic applied to each record  “main” here demonstrates the invocation of a batch job, using the JobOperator 22 Sunday, September 22, 13 © 2013 IBM Corporation
The Code Implement  An ItemProcessor encapsulates the business logic applied to each record  “main” here demonstrates the invocation of a batch job, using the JobOperator  Would typically not be in the processor implementation 22 Sunday, September 22, 13 © 2013 IBM Corporation
The Batch Descriptor and Job Specification Orchestrate 23 Sunday, September 22, 13 © 2013 IBM Corporation
The Batch Descriptor and Job Specification Orchestrate  batch.xml defines and names the batch artifacts in this application archive 23 Sunday, September 22, 13 © 2013 IBM Corporation
The Batch Descriptor and Job Specification Orchestrate  batch.xml defines and names the batch artifacts in this application archive  sample.xml is an example Job Specification Language document for SampleBatchApp 23 Sunday, September 22, 13 © 2013 IBM Corporation
The Execution Execute 24 Sunday, September 22, 13 © 2013 IBM Corporation
The Execution Execute  Package the application as a standard JAR or WAR for deployment in JavaSE or EE environments – batch.xml goes in META-INF or WEB-INF/ classes/META-INF – JSL may go in META-INF/batch-jobs, or submitted from an external source (up to the provider!) 24 Sunday, September 22, 13 © 2013 IBM Corporation
The Execution Execute 25 Sunday, September 22, 13 © 2013 IBM Corporation
The Execution  See it live? 25 Sunday, September 22, 13 Execute © 2013 IBM Corporation
Advanced Topics © 2013 IBM Corporation Sunday, September 22, 13
Job Management - Restart, Stop, Abandon Execute 27 Sunday, September 22, 13 © 2013 IBM Corporation
Job Management - Restart, Stop, Abandon  Had something gone wrong, what then? – The “main” program shown was too simple... only “started” the job 27 Sunday, September 22, 13 Execute © 2013 IBM Corporation
Job Management - Restart, Stop, Abandon  Had something gone wrong, what then? – The “main” program shown was too simple... only “started” the job Execute  JobOperator exposes APIs for a variety of job management tasks: start, stop, abandon, restart – Would have had to take advantage of these for advanced job management capabilities. 27 Sunday, September 22, 13 © 2013 IBM Corporation
Job Management - Restart, Stop, Abandon  Had something gone wrong, what then? – The “main” program shown was too simple... only “started” the job Execute  JobOperator exposes APIs for a variety of job management tasks: start, stop, abandon, restart – Would have had to take advantage of these for advanced job management capabilities.  The door is left open for more advanced batch job management systems to be built! – Integration into existing enterprise schedulers? – New Java EE batch scheduling standard? – Plenty of options, but currently left to the provider to implement 27 Sunday, September 22, 13 © 2013 IBM Corporation
Java EE Integration Execute 28 Sunday, September 22, 13 © 2013 IBM Corporation
Java EE Integration  JSR-352: Java Batch is included in Java EE 7 28 Sunday, September 22, 13 Execute © 2013 IBM Corporation
Java EE Integration  JSR-352: Java Batch is included in Java EE 7 Execute  Provides EE clustering, security, resource management, etc to Java Batch applications 28 Sunday, September 22, 13 © 2013 IBM Corporation
Java EE Integration  JSR-352: Java Batch is included in Java EE 7 Execute  Provides EE clustering, security, resource management, etc to Java Batch applications  Performance benefits to dispatching into longrunning, reusable container – JIT compilation through the first couple runs – Eliminates overhead of starting / stopping JVM 28 Sunday, September 22, 13 © 2013 IBM Corporation
Parallel Job Processing Orchestrate 29 Sunday, September 22, 13 © 2013 IBM Corporation
Parallel Job Processing  Splits and Flows provide a mechanism for executing job steps concurrently at the orchestration layer 29 Sunday, September 22, 13 Orchestrate © 2013 IBM Corporation
Parallel Job Processing  Splits and Flows provide a mechanism for executing job steps concurrently at the orchestration layer Orchestrate  A flow is a sequence of one or more steps which execute sequentially, but as a single unit. 29 Sunday, September 22, 13 © 2013 IBM Corporation
Parallel Job Processing  Splits and Flows provide a mechanism for executing job steps concurrently at the orchestration layer Orchestrate  A flow is a sequence of one or more steps which execute sequentially, but as a single unit.  A Split is a collection of flows that may execute concurrently – A split may only contain “flows”; a step is not implicitly a flow 29 Sunday, September 22, 13 © 2013 IBM Corporation
Parallel Job Processing  Splits and Flows provide a mechanism for executing job steps concurrently at the orchestration layer Orchestrate  A flow is a sequence of one or more steps which execute sequentially, but as a single unit.  A Split is a collection of flows that may execute concurrently – A split may only contain “flows”; a step is not implicitly a flow  This is done entirely in the JSL descriptor – Imposed on the batch application with no code changes! 29 Sunday, September 22, 13 © 2013 IBM Corporation
Parallel Job Processing Implement 30 Sunday, September 22, 13 © 2013 IBM Corporation
Parallel Job Processing  Step-level parallelism can be achieved programmatically using step partitioning 30 Sunday, September 22, 13 Implement © 2013 IBM Corporation
Parallel Job Processing  Step-level parallelism can be achieved programmatically using step partitioning Implement  A partitioned step runs as multiple instances with distinct property sets 30 Sunday, September 22, 13 © 2013 IBM Corporation
Parallel Job Processing  Step-level parallelism can be achieved programmatically using step partitioning Implement  A partitioned step runs as multiple instances with distinct property sets  PartitionMapper defines the number of partitions, and property values for each partition – Can be a fixed set of partitions in JSL – Can be dynamic using a PartitionMapper implementation 30 Sunday, September 22, 13 © 2013 IBM Corporation
Parallel Job Processing 31 Sunday, September 22, 13 © 2013 IBM Corporation
Wrap up... © 2013 IBM Corporation Sunday, September 22, 13
Batch Processing  The oldest “new thing” in Java  JSR 352 applies the modern thinking and abstraction of Java EE and SOA and applies it to sequential batch processing  The standardized programming model provides application developers vendor portability  Inclusion in Java EE 7 ensures wide spread availability 33 Sunday, September 22, 13 © 2013 IBM Corporation
Questions? Find this presentation and more! http://ibm.co/JavaOne2013 © 2013 IBM Corporation Sunday, September 22, 13

More Related Content

PPT
Was l iberty for java batch and jsr352
sflynn073
 
PDF
JavaOne2013: Implement a High Level Parallel API - Richard Ning
Chris Bailey
 
PDF
JavaOne BOF 5957 Lightning Fast Access to Big Data
Brian Martin
 
ODP
Bci for Beginners
IainLewis
 
ODP
Windows Debugging Tools - JavaOne 2013
MattKilner
 
PPTX
JSR 236 Concurrency Utils for EE presentation for JavaOne 2013 (CON7948)
Fred Rowe
 
PDF
Inside IBM Java 7
Tim Ellison
 
PDF
Java one 2015 [con3339]
Arshal Ameen
 
Was l iberty for java batch and jsr352
sflynn073
 
JavaOne2013: Implement a High Level Parallel API - Richard Ning
Chris Bailey
 
JavaOne BOF 5957 Lightning Fast Access to Big Data
Brian Martin
 
Bci for Beginners
IainLewis
 
Windows Debugging Tools - JavaOne 2013
MattKilner
 
JSR 236 Concurrency Utils for EE presentation for JavaOne 2013 (CON7948)
Fred Rowe
 
Inside IBM Java 7
Tim Ellison
 
Java one 2015 [con3339]
Arshal Ameen
 

What's hot (20)

PDF
JavaOne2013: Secure Engineering Practices for Java
Chris Bailey
 
PDF
JavaOne 2013 CON7370: Java Interprocess Communication Challenges in Low-Laten...
0xdaryl
 
PDF
What's New in IBM Java 8 SE?
Tim Ellison
 
ODP
Debugging Native heap OOM - JavaOne 2013
MattKilner
 
PPTX
[RakutenTechConf2013] [E-3] Financial Web System with Java EE 6
Rakuten Group, Inc.
 
PDF
SHOW107: The DataSource Session: Take XPages data boldly where no XPages data...
Stephan H. Wissel
 
PDF
AD116 XPages Extension Library: Making Application Development Even Easier
pdhannan
 
PDF
A Java Implementer's Guide to Better Apache Spark Performance
Tim Ellison
 
PDF
Java Enterprise Edition 6 Overview
Eugene Bogaart
 
PPTX
Whats Next for JCA?
Fred Rowe
 
PPTX
Top 50 java ee 7 best practices [con5669]
Ryan Cuprak
 
PPTX
All of the Performance Tuning Features in Oracle SQL Developer
Jeff Smith
 
PPTX
Java vs .Net
Tejasvi Rastogi
 
PDF
Impact2014: Introduction to the IBM Java Tools
Chris Bailey
 
PDF
JavaOne2013: Build Your Own Runtime Monitoring for the IBM JDK with the Healt...
Chris Bailey
 
PPT
JavaScript Frameworks and Java EE – A Great Match
Reza Rahman
 
PPTX
Image Converter
Navin Kumar
 
PDF
JavaOne 2015: From Java Code to Machine Code
Chris Bailey
 
PDF
Hibernate Interview Questions
Syed Shahul
 
PPTX
Future of Java EE with Java SE 8
Hirofumi Iwasaki
 
JavaOne2013: Secure Engineering Practices for Java
Chris Bailey
 
JavaOne 2013 CON7370: Java Interprocess Communication Challenges in Low-Laten...
0xdaryl
 
What's New in IBM Java 8 SE?
Tim Ellison
 
Debugging Native heap OOM - JavaOne 2013
MattKilner
 
[RakutenTechConf2013] [E-3] Financial Web System with Java EE 6
Rakuten Group, Inc.
 
SHOW107: The DataSource Session: Take XPages data boldly where no XPages data...
Stephan H. Wissel
 
AD116 XPages Extension Library: Making Application Development Even Easier
pdhannan
 
A Java Implementer's Guide to Better Apache Spark Performance
Tim Ellison
 
Java Enterprise Edition 6 Overview
Eugene Bogaart
 
Whats Next for JCA?
Fred Rowe
 
Top 50 java ee 7 best practices [con5669]
Ryan Cuprak
 
All of the Performance Tuning Features in Oracle SQL Developer
Jeff Smith
 
Java vs .Net
Tejasvi Rastogi
 
Impact2014: Introduction to the IBM Java Tools
Chris Bailey
 
JavaOne2013: Build Your Own Runtime Monitoring for the IBM JDK with the Healt...
Chris Bailey
 
JavaScript Frameworks and Java EE – A Great Match
Reza Rahman
 
Image Converter
Navin Kumar
 
JavaOne 2015: From Java Code to Machine Code
Chris Bailey
 
Hibernate Interview Questions
Syed Shahul
 
Future of Java EE with Java SE 8
Hirofumi Iwasaki
 
Ad

Viewers also liked (11)

PDF
Spring Batch - Lessons Learned out of a real life banking system.
Raffael Schmid
 
ODP
Developing Microservices using Spring - Beginner's Guide
Mohanraj Thirumoorthy
 
ODP
3978 Why is Java so different... A Session for Cobol/PLI/Assembler Developers
nick_garrod
 
PDF
Spring Batch Performance Tuning
Gunnar Hillert
 
PPT
J2EE Batch Processing
Chris Adkin
 
PDF
Design & Develop Batch Applications in Java/JEE
Naresh Chintalcheru
 
PDF
Three Key Concepts for Understanding JSR-352: Batch Programming for the Java ...
timfanelli
 
PDF
JPA 2.1 performance tuning tips
osa_ora
 
PPTX
Spring batch for large enterprises operations
Ignasi González
 
PDF
Java EE 7 Batch processing in the Real World
Roberto Cortez
 
PPTX
Java EE vs Spring Framework
Rohit Kelapure
 
Spring Batch - Lessons Learned out of a real life banking system.
Raffael Schmid
 
Developing Microservices using Spring - Beginner's Guide
Mohanraj Thirumoorthy
 
3978 Why is Java so different... A Session for Cobol/PLI/Assembler Developers
nick_garrod
 
Spring Batch Performance Tuning
Gunnar Hillert
 
J2EE Batch Processing
Chris Adkin
 
Design & Develop Batch Applications in Java/JEE
Naresh Chintalcheru
 
Three Key Concepts for Understanding JSR-352: Batch Programming for the Java ...
timfanelli
 
JPA 2.1 performance tuning tips
osa_ora
 
Spring batch for large enterprises operations
Ignasi González
 
Java EE 7 Batch processing in the Real World
Roberto Cortez
 
Java EE vs Spring Framework
Rohit Kelapure
 
Ad

Similar to Three key concepts for java batch (20)

PDF
Batch Applications for the Java Platform
Sivakumar Thyagarajan
 
PDF
Batch Applications for Java Platform 1.0: Java EE 7 and GlassFish
Arun Gupta
 
PPTX
Java Batch
Software Infrastructure
 
PPTX
Batching and Java EE (jdk.io)
Ryan Cuprak
 
PPTX
Hadoop vs Java Batch Processing JSR 352
Armel Nene
 
PDF
Spring batch overivew
Chanyeong Choi
 
PPTX
JavaFX and JEE 7
Vijay Nair
 
PPTX
Spring batch
Yukti Kaura
 
PDF
Atlanta JUG - Integrating Spring Batch and Spring Integration
Gunnar Hillert
 
PDF
Presente e Futuro: Java EE.next()
Bruno Borges
 
PPTX
Spring batch
Chandan Kumar Rana
 
PDF
Core Banking System modernization for Japanese Bank
Hirofumi Nakata
 
PPTX
Exploring Data Integration Capabilities of the WSO2 Platform
WSO2
 
PDF
F3-DP-2015-Milata-Tomas-java-ee-batch-editor (1)
Tomáš Milata
 
PDF
AI Industrial Summit - SOFIA, BULGARIA - A High-Speed Data Ingestion Microser...
Juarez Junior
 
PPTX
Java EE8 - by Kito Mann
Kile Niklawski
 
PDF
Join the Java Evolution GIDS Bangalore & Pune
Heather VanCura
 
PPT
Dev212 Comparing Net And Java The View From 2006
kkorovkin
 
PDF
AAI 2236-Using the New Java Concurrency Utilities with IBM WebSphere
Kevin Sutter
 
PDF
AAI-2236 Using the new Java Concurrency Utilities with IBM WebSphere
WASdev Community
 
Batch Applications for the Java Platform
Sivakumar Thyagarajan
 
Batch Applications for Java Platform 1.0: Java EE 7 and GlassFish
Arun Gupta
 
Java Batch
Software Infrastructure
 
Batching and Java EE (jdk.io)
Ryan Cuprak
 
Hadoop vs Java Batch Processing JSR 352
Armel Nene
 
Spring batch overivew
Chanyeong Choi
 
JavaFX and JEE 7
Vijay Nair
 
Spring batch
Yukti Kaura
 
Atlanta JUG - Integrating Spring Batch and Spring Integration
Gunnar Hillert
 
Presente e Futuro: Java EE.next()
Bruno Borges
 
Spring batch
Chandan Kumar Rana
 
Core Banking System modernization for Japanese Bank
Hirofumi Nakata
 
Exploring Data Integration Capabilities of the WSO2 Platform
WSO2
 
F3-DP-2015-Milata-Tomas-java-ee-batch-editor (1)
Tomáš Milata
 
AI Industrial Summit - SOFIA, BULGARIA - A High-Speed Data Ingestion Microser...
Juarez Junior
 
Java EE8 - by Kito Mann
Kile Niklawski
 
Join the Java Evolution GIDS Bangalore & Pune
Heather VanCura
 
Dev212 Comparing Net And Java The View From 2006
kkorovkin
 
AAI 2236-Using the New Java Concurrency Utilities with IBM WebSphere
Kevin Sutter
 
AAI-2236 Using the new Java Concurrency Utilities with IBM WebSphere
WASdev Community
 

Recently uploaded (20)

PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
PDF
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
PPTX
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
PPTX
1. Introduction to Computer Programming.pptx
bonakiduza1024
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PPTX
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
PPTX
SOPHOS-XG Firewall Administrator PPT.pptx
AgnesMunisi
 
PDF
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
IAESIJAI
 
PDF
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PDF
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
Approach and Philosophy of On baking technology
30anthuong17
 
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
1. Introduction to Computer Programming.pptx
bonakiduza1024
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
SOPHOS-XG Firewall Administrator PPT.pptx
AgnesMunisi
 
Video forgery: An extensive analysis of inter-and intra-frame manipulation al...
IAESIJAI
 
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 

Three key concepts for java batch

  • 1. Timothy C. Fanelli - Senior IT Specialist 23 September 2013 Three Key Concepts for Understanding JSR-352: Batch Applications for the Java Platform © 2013 IBM Corporation Sunday, September 22, 13
  • 2. Important Disclaimers THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE DIFFERENCES. ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE. IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE. IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: - CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS 2 Sunday, September 22, 13 © 2013 IBM Corporation
  • 3. About me  Timothy C. Fanelli  Senior IT Specialist, IBM - Mainframe Workload Modernization  Instructor of Software Engineering Clarkson University, Potsdam NY  [email protected][email protected], [email protected]  Visit the IBM booth #5112 and meet other IBM developers at JavaOne 2013 3 Sunday, September 22, 13 © 2013 IBM Corporation
  • 4. Agenda  Background  Three Key Concepts – Implementation – Orchestration – Execution  An Example JSR 352 Application  Advanced Topics – Splits and Flows – Partitioning – Java EE  Conclusion and Thoughts on What’s Next... 4 Sunday, September 22, 13 © 2013 IBM Corporation
  • 5. Background © 2013 IBM Corporation Sunday, September 22, 13
  • 6. Batch Processing  One of the oldest processing paradigms  Historically associated with mainframe computing  Still incredibly relevant today, with fresh challenges in an OLTP driven world 6 Sunday, September 22, 13 © 2013 IBM Corporation
  • 7. Batch Processing  One of the oldest processing paradigms  Historically associated with mainframe computing  Still incredibly relevant today, with fresh challenges in an OLTP driven world 6 Sunday, September 22, 13 © 2013 IBM Corporation
  • 8. Java for Batch Processing?  Mainframe developers have shied away from Java – Performance concerns over native languages – Integration concerns for legacy data – Disparate developer skill set between System Z and Java 7 Sunday, September 22, 13 © 2013 IBM Corporation
  • 9. Java for Batch Processing?  Mainframe developers have shied away from Java – Performance concerns over native languages – Integration concerns for legacy data – Disparate developer skill set between System Z and Java  Java and JavaEE have dominated the Online Transaction Processing world 7 Sunday, September 22, 13 © 2013 IBM Corporation
  • 10. Java for Batch Processing?  Mainframe developers have shied away from Java – Performance concerns over native languages – Integration concerns for legacy data – Disparate developer skill set between System Z and Java  Java and JavaEE have dominated the Online Transaction Processing world  Time to bridge the two worlds together – IBM Java for zOS, IBM WebSphere, and Spring Batch paved new paths – Just-in-Time Compilation, Garbage Collection optimizations proved it out – Adoption is wide-spread! 7 Sunday, September 22, 13 © 2013 IBM Corporation
  • 11. Java for Batch Processing?  Mainframe developers have shied away from Java – Performance concerns over native languages – Integration concerns for legacy data – Disparate developer skill set between System Z and Java  Java and JavaEE have dominated the Online Transaction Processing world  Time to bridge the two worlds together – IBM Java for zOS, IBM WebSphere, and Spring Batch paved new paths – Just-in-Time Compilation, Garbage Collection optimizations proved it out – Adoption is wide-spread!  Only remaining challenge was the lack of a standard – The need for JSR-352 was obvious 7 Sunday, September 22, 13 © 2013 IBM Corporation
  • 12. JSR 352: Batch Applications for the Java Platform  Expert working group formed 29 November 2011 – IBM*, VMWare, RedHat, Oracle, Credit Suisse, Independent participants – Broad range of talent with deep batch experience 8 Sunday, September 22, 13 © 2013 IBM Corporation
  • 13. JSR 352: Batch Applications for the Java Platform  Expert working group formed 29 November 2011 – IBM*, VMWare, RedHat, Oracle, Credit Suisse, Independent participants – Broad range of talent with deep batch experience  Final Release 24 May 2013 8 Sunday, September 22, 13 © 2013 IBM Corporation
  • 14. JSR 352: Batch Applications for the Java Platform  Expert working group formed 29 November 2011 – IBM*, VMWare, RedHat, Oracle, Credit Suisse, Independent participants – Broad range of talent with deep batch experience  Final Release 24 May 2013  Included in Java EE 7! 8 Sunday, September 22, 13 © 2013 IBM Corporation
  • 15. Three Key Concepts... © 2013 IBM Corporation Sunday, September 22, 13
  • 16. Three Key Concepts ...  JSR 352 defines – Implementation: A programming model for implementing the artifacts – Orchestration: A Job Specification Language, which orchestrates the execution of a batch artifacts within a job. – Execution: A runtime environment for executing batch application, according to a defined lifecycle. Orchestrate Implement Execute 10 Sunday, September 22, 13 © 2013 IBM Corporation
  • 17. Three Key Concepts ...  JSR 352 defines – Implementation: A programming model for implementing the artifacts – Orchestration: A Job Specification Language, which orchestrates the execution of a batch artifacts within a job. – Execution: A runtime environment for executing batch application, according to a defined lifecycle. Orchestrate Implement Execute  Note: “key” concepts, not “new” concepts! – Roles and abstractions should be familiar to SOA and JavaEE developers 10 Sunday, September 22, 13 © 2013 IBM Corporation
  • 18. Anatomy of JSR352  Those concepts define the anatomy of JSR 352: Batch Applications for the Java Platform... 11 Sunday, September 22, 13 © 2013 IBM Corporation
  • 19. Anatomy of JSR352  Those concepts define the anatomy of JSR 352: Batch Applications for the Java Platform... Orchestrate Implement Execute 11 Sunday, September 22, 13 © 2013 IBM Corporation
  • 20. Anatomy of JSR352  Those concepts define the anatomy of JSR 352: Batch Applications for the Java Platform... Listeners Contexts Listeners Partitioning Batchlet Orchestrate Implement Job Operator Job Reader Step Processor Chunk Writer Execute Job Repository Chunk Chunk Chunk Listeners 11 Sunday, September 22, 13 © 2013 IBM Corporation
  • 21. Implementation: The programming model  Chunk and Batchlet provide models for implementing a step. Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 12 Sunday, September 22, 13 © 2013 IBM Corporation
  • 22. Implementation: The programming model  Chunk and Batchlet provide models for implementing a step.  Contexts provide Job- and Step- level runtime information, and provide interim data persistence. Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 12 Sunday, September 22, 13 © 2013 IBM Corporation
  • 23. Implementation: The programming model  Chunk and Batchlet provide models for implementing a step.  Contexts provide Job- and Step- level runtime information, and provide interim data persistence.  Listeners provide callback hooks to respond to lifecycle events on batch artifacts. Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 12 Sunday, September 22, 13 © 2013 IBM Corporation
  • 24. Implementation: The programming model  Chunk and Batchlet provide models for implementing a step.  Contexts provide Job- and Step- level runtime information, and provide interim data persistence.  Listeners provide callback hooks to respond to lifecycle events on batch artifacts. Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk  Partitioning provides a mechanism imposing parallel processing on jobs and steps Writer Chunk Chunk Chunk Listeners 12 Sunday, September 22, 13 © 2013 IBM Corporation
  • 25. Implementation: The programming model Chunk vs Batchlet Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 13 Sunday, September 22, 13 © 2013 IBM Corporation
  • 26. Implementation: The programming model Chunk vs Batchlet  Both are implementations of a step within a batch job Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 13 Sunday, September 22, 13 © 2013 IBM Corporation
  • 27. Implementation: The programming model Chunk vs Batchlet  Both are implementations of a step within a batch job  The chunk model – Encapsulates a very common pattern: ETL – Single “reader”, “processor” and “writer” – Reader/Processor combination is invoked until an entire “chunk” of data is processed – Output “chunk” is written atomically Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 13 Sunday, September 22, 13 © 2013 IBM Corporation
  • 28. Implementation: The programming model Chunk vs Batchlet  Both are implementations of a step within a batch job  The chunk model – Encapsulates a very common pattern: ETL – Single “reader”, “processor” and “writer” – Reader/Processor combination is invoked until an entire “chunk” of data is processed – Output “chunk” is written atomically  Batchlet provides a “roll your own” step type – Invoked and runs to completion, producing a return code upon exit. 13 Sunday, September 22, 13 Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners © 2013 IBM Corporation
  • 29. Orchestration: The Job Specification Language (JSL)  The JSL defines a batch job as an XML document Job 14 Sunday, September 22, 13 Step © 2013 IBM Corporation
  • 30. Orchestration: The Job Specification Language (JSL)  The JSL defines a batch job as an XML document  Describes a step as an assemblage of batch artifacts 14 Sunday, September 22, 13 Job Step © 2013 IBM Corporation
  • 31. Orchestration: The Job Specification Language (JSL)  The JSL defines a batch job as an XML document  Describes a step as an assemblage of batch artifacts Job Step  Provides for the description of steps, step groupings, and execution sequencing 14 Sunday, September 22, 13 © 2013 IBM Corporation
  • 32. Execution: The JobOperator and Repository  JobOperator is the runtime interface for job management, including start, stop, restart and job repository related commands Job Operator Job Repository 15 Sunday, September 22, 13 © 2013 IBM Corporation
  • 33. Execution: The JobOperator and Repository  JobOperator is the runtime interface for job management, including start, stop, restart and job repository related commands  The Job Repository holds information about completed and executing jobs 15 Sunday, September 22, 13 Job Operator Job Repository © 2013 IBM Corporation
  • 34. Execution: The JobOperator and Repository  JobOperator is the runtime interface for job management, including start, stop, restart and job repository related commands  The Job Repository holds information about completed and executing jobs Job Operator Job Repository  To start a batch job, get a JobOperator instance use it to start a job described (described by JSL). 15 Sunday, September 22, 13 © 2013 IBM Corporation
  • 35. Execution: JobInstance, JobExecution, and StepExecution Job Step * JobInstance * JobExecution * * StepExecution Job Operator 16 Sunday, September 22, 13 Job Repository © 2013 IBM Corporation
  • 36. Execution: JobInstance, JobExecution, and StepExecution  The state of a job is broken down into various parts, and persisted in the repository – Submitting a job creates a JobInstance, a logical representation of a particular “run” of a job. – A JobExecution is a single attempt to run a JobInstance. A restart attempt creates another JobExecution – Similarly, a StepExecution is a single attempt to run a step within a job. It is created when a step starts execution. Job Operator 16 Sunday, September 22, 13 Job Step * JobInstance * JobExecution * * StepExecution Job Repository © 2013 IBM Corporation
  • 37. An Example JSR 352 Application © 2013 IBM Corporation Sunday, September 22, 13
  • 38. The Application 18 Sunday, September 22, 13 © 2013 IBM Corporation
  • 39. The Application ‣ A typical “batch hello world”: 18 Sunday, September 22, 13 © 2013 IBM Corporation
  • 40. The Application ‣ A typical “batch hello world”: – Reads strings from an input file 18 Sunday, September 22, 13 © 2013 IBM Corporation
  • 41. The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms 18 Sunday, September 22, 13 © 2013 IBM Corporation
  • 42. The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file 18 Sunday, September 22, 13 © 2013 IBM Corporation
  • 43. The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file ‣ Key capabilities 18 Sunday, September 22, 13 © 2013 IBM Corporation
  • 44. The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file ‣ Key capabilities – If something goes wrong, we don’t want to discard all the prior work; and we want to pick up where we left off 18 Sunday, September 22, 13 © 2013 IBM Corporation
  • 45. The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file ‣ Key capabilities – If something goes wrong, we don’t want to discard all the prior work; and we want to pick up where we left off – We want control over the transaction scoping so prevent lock contention in high volume periods 18 Sunday, September 22, 13 © 2013 IBM Corporation
  • 46. The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file ‣ Key capabilities – If something goes wrong, we don’t want to discard all the prior work; and we want to pick up where we left off – We want control over the transaction scoping so prevent lock contention in high volume periods – We want flexibility to “plug and play” where our records come from 18 Sunday, September 22, 13 © 2013 IBM Corporation
  • 47. The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file ‣ Key capabilities – If something goes wrong, we don’t want to discard all the prior work; and we want to pick up where we left off – We want control over the transaction scoping so prevent lock contention in high volume periods – We want flexibility to “plug and play” where our records come from – For unit testing, development testing, and QA testing records may come from a variety of sources 18 Sunday, September 22, 13 © 2013 IBM Corporation
  • 48. The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern 19 Sunday, September 22, 13 © 2013 IBM Corporation
  • 49. The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model 19 Sunday, September 22, 13 © 2013 IBM Corporation
  • 50. The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model – Encapsulates the ETL pattern components as Reader, Processor, and Writer interfaces 19 Sunday, September 22, 13 © 2013 IBM Corporation
  • 51. The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model – Encapsulates the ETL pattern components as Reader, Processor, and Writer interfaces – Loosely coupled artifacts will be orchestrated into a single-step job later 19 Sunday, September 22, 13 © 2013 IBM Corporation
  • 52. The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model – Encapsulates the ETL pattern components as Reader, Processor, and Writer interfaces – Loosely coupled artifacts will be orchestrated into a single-step job later – “Free” checkpoint/restart capability 19 Sunday, September 22, 13 © 2013 IBM Corporation
  • 53. The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model – Encapsulates the ETL pattern components as Reader, Processor, and Writer interfaces – Loosely coupled artifacts will be orchestrated into a single-step job later – “Free” checkpoint/restart capability – Transaction scoping imposed externally in the job descriptor 19 Sunday, September 22, 13 © 2013 IBM Corporation
  • 54. The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model – Encapsulates the ETL pattern components as Reader, Processor, and Writer interfaces – Loosely coupled artifacts will be orchestrated into a single-step job later – “Free” checkpoint/restart capability – Transaction scoping imposed externally in the job descriptor ‣ Job will be executed as a Java SE command line batch application 19 Sunday, September 22, 13 © 2013 IBM Corporation
  • 55. The Code Implement 20 Sunday, September 22, 13 © 2013 IBM Corporation
  • 56. The Code Implement ‣ An ItemReader encapsulates the data access and deserialization of a record. 20 Sunday, September 22, 13 © 2013 IBM Corporation
  • 57. The Code Implement ‣ An ItemReader encapsulates the data access and deserialization of a record. ‣ No restriction on data access paradigm: use DAO patterns, JDBC, JPA, Hibernate, Spring Data, etc! 20 Sunday, September 22, 13 © 2013 IBM Corporation
  • 58. The Code Implement ‣ An ItemReader encapsulates the data access and deserialization of a record. ‣ No restriction on data access paradigm: use DAO patterns, JDBC, JPA, Hibernate, Spring Data, etc! ‣ Checkpoint/Restart data provided as Serializable argument to “open” and from “checkpointInfo” methods. 20 Sunday, September 22, 13 © 2013 IBM Corporation
  • 59. The Code Implement 21 Sunday, September 22, 13 © 2013 IBM Corporation
  • 60. The Code Implement  An ItemWriter is the output counterpart to ItemReader 21 Sunday, September 22, 13 © 2013 IBM Corporation
  • 61. The Code Implement  An ItemWriter is the output counterpart to ItemReader  Primary difference is that writeItems accepts a “chunk” of output objects (as a list) to serialize. 21 Sunday, September 22, 13 © 2013 IBM Corporation
  • 62. The Code Implement  An ItemWriter is the output counterpart to ItemReader  Primary difference is that writeItems accepts a “chunk” of output objects (as a list) to serialize.  Again, no restriction on data access paradigm! 21 Sunday, September 22, 13 © 2013 IBM Corporation
  • 63. The Code Implement 22 Sunday, September 22, 13 © 2013 IBM Corporation
  • 64. The Code Implement  An ItemProcessor encapsulates the business logic applied to each record 22 Sunday, September 22, 13 © 2013 IBM Corporation
  • 65. The Code Implement  An ItemProcessor encapsulates the business logic applied to each record  “main” here demonstrates the invocation of a batch job, using the JobOperator 22 Sunday, September 22, 13 © 2013 IBM Corporation
  • 66. The Code Implement  An ItemProcessor encapsulates the business logic applied to each record  “main” here demonstrates the invocation of a batch job, using the JobOperator  Would typically not be in the processor implementation 22 Sunday, September 22, 13 © 2013 IBM Corporation
  • 67. The Batch Descriptor and Job Specification Orchestrate 23 Sunday, September 22, 13 © 2013 IBM Corporation
  • 68. The Batch Descriptor and Job Specification Orchestrate  batch.xml defines and names the batch artifacts in this application archive 23 Sunday, September 22, 13 © 2013 IBM Corporation
  • 69. The Batch Descriptor and Job Specification Orchestrate  batch.xml defines and names the batch artifacts in this application archive  sample.xml is an example Job Specification Language document for SampleBatchApp 23 Sunday, September 22, 13 © 2013 IBM Corporation
  • 70. The Execution Execute 24 Sunday, September 22, 13 © 2013 IBM Corporation
  • 71. The Execution Execute  Package the application as a standard JAR or WAR for deployment in JavaSE or EE environments – batch.xml goes in META-INF or WEB-INF/ classes/META-INF – JSL may go in META-INF/batch-jobs, or submitted from an external source (up to the provider!) 24 Sunday, September 22, 13 © 2013 IBM Corporation
  • 72. The Execution Execute 25 Sunday, September 22, 13 © 2013 IBM Corporation
  • 73. The Execution  See it live? 25 Sunday, September 22, 13 Execute © 2013 IBM Corporation
  • 74. Advanced Topics © 2013 IBM Corporation Sunday, September 22, 13
  • 75. Job Management - Restart, Stop, Abandon Execute 27 Sunday, September 22, 13 © 2013 IBM Corporation
  • 76. Job Management - Restart, Stop, Abandon  Had something gone wrong, what then? – The “main” program shown was too simple... only “started” the job 27 Sunday, September 22, 13 Execute © 2013 IBM Corporation
  • 77. Job Management - Restart, Stop, Abandon  Had something gone wrong, what then? – The “main” program shown was too simple... only “started” the job Execute  JobOperator exposes APIs for a variety of job management tasks: start, stop, abandon, restart – Would have had to take advantage of these for advanced job management capabilities. 27 Sunday, September 22, 13 © 2013 IBM Corporation
  • 78. Job Management - Restart, Stop, Abandon  Had something gone wrong, what then? – The “main” program shown was too simple... only “started” the job Execute  JobOperator exposes APIs for a variety of job management tasks: start, stop, abandon, restart – Would have had to take advantage of these for advanced job management capabilities.  The door is left open for more advanced batch job management systems to be built! – Integration into existing enterprise schedulers? – New Java EE batch scheduling standard? – Plenty of options, but currently left to the provider to implement 27 Sunday, September 22, 13 © 2013 IBM Corporation
  • 79. Java EE Integration Execute 28 Sunday, September 22, 13 © 2013 IBM Corporation
  • 80. Java EE Integration  JSR-352: Java Batch is included in Java EE 7 28 Sunday, September 22, 13 Execute © 2013 IBM Corporation
  • 81. Java EE Integration  JSR-352: Java Batch is included in Java EE 7 Execute  Provides EE clustering, security, resource management, etc to Java Batch applications 28 Sunday, September 22, 13 © 2013 IBM Corporation
  • 82. Java EE Integration  JSR-352: Java Batch is included in Java EE 7 Execute  Provides EE clustering, security, resource management, etc to Java Batch applications  Performance benefits to dispatching into longrunning, reusable container – JIT compilation through the first couple runs – Eliminates overhead of starting / stopping JVM 28 Sunday, September 22, 13 © 2013 IBM Corporation
  • 83. Parallel Job Processing Orchestrate 29 Sunday, September 22, 13 © 2013 IBM Corporation
  • 84. Parallel Job Processing  Splits and Flows provide a mechanism for executing job steps concurrently at the orchestration layer 29 Sunday, September 22, 13 Orchestrate © 2013 IBM Corporation
  • 85. Parallel Job Processing  Splits and Flows provide a mechanism for executing job steps concurrently at the orchestration layer Orchestrate  A flow is a sequence of one or more steps which execute sequentially, but as a single unit. 29 Sunday, September 22, 13 © 2013 IBM Corporation
  • 86. Parallel Job Processing  Splits and Flows provide a mechanism for executing job steps concurrently at the orchestration layer Orchestrate  A flow is a sequence of one or more steps which execute sequentially, but as a single unit.  A Split is a collection of flows that may execute concurrently – A split may only contain “flows”; a step is not implicitly a flow 29 Sunday, September 22, 13 © 2013 IBM Corporation
  • 87. Parallel Job Processing  Splits and Flows provide a mechanism for executing job steps concurrently at the orchestration layer Orchestrate  A flow is a sequence of one or more steps which execute sequentially, but as a single unit.  A Split is a collection of flows that may execute concurrently – A split may only contain “flows”; a step is not implicitly a flow  This is done entirely in the JSL descriptor – Imposed on the batch application with no code changes! 29 Sunday, September 22, 13 © 2013 IBM Corporation
  • 88. Parallel Job Processing Implement 30 Sunday, September 22, 13 © 2013 IBM Corporation
  • 89. Parallel Job Processing  Step-level parallelism can be achieved programmatically using step partitioning 30 Sunday, September 22, 13 Implement © 2013 IBM Corporation
  • 90. Parallel Job Processing  Step-level parallelism can be achieved programmatically using step partitioning Implement  A partitioned step runs as multiple instances with distinct property sets 30 Sunday, September 22, 13 © 2013 IBM Corporation
  • 91. Parallel Job Processing  Step-level parallelism can be achieved programmatically using step partitioning Implement  A partitioned step runs as multiple instances with distinct property sets  PartitionMapper defines the number of partitions, and property values for each partition – Can be a fixed set of partitions in JSL – Can be dynamic using a PartitionMapper implementation 30 Sunday, September 22, 13 © 2013 IBM Corporation
  • 92. Parallel Job Processing 31 Sunday, September 22, 13 © 2013 IBM Corporation
  • 93. Wrap up... © 2013 IBM Corporation Sunday, September 22, 13
  • 94. Batch Processing  The oldest “new thing” in Java  JSR 352 applies the modern thinking and abstraction of Java EE and SOA and applies it to sequential batch processing  The standardized programming model provides application developers vendor portability  Inclusion in Java EE 7 ensures wide spread availability 33 Sunday, September 22, 13 © 2013 IBM Corporation
  • 95. Questions? Find this presentation and more! http://ibm.co/JavaOne2013 © 2013 IBM Corporation Sunday, September 22, 13