SlideShare a Scribd company logo

Data Microservices with Spring Cloud Stream, Task, and Data Flow #jsug #springday

Toshiaki Maki, profile picture
Toshiaki Maki

The document discusses Spring Cloud Data Flow, which provides a framework for building data pipelines using microservices. It describes how Spring Cloud Data Flow orchestrates long-lived streaming applications using Spring Cloud Stream and short-lived batch processing applications using Spring Cloud Task. It also discusses how these applications can be deployed on modern platforms using deployment-specific Spring Cloud Deployers.

Technology
1 of 117
Downloaded 263 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
‹#›© 2016 Pivotal Software, Inc. All rights reserved. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices with Spring Cloud Stream, Task, and Data Flow Toshiaki Maki (@making) 2016-11-18 Spring Day 2016 #springday #jsug
© 2016 Pivotal Software, Inc. All rights reserved. Who am I ? • Toshiaki Maki (@making) http://blog.ik.am • Sr. Solutions Architect @Pivotal • Spring Framework enthusiast bit.ly/hajiboot2
© 2016 Pivotal Software, Inc. All rights reserved. Input Motivation HTTP TCP S3 JDBC Rabbit JMS Twitter Syslog Output HDFS Cassandra HAWQ Greenplum JDBC S3 TCP Gemfire ❓❔❓ • Real Time Analysis • Data Ingestion • ETL Process
© 2016 Pivotal Software, Inc. All rights reserved. Spring XD (eXtreme Data)
© 2016 Pivotal Software, Inc. All rights reserved. Spring XD (eXtreme Data) 🤔
© 2016 Pivotal Software, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs
© 2016 Pivotal Software, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs 😇
© 2016 Pivotal Software, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs
© 2016 Pivotal Software, Inc. All rights reserved. Modern Platforms Cloud Foundry YARN Kubernetes Mesos Docker Swarm Nomad OpenShift
© 2016 Pivotal Software, Inc. All rights reserved. Cloud Native Redesign •From multiple modules embedded in a container to standalone executable applications •From our own runtime to delegating to existing modern platforms
© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing
© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing
© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers
© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers
© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications
© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream
© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream Spring Cloud Task
© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream Spring Cloud Task Orchestration Layer
© 2016 Pivotal Software, Inc. All rights reserved. Structure Spring Cloud Data Flow Spring Cloud Deployer (SPI) Spring Cloud Stream Spring Cloud Task Spring Integration Spring Boot Spring Batch
© 2016 Pivotal Software, Inc. All rights reserved. Structure Spring Cloud Data Flow Spring Cloud Deployer (SPI) Spring Cloud Stream Spring Cloud Task Spring Integration Spring Boot Spring Batch • Spring Cloud Deployer Local • Spring Cloud Deployer Cloud Foundry • Spring Cloud Deployer Yarn • Spring Cloud Deployer Kubernetes • Spring Cloud Deployer Mesos
© 2016 Pivotal Software, Inc. All rights reserved. SCDF Deployment Platform Spring Cloud Data Flow Server Deployer SPI REST API SCDFShell SCDF Flo
© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow Architecture gpfdist cassandra jms http stream1 = http | cassandra stream2 = jms | gpfdist Message Broker Data Flow Server DB Platform Runtime
© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow Architecture gpfdist cassandra jms http stream1 = http | cassandra stream2 = jms | gpfdist Message Broker Data Flow Server DB Platform Runtime
‹#›© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Stream
© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Stream • Event-driven microservice framework • Built on battle-tested components (Spring Boot / Spring Integration) • Opinionated primitives for streaming applications • Persistent Pub/Sub • Consumer Groups • Partitioning Support • Pluggable messaging middleware bindings source | processor | sink
© 2016 Pivotal Software, Inc. All rights reserved. JJUG CCC in 2 weeks
© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Stream Applications Twitter Stream Cassandra java -jar twittersource.jar --server.port=8080 --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings. output.destination=ingest Source Sink java -jar cassandrasink.jar --server.port=8081 --spring.cassandra.keyspace=tweet --spring.cloud.stream.bindings. input.destination=ingest
© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Stream Applications Twitter Stream Cassandra java -jar twittersource.jar --server.port=8080 --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings. output.destination=ingest Source Sink java -jar cassandrasink.jar --server.port=8081 --spring.cassandra.keyspace=tweet --spring.cloud.stream.bindings. input.destination=ingest Twitter Stream Cassandraingest
© 2016 Pivotal Software, Inc. All rights reserved. Message Binders • @EnableBinding • Binder Implementations • Production-Ready • Rabbit MQ • Apache Kafka • Experimental • JMS • Google PubSub
© 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Sink) @SpringBootApplication
 @EnableBinding(Sink.class)
 public class DemoSinkApp { @StreamListener(Sink.INPUT)
 void receive(Message<String> message) {
 System.out.println("Received " + message);
 }
 public static void main(String[] args) {
 SpringApplication.run(DemoSinkApp.class, args);
 }
 }
© 2016 Pivotal Software, Inc. All rights reserved. Sink Properties (Sink) spring.cloud.stream.bindings.input.destination=demo-strm demo- strm input
© 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Source) @SpringBootApplication @RestController @EnableBinding(Source.class)
 public class DemoSourceApp {
 @Autowired Source source; @GetMapping void send(@RequestParam String text) {
 source.output() .send(MessageBuilder.withPayload(text).build());
 }
 public static void main(String[] args) {
 SpringApplication.run(DemoSourceApp.class, args);
 }
 }
© 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Source) @SpringBootApplication @RestController @EnableBinding(Source.class)
 public class DemoSourceApp { @Bean @InboundChannelAdapter(channel = Source.OUTPUT, poller = @Poller(fixedDelay = "1000", maxMessagesPerPoll = "1")) MessageSource<String> source() {
 return () -> MessageBuilder.withPayload("Hi").build());
 }
 public static void main(String[] args) {
 SpringApplication.run(DemoSourceApp.class, args);
 }}
© 2016 Pivotal Software, Inc. All rights reserved. Properties (Source) spring.cloud.stream.bindings.output.destination=demo-strm demo- strm Source output
© 2016 Pivotal Software, Inc. All rights reserved. Binder (RabbitMQ) <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-rabbit</artifactId> </dependency> demo- strm
© 2016 Pivotal Software, Inc. All rights reserved. Binder (Apache Kafka) <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-kafka</artifactId> </dependency> demo- strm
© 2016 Pivotal Software, Inc. All rights reserved. Sink Pipeline demo- strm input Source output source | sink
© 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Processor) @SpringBootApplication
 @EnableBinding(Processor.class)
 public class DemoProcessorApp { @StreamListener(Processor.INPUT) @SendTo(Processor.OUTPUT)
 void receive(String text) {
 return "[[" + text + "]]";
 }
 public static void main(String[] args) {
 SpringApplication.run(DemoProcessorApp.class, args);
 }
 }
© 2016 Pivotal Software, Inc. All rights reserved. Properties (Processor) spring.cloud.stream.bindings.output.destination=my-source spring.cloud.stream.bindings.input.destination=my-source spring.cloud.stream.bindings.output.destination=my-proc spring.cloud.stream.bindings.input.destination=my-proc Source Processor Sink
© 2016 Pivotal Software, Inc. All rights reserved. Pipeline my- source Source output source | processor | sink Processor output input my-proc Sink input
© 2016 Pivotal Software, Inc. All rights reserved. Reactive API Support @SpringBootApplication
 @EnableBinding(Processor.class)
 public class DemoProcessorRxApp { @StreamListener @Output(Processor.OUTPUT)
 public Flux<String> receive(@Input(Processor.INPUT) Flux<String> stream) {
 return stream.map(text -> "[[" + text + "]]");
 }
 public static void main(String[] args) {
 SpringApplication.run(DemoProcessorRxApp.class, args);
 }
 }
© 2016 Pivotal Software, Inc. All rights reserved. Reactive API Support @StreamListener @Output(Processor.OUTPUT)
 public Flux<AverageData> receive(@Input(Processor.INPUT) Flux<SensorData> stream) {
 return stream.window(Duration.ofSecond(20), Duration.ofSecond(10)) .flatMap(win -> win.groupBy(sensor -> sensor.id)) .flatMap(group -> calcAverage(group));
 }
© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Stream App Starters Source Processor Sink file ftp gemfire gemfire-cq http jdbc jms load-generator loggregator mail mongodb rabbit s3 sftp syslog tcp tcp-client time trigger triggertask twitterstream bridge filter groovy-filter groovy- transform httpclient pommel scriptable- transform splitter tcp-client transform aggregate- counter cassandra counter field-value- counter file ftp gemfire gpfdist hfs hdfs-dataset jdbc log rabbit redis-pubsub router s3 sftp task-launcher- local task-lancher- yarn tcp throughput web socket http://cloud.spring.io/spring-cloud-stream-app-starters/
© 2016 Pivotal Software, Inc. All rights reserved. Stream Orchestration in SCDF ingest = twitterstream | cassandra dataflow:> stream create --name=ingest --definition="twitterstream | cassandra" dataflow:> stream deploy --name=ingest |
© 2016 Pivotal Software, Inc. All rights reserved. Import apps dataflow:> app import --uri http://bit.ly/1-0-4- GA-stream-applications-rabbit-maven rabbit maven kafka docker stream-applications- http://cloud.spring.io/spring-cloud-stream-app-starters/
© 2016 Pivotal Software, Inc. All rights reserved. Register your own apps dataflow:> app register --name foo --type source --uri http://example.com/foo-source-1.0.jar dataflow:> app register --name foo --type source --uri maven://com.example:foo-source:1.0 dataflow:> app register --name foo --type source --uri docker:myapps/foo-source:1.0 dataflow:> app register --name foo --type source --uri file:///tmp/foo-source:1.0
© 2016 Pivotal Software, Inc. All rights reserved. Instance Count dataflow:> stream create --name=s1 --definition="http | work | hdfs" dataflow:> stream deploy --name=s1 -- properties="app.http.count=2,app.work.count=3,app .hdfs.count=4"
© 2016 Pivotal Software, Inc. All rights reserved. Instance Count http work hdfs http work work hdfs hdfs hdfs LoadBalancer
© 2016 Pivotal Software, Inc. All rights reserved. Resource Management dataflow:> stream deploy --name=s1 -- properties="app.work.spring.cloud.deployer.cloudf oundry.memory=2048"
© 2016 Pivotal Software, Inc. All rights reserved. Resource Management http http work work work hdfs hdfs hdfs hdfs
© 2016 Pivotal Software, Inc. All rights reserved. Spring Flo http://projects.spring.io/spring-flo/
© 2016 Pivotal Software, Inc. All rights reserved. Stream Core Features •Persistent Pub-Sub •Consumer Group •Partitioning Support
© 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average Top Ns1.http s1.ave Message Broker
© 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top Ns1.http s1.ave Message Broker
© 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker
© 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}
© 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}{"id":1, "temperature":38}
© 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38}
© 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature": {"id":1, "temperature":38} {"id":1, "temperature":38}
© 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS
© 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} Average Average HDFS HDFS
© 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}{"id":1, "temperature":38} Average Average HDFS HDFS
© 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38}
© 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩
© 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩 Consumer Group
© 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩 spring.cloud.stream.bindings.<channelName>.group=ave Spring Cloud Stream group is configured by default 😁 Spring Cloud Data Flow Consumer Group
© 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs
© 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38}
© 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38}{"id":1, "temperature":38}
© 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38}
© 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38} 🤓
© 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38} 🤓consumer group subscriptions are durable 😁
© 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average
© 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩
© 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩 Partitioning Support(Stateful Stream)
© 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩stream deploy --name=xxxx --properties= "app.http.producer.partitionKeyExpression=payload.id" spring.cloud.stream.bindings.<channelName>.producer.par titionKeyExpression=payload.id Spring Cloud Stream Spring Cloud Data Flow Partitioning Support(Stateful Stream)
© 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average
© 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
© 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 🤓
© 2016 Pivotal Software, Inc. All rights reserved. Simple Real Time Analytics tweets = twitterstream | hdfs analytics = :ingest.twitterstream > field-value-counter --fieldName=lang HTTP s1.http HDFS COUNTER Data Flow Server REST API
‹#›© 2016 Pivotal Software, Inc. All rights reserved. Demo (Twitter Stream!)
© 2016 Pivotal Software, Inc. All rights reserved.
‹#›© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Task
© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Task • Spring Boot based framework for short lived task • Spring Boot's CommandLineRunner or Spring Batch • Result of each process persists in Task Repository • Well Integrated with Spring Batch (Job Repository)
© 2016 Pivotal Software, Inc. All rights reserved. Programming Model (CLR) @SpringBootApplication
 @EnableTask public class DemoTaskApp {
 @Bean
 CommandLineRunner clr() {
 return args -> System.out.println("Task!");
 } 
 public static void main(String[] args) {
 SpringApplication.run(DemoTaskApp.class, args);
 }
 } spring.application.name=hello
© 2016 Pivotal Software, Inc. All rights reserved. Task Execution List Task Name ID Start Time End Time Exit Code hello 3 Tue Nov 15 16:49:08 JST 2016 Tue Nov 15 16:49:08 JST 2016 0 hello 2 Tue Nov 15 16:48:54 JST 2016 Tue Nov 15 16:48:55 JST 2016 0 hello 1 Tue Nov 15 16:48:23 JST 2016 Tue Nov 15 16:48:23 JST 2016 0
© 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Spring Batch) @SpringBootApplication @EnableBatchProcessing
 @EnableTask public class DemoBatchApp { @Autowired JobBuilderFactory jobBuilderFactory; @Autowired StepBuilderFactory stepBuilderFactory;
 @Bean Step step1() { /* ... */ } @Bean Step step2() { /* ... */ } @Bean Job job() { return jobBuilderFactory.get("job") .start(step1()).next(step2()).build()}
 public static void main(String[] args) {
 SpringApplication.run(DemoTaskApp.class, args);
 }} https://github.com/making/cf-spring-batch-demo spring.application.name=hello-batch
© 2016 Pivotal Software, Inc. All rights reserved. Task Execution List Task Name ID Start Time End Time Exit Code hello- batch 5 Tue Nov 15 17:28:49 JST 2016 Tue Nov 15 17:28:49 JST 2016 0 hello- batch 4 Tue Nov 15 17:27:56 JST 2016 Tue Nov 15 17:27:57 JST 2016 0
© 2016 Pivotal Software, Inc. All rights reserved. Job Execution List ID Task ID Start Time Step Execution Count Definition Status 2 5 Tue Nov 15 17:28:49 JST 2016 2 Created 1 4 Tue Nov 15 17:27:56 JST 2016 2 Created
© 2016 Pivotal Software, Inc. All rights reserved. Step Execution List ID Step Name Job Exec Id Start Time End Time Status 1 step1 1 Tue Nov 15 17:28:49 JST 2016 Tue Nov 15 17:28:49 JST 2016 COMPLETED 2 step2 1 Tue Nov 15 17:27:56 JST 2016 Tue Nov 15 17:27:57 JST 2016 COMPLETED
© 2016 Pivotal Software, Inc. All rights reserved. Task Orchestration in SCDF >task create hello --definition="timestamp --format=¥"yyyy¥"" >task launch hello timestamp Data Flow Server DB Task Name Start Time End Time Exit Code Exit Message Last Updated Time Parameters Message Broker
© 2016 Pivotal Software, Inc. All rights reserved. Task Execution List
© 2016 Pivotal Software, Inc. All rights reserved. Job Execution List
© 2016 Pivotal Software, Inc. All rights reserved. Job Execution Details
© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow -> Task http | tasklaunchrequest --uri=... | task-lancher timestamp Data Flow Server DB Message Broker
‹#›© 2016 Pivotal Software, Inc. All rights reserved. Demo (Scalable Tasks)
© 2016 Pivotal Software, Inc. All rights reserved. Pivotal Cloud Foundry PDF PDF PDF PDF S3 Spring Cloud Data Flow Task Task Task Task PDF PDF Spring Cloud Task Spring Batch index index source sink
‹#›© 2016 Pivotal Software, Inc. All rights reserved. Getting Started with SCDF on Cloud Foundry
© 2016 Pivotal Software, Inc. All rights reserved. Install PCF Dev • https://docs.pivotal.io/pcf-dev • Cloud Foundry on your laptop • Included • Redis / RabbitMQ / MySQL • Spring Cloud Services • Install with cf dev start
© 2016 Pivotal Software, Inc. All rights reserved. cf create-service p-mysql 512mb df-mysql cf create-service p-rabbitmq standard df-rebbitmq cf create-service p-redis shared-vm df-redis cf push dataflow-server -p spring-cloud-dataflow- server-cloudfoundry-1.0.1.RELEASE.jar Deploy SCDF Server on PCF Dev
© 2016 Pivotal Software, Inc. All rights reserved. Tutorial • 📗 https://blog.ik.am/entries/396 • 💎 https://github.com/making-demo-scdf/spring-cloud- dataflow-cookbook
© 2016 Pivotal Software, Inc. All rights reserved. Upcoming Features • Some ‘porting’ from XD • Batch Job DSL + Designer • Role based access • Looking forward • Spring Cloud Sleuth • JavaDSL • In-place application version upgrades with Spinnaker • Application Groups • Polyglot • Expanded analytics with Redis and Python/R ecosystem • More provided apps/tasks •SCDF Tile for Pivotal Cloud Foundry
© 2016 Pivotal Software, Inc. All rights reserved. Thanks! • References • http://www.slideshare.net/SpringCentral/data-microservices-in-t cloud ✨ • https://cloud.spring.io/spring-cloud-dataflow/ • http://cloud.spring.io/spring-cloud-stream/ • http://cloud.spring.io/spring-cloud-stream-app-starters/ • https://github.com/spring-cloud/spring-cloud-task/ • http://cloud.spring.io/spring-cloud-dataflow-server-cloudfoundry

More Related Content

PDF
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1
Toshiaki Maki
 
PDF
Yahoo! JAPANのコンテンツプラットフォームを支えるSpring Cloud Streamによるマイクロサービスアーキテクチャ #jsug #sf_52
Yahoo!デベロッパーネットワーク
 
PDF
20190514 AWS Black Belt Online Seminar Amazon API Gateway
Amazon Web Services Japan
 
PDF
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #cndt2019 #osdt2019 #keynote
Junya Suzuki
 
PPTX
Qiita Night 足場固めからやるマイクロサービス
Tomoki Kuriyama
 
PPTX
[20220126] JAWS-UG 2022初頭までに葬ったAWSアンチパターン大紹介
Amazon Web Services Japan
 
PDF
20200630 AWS Black Belt Online Seminar Amazon Cognito
Amazon Web Services Japan
 
PDF
[Cloud OnAir] BigQuery の仕組みからベストプラクティスまでのご紹介 2018年9月6日 放送
Google Cloud Platform - Japan
 
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1
Toshiaki Maki
 
Yahoo! JAPANのコンテンツプラットフォームを支えるSpring Cloud Streamによるマイクロサービスアーキテクチャ #jsug #sf_52
Yahoo!デベロッパーネットワーク
 
20190514 AWS Black Belt Online Seminar Amazon API Gateway
Amazon Web Services Japan
 
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #cndt2019 #osdt2019 #keynote
Junya Suzuki
 
Qiita Night 足場固めからやるマイクロサービス
Tomoki Kuriyama
 
[20220126] JAWS-UG 2022初頭までに葬ったAWSアンチパターン大紹介
Amazon Web Services Japan
 
20200630 AWS Black Belt Online Seminar Amazon Cognito
Amazon Web Services Japan
 
[Cloud OnAir] BigQuery の仕組みからベストプラクティスまでのご紹介 2018年9月6日 放送
Google Cloud Platform - Japan
 

What's hot (20)

PPTX
[社内勉強会]ELBとALBと数万スパイク負荷テスト
Takahiro Moteki
 
PPTX
Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...
Lucas Jellema
 
PDF
[Aurora事例祭り]Amazon Aurora を使いこなすためのベストプラクティス
Amazon Web Services Japan
 
PDF
PayPayでのk8s活用事例
PayPay Corporation
 
PDF
AWS Transit Gateway를 통한 Multi-VPC 아키텍처 패턴 - 강동환 솔루션즈 아키텍트, AWS :: AWS Summit ...
Amazon Web Services Korea
 
PDF
20190129 AWS Black Belt Online Seminar AWS Identity and Access Management (AW...
Amazon Web Services Japan
 
PDF
AWS Black Belt Online Seminar 2017 Amazon Kinesis
Amazon Web Services Japan
 
PDF
ここから始めるAWSセキュリティ
Amazon Web Services Japan
 
PDF
20200722 AWS Black Belt Online Seminar AWSアカウント シングルサインオンの設計と運用
Amazon Web Services Japan
 
PDF
ヤフー発のメッセージキュー「Pulsar」のご紹介
Yahoo!デベロッパーネットワーク
 
PDF
AngularとSpring Bootで作るSPA + RESTful Web Serviceアプリケーション
ssuser070fa9
 
PDF
20190206 AWS Black Belt Online Seminar Amazon SageMaker Basic Session
Amazon Web Services Japan
 
PDF
20211203 AWS Black Belt Online Seminar AWS re:Invent 2021アップデート速報
Amazon Web Services Japan
 
PDF
Spring Cloud Data Flow の紹介 #streamctjp
Yahoo!デベロッパーネットワーク
 
PDF
アプリ開発者、DB 管理者視点での Cloud Spanner 活用方法 | 第 10 回 Google Cloud INSIDE Games & App...
Google Cloud Platform - Japan
 
PDF
Infrastructure as Code (IaC) 談義 2022
Amazon Web Services Japan
 
PDF
20200526 AWS Black Belt Online Seminar AWS X-Ray
Amazon Web Services Japan
 
PDF
AWS Black Belt Techシリーズ Amazon Kinesis
Amazon Web Services Japan
 
PDF
Docker道場オンライン#1 Docker基礎概念と用語の理解
Masahito Zembutsu
 
PDF
コンテナDojo #4:VSCodeを使ったPodmanコンテナアプリ開発.pdf
Teruyoshi Matsushima
 
[社内勉強会]ELBとALBと数万スパイク負荷テスト
Takahiro Moteki
 
Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...
Lucas Jellema
 
[Aurora事例祭り]Amazon Aurora を使いこなすためのベストプラクティス
Amazon Web Services Japan
 
PayPayでのk8s活用事例
PayPay Corporation
 
AWS Transit Gateway를 통한 Multi-VPC 아키텍처 패턴 - 강동환 솔루션즈 아키텍트, AWS :: AWS Summit ...
Amazon Web Services Korea
 
20190129 AWS Black Belt Online Seminar AWS Identity and Access Management (AW...
Amazon Web Services Japan
 
AWS Black Belt Online Seminar 2017 Amazon Kinesis
Amazon Web Services Japan
 
ここから始めるAWSセキュリティ
Amazon Web Services Japan
 
20200722 AWS Black Belt Online Seminar AWSアカウント シングルサインオンの設計と運用
Amazon Web Services Japan
 
ヤフー発のメッセージキュー「Pulsar」のご紹介
Yahoo!デベロッパーネットワーク
 
AngularとSpring Bootで作るSPA + RESTful Web Serviceアプリケーション
ssuser070fa9
 
20190206 AWS Black Belt Online Seminar Amazon SageMaker Basic Session
Amazon Web Services Japan
 
20211203 AWS Black Belt Online Seminar AWS re:Invent 2021アップデート速報
Amazon Web Services Japan
 
Spring Cloud Data Flow の紹介 #streamctjp
Yahoo!デベロッパーネットワーク
 
アプリ開発者、DB 管理者視点での Cloud Spanner 活用方法 | 第 10 回 Google Cloud INSIDE Games & App...
Google Cloud Platform - Japan
 
Infrastructure as Code (IaC) 談義 2022
Amazon Web Services Japan
 
20200526 AWS Black Belt Online Seminar AWS X-Ray
Amazon Web Services Japan
 
AWS Black Belt Techシリーズ Amazon Kinesis
Amazon Web Services Japan
 
Docker道場オンライン#1 Docker基礎概念と用語の理解
Masahito Zembutsu
 
コンテナDojo #4:VSCodeを使ったPodmanコンテナアプリ開発.pdf
Teruyoshi Matsushima
 
Ad

Viewers also liked (20)

PDF
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
Toshiaki Maki
 
PDF
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Toshiaki Maki
 
PDF
Spring Day 2016 - Web API アクセス制御の最適解
都元ダイスケ Miyamoto
 
PDF
Lineにおけるspring frameworkの活用
Tokuhiro Matsuno
 
PDF
Spring Day 2016 springの現在過去未来
Yuichi Hasegawa
 
PPTX
Spring bootで学ぶ初めてのwebアプリ開発
terahide
 
PPTX
Spring 5に備えるリアクティブプログラミング入門
Takuya Iwatsuka
 
PPT
Springを使ったwebアプリにリファクタリングしよう
土岐 孝平
 
PDF
Application Re-Architecture Technology ～ StrutsからSpring MVCへ ～
Yuichi Hasegawa
 
PDF
Grailsでドメイン駆動設計を実践する時の勘所
Takuma Watabiki
 
PPTX
Cloud Foundry x Wagby
Yoshinori Nie
 
PDF
楽天トラベルとSpring(Spring Day 2016)
Rakuten Group, Inc.
 
PDF
ビッグじゃなくても使えるSpark Streaming
chibochibo
 
PPT
Distributed Tracing Velocity2016
Reshmi Krishna
 
PDF
Distributed tracing - get a grasp on your production
nklmish
 
PPTX
Spring CloudとZipkinを利用した分散トレーシング
Rakuten Group, Inc.
 
PDF
形態素解析
Works Applications
 
PPTX
WalB: Real-time and Incremental Backup System for Block Devices
uchan_nos
 
PDF
3000社の業務データ絞り込みを支える技術
Ryo Mitoma
 
PDF
離れた場所でも最高のチームワークを実現する方法 ーサイボウズ開発チームのリモートワーク事例ー
Teppei Sato
 
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
Toshiaki Maki
 
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Toshiaki Maki
 
Spring Day 2016 - Web API アクセス制御の最適解
都元ダイスケ Miyamoto
 
Lineにおけるspring frameworkの活用
Tokuhiro Matsuno
 
Spring Day 2016 springの現在過去未来
Yuichi Hasegawa
 
Spring bootで学ぶ初めてのwebアプリ開発
terahide
 
Spring 5に備えるリアクティブプログラミング入門
Takuya Iwatsuka
 
Springを使ったwebアプリにリファクタリングしよう
土岐 孝平
 
Application Re-Architecture Technology ～ StrutsからSpring MVCへ ～
Yuichi Hasegawa
 
Grailsでドメイン駆動設計を実践する時の勘所
Takuma Watabiki
 
Cloud Foundry x Wagby
Yoshinori Nie
 
楽天トラベルとSpring(Spring Day 2016)
Rakuten Group, Inc.
 
ビッグじゃなくても使えるSpark Streaming
chibochibo
 
Distributed Tracing Velocity2016
Reshmi Krishna
 
Distributed tracing - get a grasp on your production
nklmish
 
Spring CloudとZipkinを利用した分散トレーシング
Rakuten Group, Inc.
 
形態素解析
Works Applications
 
WalB: Real-time and Incremental Backup System for Block Devices
uchan_nos
 
3000社の業務データ絞り込みを支える技術
Ryo Mitoma
 
離れた場所でも最高のチームワークを実現する方法 ーサイボウズ開発チームのリモートワーク事例ー
Teppei Sato
 
Ad

Similar to Data Microservices with Spring Cloud Stream, Task, and Data Flow #jsug #springday (20)

PDF
Spring Cloud Servicesの紹介 #pcf_tokyo
Toshiaki Maki
 
PDF
Meetup: Streaming Data Pipeline Development
Timothy Spann
 
PDF
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
Timothy Spann
 
PDF
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
Timothy Spann
 
PDF
Spring Framework 5.0による Reactive Web Application #JavaDayTokyo
Toshiaki Maki
 
PDF
Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...
VMware Tanzu
 
PPTX
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...
Docker, Inc.
 
PDF
Hadoop application architectures - using Customer 360 as an example
hadooparchbook
 
PPTX
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
VMware Tanzu
 
PPTX
Episode 3: Kubernetes and Big Data Services
Mesosphere Inc.
 
PDF
Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝
OSS On Azure
 
PPTX
Big Data Analytics with Spark
DataStax Academy
 
PDF
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
Kai Wähner
 
PDF
Data Science at Scale on MPP databases - Use Cases & Open Source Tools
Esther Vasiete
 
PPTX
Fraud Detection for Israel BigThings Meetup
Gwen (Chen) Shapira
 
PDF
DevOps Days Tel Aviv - Serverless Architecture
Antons Kranga
 
PPTX
Spark Streaming with Azure Databricks
Dustin Vannoy
 
PDF
Spark Summit EU talk by Christos Erotocritou
Spark Summit
 
PDF
Apache Kafka - Scalable Message-Processing and more !
Guido Schmutz
 
PDF
Headaches and Breakthroughs in Building Continuous Applications
Databricks
 
Spring Cloud Servicesの紹介 #pcf_tokyo
Toshiaki Maki
 
Meetup: Streaming Data Pipeline Development
Timothy Spann
 
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
Timothy Spann
 
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
Timothy Spann
 
Spring Framework 5.0による Reactive Web Application #JavaDayTokyo
Toshiaki Maki
 
Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...
VMware Tanzu
 
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...
Docker, Inc.
 
Hadoop application architectures - using Customer 360 as an example
hadooparchbook
 
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
VMware Tanzu
 
Episode 3: Kubernetes and Big Data Services
Mesosphere Inc.
 
Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝
OSS On Azure
 
Big Data Analytics with Spark
DataStax Academy
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
Kai Wähner
 
Data Science at Scale on MPP databases - Use Cases & Open Source Tools
Esther Vasiete
 
Fraud Detection for Israel BigThings Meetup
Gwen (Chen) Shapira
 
DevOps Days Tel Aviv - Serverless Architecture
Antons Kranga
 
Spark Streaming with Azure Databricks
Dustin Vannoy
 
Spark Summit EU talk by Christos Erotocritou
Spark Summit
 
Apache Kafka - Scalable Message-Processing and more !
Guido Schmutz
 
Headaches and Breakthroughs in Building Continuous Applications
Databricks
 

More from Toshiaki Maki (20)

PDF
From Spring Boot 2.2 to Spring Boot 2.3 #jsug
Toshiaki Maki
 
PDF
Concourse x Spinnaker #concourse_tokyo
Toshiaki Maki
 
PDF
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
Toshiaki Maki
 
PDF
Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1
Toshiaki Maki
 
PDF
Spring Boot Actuator 2.0 & Micrometer
Toshiaki Maki
 
PDF
Open Service Broker APIとKubernetes Service Catalog #k8sjp
Toshiaki Maki
 
PDF
Spring Cloud Function & Project riff #jsug
Toshiaki Maki
 
PDF
Introduction to Spring WebFlux #jsug #sf_a1
Toshiaki Maki
 
PDF
BOSH / CF Deployment in modern ways #cf_tokyo
Toshiaki Maki
 
PDF
Why PCF is the best platform for Spring Boot
Toshiaki Maki
 
PDF
Zipkin Components #zipkin_jp
Toshiaki Maki
 
PPTX
マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07
Toshiaki Maki
 
PDF
実例で学ぶ、明日から使えるSpring Boot Tips #jsug
Toshiaki Maki
 
PDF
Spring ❤️ Kotlin #jjug
Toshiaki Maki
 
PDF
Managing your Docker image continuously with Concourse CI
Toshiaki Maki
 
PDF
Short Lived Tasks in Cloud Foundry #cfdtokyo
Toshiaki Maki
 
PDF
今すぐ始めるCloud Foundry #hackt #hackt_k
Toshiaki Maki
 
PDF
Team Support in Concourse CI 2.0 #concourse_tokyo
Toshiaki Maki
 
PDF
From Zero to Hero with REST and OAuth2 #jjug
Toshiaki Maki
 
PDF
Consumer Driven Contractsで REST API/マイクロサービスをテスト #m3tech
Toshiaki Maki
 
From Spring Boot 2.2 to Spring Boot 2.3 #jsug
Toshiaki Maki
 
Concourse x Spinnaker #concourse_tokyo
Toshiaki Maki
 
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
Toshiaki Maki
 
Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1
Toshiaki Maki
 
Spring Boot Actuator 2.0 & Micrometer
Toshiaki Maki
 
Open Service Broker APIとKubernetes Service Catalog #k8sjp
Toshiaki Maki
 
Spring Cloud Function & Project riff #jsug
Toshiaki Maki
 
Introduction to Spring WebFlux #jsug #sf_a1
Toshiaki Maki
 
BOSH / CF Deployment in modern ways #cf_tokyo
Toshiaki Maki
 
Why PCF is the best platform for Spring Boot
Toshiaki Maki
 
Zipkin Components #zipkin_jp
Toshiaki Maki
 
マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07
Toshiaki Maki
 
実例で学ぶ、明日から使えるSpring Boot Tips #jsug
Toshiaki Maki
 
Spring ❤️ Kotlin #jjug
Toshiaki Maki
 
Managing your Docker image continuously with Concourse CI
Toshiaki Maki
 
Short Lived Tasks in Cloud Foundry #cfdtokyo
Toshiaki Maki
 
今すぐ始めるCloud Foundry #hackt #hackt_k
Toshiaki Maki
 
Team Support in Concourse CI 2.0 #concourse_tokyo
Toshiaki Maki
 
From Zero to Hero with REST and OAuth2 #jjug
Toshiaki Maki
 
Consumer Driven Contractsで REST API/マイクロサービスをテスト #m3tech
Toshiaki Maki
 

Recently uploaded (20)

PDF
NewMind AI Weekly Chronicles - August'25-Week II
NewMind AI
 
PDF
Encapsulation theory and applications.pdf
gurumoop
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
PPTX
Machine Learning_overview_presentation.pptx
kondavamsidharreddy2
 
PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
PPTX
Spectroscopy.pptx food analysis technology
nawahassan45
 
PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
PDF
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
PDF
A comparative analysis of optical character recognition models for extracting...
IAESIJAI
 
NewMind AI Weekly Chronicles - August'25-Week II
NewMind AI
 
Encapsulation theory and applications.pdf
gurumoop
 
Approach and Philosophy of On baking technology
30anthuong17
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
Machine Learning_overview_presentation.pptx
kondavamsidharreddy2
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
Spectroscopy.pptx food analysis technology
nawahassan45
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
A comparative analysis of optical character recognition models for extracting...
IAESIJAI
 

Data Microservices with Spring Cloud Stream, Task, and Data Flow #jsug #springday

  • 1. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices with Spring Cloud Stream, Task, and Data Flow Toshiaki Maki (@making) 2016-11-18 Spring Day 2016 #springday #jsug
  • 2. © 2016 Pivotal Software, Inc. All rights reserved. Who am I ? • Toshiaki Maki (@making) http://blog.ik.am • Sr. Solutions Architect @Pivotal • Spring Framework enthusiast bit.ly/hajiboot2
  • 3. © 2016 Pivotal Software, Inc. All rights reserved. Input Motivation HTTP TCP S3 JDBC Rabbit JMS Twitter Syslog Output HDFS Cassandra HAWQ Greenplum JDBC S3 TCP Gemfire ❓❔❓ • Real Time Analysis • Data Ingestion • ETL Process
  • 4. © 2016 Pivotal Software, Inc. All rights reserved. Spring XD (eXtreme Data)
  • 5. © 2016 Pivotal Software, Inc. All rights reserved. Spring XD (eXtreme Data) 🤔
  • 6. © 2016 Pivotal Software, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs
  • 7. © 2016 Pivotal Software, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs 😇
  • 8. © 2016 Pivotal Software, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs
  • 9. © 2016 Pivotal Software, Inc. All rights reserved. Modern Platforms Cloud Foundry YARN Kubernetes Mesos Docker Swarm Nomad OpenShift
  • 10. © 2016 Pivotal Software, Inc. All rights reserved. Cloud Native Redesign •From multiple modules embedded in a container to standalone executable applications •From our own runtime to delegating to existing modern platforms
  • 11. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
  • 12. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
  • 13. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing
  • 14. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing
  • 15. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers
  • 16. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers
  • 17. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
  • 18. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
  • 19. © 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
  • 20. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications
  • 21. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream
  • 22. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream Spring Cloud Task
  • 23. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream Spring Cloud Task Orchestration Layer
  • 24. © 2016 Pivotal Software, Inc. All rights reserved. Structure Spring Cloud Data Flow Spring Cloud Deployer (SPI) Spring Cloud Stream Spring Cloud Task Spring Integration Spring Boot Spring Batch
  • 25. © 2016 Pivotal Software, Inc. All rights reserved. Structure Spring Cloud Data Flow Spring Cloud Deployer (SPI) Spring Cloud Stream Spring Cloud Task Spring Integration Spring Boot Spring Batch • Spring Cloud Deployer Local • Spring Cloud Deployer Cloud Foundry • Spring Cloud Deployer Yarn • Spring Cloud Deployer Kubernetes • Spring Cloud Deployer Mesos
  • 26. © 2016 Pivotal Software, Inc. All rights reserved. SCDF Deployment Platform Spring Cloud Data Flow Server Deployer SPI REST API SCDFShell SCDF Flo
  • 27. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow Architecture gpfdist cassandra jms http stream1 = http | cassandra stream2 = jms | gpfdist Message Broker Data Flow Server DB Platform Runtime
  • 28. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow Architecture gpfdist cassandra jms http stream1 = http | cassandra stream2 = jms | gpfdist Message Broker Data Flow Server DB Platform Runtime
  • 29. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Stream
  • 30. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Stream • Event-driven microservice framework • Built on battle-tested components (Spring Boot / Spring Integration) • Opinionated primitives for streaming applications • Persistent Pub/Sub • Consumer Groups • Partitioning Support • Pluggable messaging middleware bindings source | processor | sink
  • 31. © 2016 Pivotal Software, Inc. All rights reserved. JJUG CCC in 2 weeks
  • 32. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Stream Applications Twitter Stream Cassandra java -jar twittersource.jar --server.port=8080 --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings. output.destination=ingest Source Sink java -jar cassandrasink.jar --server.port=8081 --spring.cassandra.keyspace=tweet --spring.cloud.stream.bindings. input.destination=ingest
  • 33. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Stream Applications Twitter Stream Cassandra java -jar twittersource.jar --server.port=8080 --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings. output.destination=ingest Source Sink java -jar cassandrasink.jar --server.port=8081 --spring.cassandra.keyspace=tweet --spring.cloud.stream.bindings. input.destination=ingest Twitter Stream Cassandraingest
  • 34. © 2016 Pivotal Software, Inc. All rights reserved. Message Binders • @EnableBinding • Binder Implementations • Production-Ready • Rabbit MQ • Apache Kafka • Experimental • JMS • Google PubSub
  • 35. © 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Sink) @SpringBootApplication
 @EnableBinding(Sink.class)
 public class DemoSinkApp { @StreamListener(Sink.INPUT)
 void receive(Message<String> message) {
 System.out.println("Received " + message);
 }
 public static void main(String[] args) {
 SpringApplication.run(DemoSinkApp.class, args);
 }
 }
  • 36. © 2016 Pivotal Software, Inc. All rights reserved. Sink Properties (Sink) spring.cloud.stream.bindings.input.destination=demo-strm demo- strm input
  • 37. © 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Source) @SpringBootApplication @RestController @EnableBinding(Source.class)
 public class DemoSourceApp {
 @Autowired Source source; @GetMapping void send(@RequestParam String text) {
 source.output() .send(MessageBuilder.withPayload(text).build());
 }
 public static void main(String[] args) {
 SpringApplication.run(DemoSourceApp.class, args);
 }
 }
  • 38. © 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Source) @SpringBootApplication @RestController @EnableBinding(Source.class)
 public class DemoSourceApp { @Bean @InboundChannelAdapter(channel = Source.OUTPUT, poller = @Poller(fixedDelay = "1000", maxMessagesPerPoll = "1")) MessageSource<String> source() {
 return () -> MessageBuilder.withPayload("Hi").build());
 }
 public static void main(String[] args) {
 SpringApplication.run(DemoSourceApp.class, args);
 }}
  • 39. © 2016 Pivotal Software, Inc. All rights reserved. Properties (Source) spring.cloud.stream.bindings.output.destination=demo-strm demo- strm Source output
  • 40. © 2016 Pivotal Software, Inc. All rights reserved. Binder (RabbitMQ) <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-rabbit</artifactId> </dependency> demo- strm
  • 41. © 2016 Pivotal Software, Inc. All rights reserved. Binder (Apache Kafka) <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-kafka</artifactId> </dependency> demo- strm
  • 42. © 2016 Pivotal Software, Inc. All rights reserved. Sink Pipeline demo- strm input Source output source | sink
  • 43. © 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Processor) @SpringBootApplication
 @EnableBinding(Processor.class)
 public class DemoProcessorApp { @StreamListener(Processor.INPUT) @SendTo(Processor.OUTPUT)
 void receive(String text) {
 return "[[" + text + "]]";
 }
 public static void main(String[] args) {
 SpringApplication.run(DemoProcessorApp.class, args);
 }
 }
  • 44. © 2016 Pivotal Software, Inc. All rights reserved. Properties (Processor) spring.cloud.stream.bindings.output.destination=my-source spring.cloud.stream.bindings.input.destination=my-source spring.cloud.stream.bindings.output.destination=my-proc spring.cloud.stream.bindings.input.destination=my-proc Source Processor Sink
  • 45. © 2016 Pivotal Software, Inc. All rights reserved. Pipeline my- source Source output source | processor | sink Processor output input my-proc Sink input
  • 46. © 2016 Pivotal Software, Inc. All rights reserved. Reactive API Support @SpringBootApplication
 @EnableBinding(Processor.class)
 public class DemoProcessorRxApp { @StreamListener @Output(Processor.OUTPUT)
 public Flux<String> receive(@Input(Processor.INPUT) Flux<String> stream) {
 return stream.map(text -> "[[" + text + "]]");
 }
 public static void main(String[] args) {
 SpringApplication.run(DemoProcessorRxApp.class, args);
 }
 }
  • 47. © 2016 Pivotal Software, Inc. All rights reserved. Reactive API Support @StreamListener @Output(Processor.OUTPUT)
 public Flux<AverageData> receive(@Input(Processor.INPUT) Flux<SensorData> stream) {
 return stream.window(Duration.ofSecond(20), Duration.ofSecond(10)) .flatMap(win -> win.groupBy(sensor -> sensor.id)) .flatMap(group -> calcAverage(group));
 }
  • 48. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Stream App Starters Source Processor Sink file ftp gemfire gemfire-cq http jdbc jms load-generator loggregator mail mongodb rabbit s3 sftp syslog tcp tcp-client time trigger triggertask twitterstream bridge filter groovy-filter groovy- transform httpclient pommel scriptable- transform splitter tcp-client transform aggregate- counter cassandra counter field-value- counter file ftp gemfire gpfdist hfs hdfs-dataset jdbc log rabbit redis-pubsub router s3 sftp task-launcher- local task-lancher- yarn tcp throughput web socket http://cloud.spring.io/spring-cloud-stream-app-starters/
  • 49. © 2016 Pivotal Software, Inc. All rights reserved. Stream Orchestration in SCDF ingest = twitterstream | cassandra dataflow:> stream create --name=ingest --definition="twitterstream | cassandra" dataflow:> stream deploy --name=ingest |
  • 50. © 2016 Pivotal Software, Inc. All rights reserved. Import apps dataflow:> app import --uri http://bit.ly/1-0-4- GA-stream-applications-rabbit-maven rabbit maven kafka docker stream-applications- http://cloud.spring.io/spring-cloud-stream-app-starters/
  • 51. © 2016 Pivotal Software, Inc. All rights reserved. Register your own apps dataflow:> app register --name foo --type source --uri http://example.com/foo-source-1.0.jar dataflow:> app register --name foo --type source --uri maven://com.example:foo-source:1.0 dataflow:> app register --name foo --type source --uri docker:myapps/foo-source:1.0 dataflow:> app register --name foo --type source --uri file:///tmp/foo-source:1.0
  • 52. © 2016 Pivotal Software, Inc. All rights reserved. Instance Count dataflow:> stream create --name=s1 --definition="http | work | hdfs" dataflow:> stream deploy --name=s1 -- properties="app.http.count=2,app.work.count=3,app .hdfs.count=4"
  • 53. © 2016 Pivotal Software, Inc. All rights reserved. Instance Count http work hdfs http work work hdfs hdfs hdfs LoadBalancer
  • 54. © 2016 Pivotal Software, Inc. All rights reserved. Resource Management dataflow:> stream deploy --name=s1 -- properties="app.work.spring.cloud.deployer.cloudf oundry.memory=2048"
  • 55. © 2016 Pivotal Software, Inc. All rights reserved. Resource Management http http work work work hdfs hdfs hdfs hdfs
  • 56. © 2016 Pivotal Software, Inc. All rights reserved. Spring Flo http://projects.spring.io/spring-flo/
  • 57. © 2016 Pivotal Software, Inc. All rights reserved. Stream Core Features •Persistent Pub-Sub •Consumer Group •Partitioning Support
  • 58. © 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average Top Ns1.http s1.ave Message Broker
  • 59. © 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top Ns1.http s1.ave Message Broker
  • 60. © 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker
  • 61. © 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}
  • 62. © 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}{"id":1, "temperature":38}
  • 63. © 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38}
  • 64. © 2016 Pivotal Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature": {"id":1, "temperature":38} {"id":1, "temperature":38}
  • 65. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS
  • 66. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} Average Average HDFS HDFS
  • 67. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}{"id":1, "temperature":38} Average Average HDFS HDFS
  • 68. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38}
  • 69. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩
  • 70. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩 Consumer Group
  • 71. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩 spring.cloud.stream.bindings.<channelName>.group=ave Spring Cloud Stream group is configured by default 😁 Spring Cloud Data Flow Consumer Group
  • 72. © 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs
  • 73. © 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38}
  • 74. © 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38}{"id":1, "temperature":38}
  • 75. © 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38}
  • 76. © 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38} 🤓
  • 77. © 2016 Pivotal Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38} 🤓consumer group subscriptions are durable 😁
  • 78. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average
  • 79. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 80. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 81. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 82. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 83. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 84. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩
  • 85. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩 Partitioning Support(Stateful Stream)
  • 86. © 2016 Pivotal Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩stream deploy --name=xxxx --properties= "app.http.producer.partitionKeyExpression=payload.id" spring.cloud.stream.bindings.<channelName>.producer.par titionKeyExpression=payload.id Spring Cloud Stream Spring Cloud Data Flow Partitioning Support(Stateful Stream)
  • 87. © 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average
  • 88. © 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 89. © 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 90. © 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 91. © 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 92. © 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
  • 93. © 2016 Pivotal Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 🤓
  • 94. © 2016 Pivotal Software, Inc. All rights reserved. Simple Real Time Analytics tweets = twitterstream | hdfs analytics = :ingest.twitterstream > field-value-counter --fieldName=lang HTTP s1.http HDFS COUNTER Data Flow Server REST API
  • 95. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. Demo (Twitter Stream!)
  • 96. © 2016 Pivotal Software, Inc. All rights reserved.
  • 97. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Task
  • 98. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Task • Spring Boot based framework for short lived task • Spring Boot's CommandLineRunner or Spring Batch • Result of each process persists in Task Repository • Well Integrated with Spring Batch (Job Repository)
  • 99. © 2016 Pivotal Software, Inc. All rights reserved. Programming Model (CLR) @SpringBootApplication
 @EnableTask public class DemoTaskApp {
 @Bean
 CommandLineRunner clr() {
 return args -> System.out.println("Task!");
 } 
 public static void main(String[] args) {
 SpringApplication.run(DemoTaskApp.class, args);
 }
 } spring.application.name=hello
  • 100. © 2016 Pivotal Software, Inc. All rights reserved. Task Execution List Task Name ID Start Time End Time Exit Code hello 3 Tue Nov 15 16:49:08 JST 2016 Tue Nov 15 16:49:08 JST 2016 0 hello 2 Tue Nov 15 16:48:54 JST 2016 Tue Nov 15 16:48:55 JST 2016 0 hello 1 Tue Nov 15 16:48:23 JST 2016 Tue Nov 15 16:48:23 JST 2016 0
  • 101. © 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Spring Batch) @SpringBootApplication @EnableBatchProcessing
 @EnableTask public class DemoBatchApp { @Autowired JobBuilderFactory jobBuilderFactory; @Autowired StepBuilderFactory stepBuilderFactory;
 @Bean Step step1() { /* ... */ } @Bean Step step2() { /* ... */ } @Bean Job job() { return jobBuilderFactory.get("job") .start(step1()).next(step2()).build()}
 public static void main(String[] args) {
 SpringApplication.run(DemoTaskApp.class, args);
 }} https://github.com/making/cf-spring-batch-demo spring.application.name=hello-batch
  • 102. © 2016 Pivotal Software, Inc. All rights reserved. Task Execution List Task Name ID Start Time End Time Exit Code hello- batch 5 Tue Nov 15 17:28:49 JST 2016 Tue Nov 15 17:28:49 JST 2016 0 hello- batch 4 Tue Nov 15 17:27:56 JST 2016 Tue Nov 15 17:27:57 JST 2016 0
  • 103. © 2016 Pivotal Software, Inc. All rights reserved. Job Execution List ID Task ID Start Time Step Execution Count Definition Status 2 5 Tue Nov 15 17:28:49 JST 2016 2 Created 1 4 Tue Nov 15 17:27:56 JST 2016 2 Created
  • 104. © 2016 Pivotal Software, Inc. All rights reserved. Step Execution List ID Step Name Job Exec Id Start Time End Time Status 1 step1 1 Tue Nov 15 17:28:49 JST 2016 Tue Nov 15 17:28:49 JST 2016 COMPLETED 2 step2 1 Tue Nov 15 17:27:56 JST 2016 Tue Nov 15 17:27:57 JST 2016 COMPLETED
  • 105. © 2016 Pivotal Software, Inc. All rights reserved. Task Orchestration in SCDF >task create hello --definition="timestamp --format=¥"yyyy¥"" >task launch hello timestamp Data Flow Server DB Task Name Start Time End Time Exit Code Exit Message Last Updated Time Parameters Message Broker
  • 106. © 2016 Pivotal Software, Inc. All rights reserved. Task Execution List
  • 107. © 2016 Pivotal Software, Inc. All rights reserved. Job Execution List
  • 108. © 2016 Pivotal Software, Inc. All rights reserved. Job Execution Details
  • 109. © 2016 Pivotal Software, Inc. All rights reserved. Spring Cloud Data Flow -> Task http | tasklaunchrequest --uri=... | task-lancher timestamp Data Flow Server DB Message Broker
  • 110. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. Demo (Scalable Tasks)
  • 111. © 2016 Pivotal Software, Inc. All rights reserved. Pivotal Cloud Foundry PDF PDF PDF PDF S3 Spring Cloud Data Flow Task Task Task Task PDF PDF Spring Cloud Task Spring Batch index index source sink
  • 112. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. Getting Started with SCDF on Cloud Foundry
  • 113. © 2016 Pivotal Software, Inc. All rights reserved. Install PCF Dev • https://docs.pivotal.io/pcf-dev • Cloud Foundry on your laptop • Included • Redis / RabbitMQ / MySQL • Spring Cloud Services • Install with cf dev start
  • 114. © 2016 Pivotal Software, Inc. All rights reserved. cf create-service p-mysql 512mb df-mysql cf create-service p-rabbitmq standard df-rebbitmq cf create-service p-redis shared-vm df-redis cf push dataflow-server -p spring-cloud-dataflow- server-cloudfoundry-1.0.1.RELEASE.jar Deploy SCDF Server on PCF Dev
  • 115. © 2016 Pivotal Software, Inc. All rights reserved. Tutorial • 📗 https://blog.ik.am/entries/396 • 💎 https://github.com/making-demo-scdf/spring-cloud- dataflow-cookbook
  • 116. © 2016 Pivotal Software, Inc. All rights reserved. Upcoming Features • Some ‘porting’ from XD • Batch Job DSL + Designer • Role based access • Looking forward • Spring Cloud Sleuth • JavaDSL • In-place application version upgrades with Spinnaker • Application Groups • Polyglot • Expanded analytics with Redis and Python/R ecosystem • More provided apps/tasks •SCDF Tile for Pivotal Cloud Foundry
  • 117. © 2016 Pivotal Software, Inc. All rights reserved. Thanks! • References • http://www.slideshare.net/SpringCentral/data-microservices-in-t cloud ✨ • https://cloud.spring.io/spring-cloud-dataflow/ • http://cloud.spring.io/spring-cloud-stream/ • http://cloud.spring.io/spring-cloud-stream-app-starters/ • https://github.com/spring-cloud/spring-cloud-task/ • http://cloud.spring.io/spring-cloud-dataflow-server-cloudfoundry