CloudLand - A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMP
0 likes23 views
This document discusses a high-speed data ingestion service in Java using MQTT, AMQP, and STOMP messaging protocols. It provides an overview of message-oriented middleware (MOM), describes the MQTT, AMQP, and STOMP protocols, and demonstrates consuming messages from ActiveMQ using JMS.
![Copyright © 2023, Oracle and/or its affiliates
Demo # 2: Reactive Streams Ingestion (RSI)
⚫ PushPublisher API
// Push Publisher for Simple Usage
PushPublisher<Customer> pushPublisher =
ReactiveStreamsIngestion.pushPublisher();
pushPublisher.subscribe(rsi.subscriber());
// Ad-hoc usage
pushPublisher.accept(
new Customer(1, "John Doe", "North"));
// As a Consumer of a Stream
Customer[] customers = …
Stream
.of(customers)
.filter(c -> c.region.equals("NORTH"))
.forEach(pushPublisher::accept);
// As a Consumer for 3rd party Reactive Stream
// Libraries eg: Reactor https://projectreactor.io/
Flux
.just(
new Customer(1, "John Doe", "North"),
new Customer(2, "Jane Smith", "South"))
.concatWithValues(new Customer(3, "Bob", "South"))
.doOnComplete(() -> {System.out.println("Done!");})
.doOnCancel(() -> {System.out.println("Canceled!");})
.subscribe(pushPublisher::accept);](https://image.slidesharecdn.com/cloudland-ahigh-speeddataingestionserviceinjavausingmqttamqpandstomp-240215223740-bcc78d56/85/CloudLand-A-High-Speed-Data-Ingestion-Service-in-Java-Using-MQTT-AMQP-and-STOMP-12-320.jpg)
CloudLand - A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMP
- 1. A High-Speed Data Ingestion Service in Java Using MQTT, AMQP, and STOMP CloudLand 2023 Juarez Barbosa Junior - @juarezjunior Sr. Principal Java Developer Evangelist ORACLE June 2023 Copyright © 2023, Oracle and/or its affiliates
- 2. About me • Juarez Barbosa Junior - @juarezjunior • Senior Principal Java Developer Evangelist • Over 20 years of experience in IT • SW Engineering, Developer Relations • Microsoft, Oracle, IBM, Nokia, Unisys, Accenture, and a few startups • Microsoft Azure Developer Relations Lead • IBM Watson Tech Evangelist & Cloud Rockstar • IBM Mobile Tech Evangelist & Global Thought Leader • Nokia Developers Global Champion • Lead Software/DevOps Architect • Expertise • Java, Cloud, DevOps, Cloud-native, Blockchain Copyright © 2023, Oracle and/or its affiliates
- 3. Copyright © 2023, Oracle and/or its affiliates Java App Dev with Oracle Database
- 4. Copyright © 2023, Oracle and/or its affiliates Overview of Oracle DB Access with Java User Java Code JDBC Reactive Extension Standard JDBC API R2DBC + 3rd party Reactive Streams Libraries Async call with non-blocking backpressure operators (map, reduce, filters), concurrency modeling, monitoring, tracing Implements Java SE reactive stream interface (Flow) Full Reactive Streams Sync/blocking JDBC calls Java Business Logic User Java code Oracle Database Oracle JDBC driver VTs/lightweight JDBC calls
- 5. Copyright © 2023, Oracle and/or its affiliates Reactive JDBC - Sync vs Async JDBC Setup Blocking Handle Result Setup Non-Blocking Handle Result Synchronous JDBC Setup Blocking Handle Result Setup Non-Blocking Handle Result Setup Non-Blocking Handle Result Reactive JDBC Database Setup Blocking Handle Result Database
- 6. Copyright © 2023, Oracle and/or its affiliates Support for the Latest Java Versions • Java 11 - native support, compiled with it • Java 17 - certified • JDBC Standards - 4.2 and 4.3 • GraalVM - native image instrumentation • Reactive Streams - Java Flow API support • Project Loom - Virtual Threads support • Data access is critical in mission-critical apps
- 7. Copyright © 2023, Oracle and/or its affiliates Classic Java Platform Threads ⚫ Database access with blocking threads ⚫ A JDBC call blocks a thread for 100’s of milliseconds (thread-per-request style) ⚫ Thread count increases linearly with the number of JDBC calls ⚫ Performance degrades as the number of threads increases ⚫ 1 MB of stack memory per thread ⚫ Scheduling many platform threads is expensive ⚫ Preemptive scheduling vs time-slicing
- 8. Copyright © 2022, Oracle and/or its affiliates Virtual Threads - JEPs 425, 436, 444 ⚫ Virtual Threads – JEPs (JDK Enhancement Proposals) ⚫ Lightweight threads that dramatically reduce the effort of writing, maintaining, and observing high-throughput concurrent applications ⚫ Enable applications written in the simple thread-per-request style to scale with near- optimal hardware utilization ⚫ Enable easy troubleshooting, debugging, and profiling of virtual threads with existing JDK tools ⚫ Do not remove the traditional implementation of threads ⚫ Do not alter the basic concurrency model of Java ⚫ Enable existing code that uses Java threads (java.lang.Thread, etc) to adopt virtual threads with minimal change
- 9. Copyright © 2023, Oracle and/or its affiliates Virtual Threads - JEPs 425, 436, 444 ⚫ java.lang.Thread ⚫ final Boolean - isVirtual() ⚫ static Thread.Builder.OfVirtual - ofVirtual() ⚫ static Thread.Builder.OfPlatform - ofPlatform() ⚫ static Thread - startVirtualThread(Runnable task) ⚫ java.util.concurrent.Executors ⚫ static ExecutorService - newVirtualThreadPerTaskExecutor()
- 10. Copyright © 2022, Oracle and/or its affiliates Demo # 1: Virtual vs Platform Threads ⚫ Virtual Threads versus Platform Threads ⚫ JEP 425 Virtual Threads (Preview) ⚫ JEP 436 (Second Preview) ⚫ JEP 444 (Target -> JDK 21) ⚫ Runs on Java 19, 20, 21 ⚫ javac --release 19 --enable-preview ⚫ java --enable-preview ⚫ Oracle JDBC Driver 21c instrumented to support Virtual Threads ⚫ Verifiable comparison of OS/HW resources consumption (Platform Threads x Virtual Threads)
- 11. Copyright © 2022, Oracle and/or its affiliates Reactive Streams Ingestion (RSI) ⚫ Java Library for Reactive Streams Ingestion ⚫ Streaming capability: Ingest data in an unblocking, and reactive way from a large group of clients ⚫ Group records through RAC (Real App Clusters), and Shard affinity using native UCP (Universal Connection Pool) ⚫ Optimize CPU allocation while decoupling record Processing from I/O ⚫ Fastest insert method for the Oracle Database through Direct Path Insert, bypassing SQL and writing directly into the DB files
- 12. Copyright © 2023, Oracle and/or its affiliates Demo # 2: Reactive Streams Ingestion (RSI) ⚫ PushPublisher API // Push Publisher for Simple Usage PushPublisher<Customer> pushPublisher = ReactiveStreamsIngestion.pushPublisher(); pushPublisher.subscribe(rsi.subscriber()); // Ad-hoc usage pushPublisher.accept( new Customer(1, "John Doe", "North")); // As a Consumer of a Stream Customer[] customers = … Stream .of(customers) .filter(c -> c.region.equals("NORTH")) .forEach(pushPublisher::accept); // As a Consumer for 3rd party Reactive Stream // Libraries eg: Reactor https://projectreactor.io/ Flux .just( new Customer(1, "John Doe", "North"), new Customer(2, "Jane Smith", "South")) .concatWithValues(new Customer(3, "Bob", "South")) .doOnComplete(() -> {System.out.println("Done!");}) .doOnCancel(() -> {System.out.println("Canceled!");}) .subscribe(pushPublisher::accept);
- 13. Copyright © 2023, Oracle and/or its affiliates From Sync to Reactive JDBC: Oracle R2DBC static String queryJdbc(java.sql.Connection connection) throws SQLException { try (java.sql.Statement statement = connection.createStatement()) { ResultSet resultSet = statement.executeQuery("SELECT * FROM CUSTOMERS"); if (resultSet.next()) return resultSet.getString(1); else throw new NoSuchElementException("Query returned zero rows"); } } static Publisher<String> queryR2dbc(io.r2dbc.spi.Connection connection) { return Flux.from(connection.createStatement( "SELECT * FROM CUSTOMERS") .execute()) .flatMap(result -> result.map(row -> row.get(0, String.class))) .switchIfEmpty(Flux.error( new NoSuchElementException("Query returned zero rows"))); }
- 14. Copyright © 2022, Oracle and/or its affiliates Demo # 3: Oracle R2DBC ⚫ Oracle Reactive Relational Database Connectivity (R2DBC) ⚫ Oracle R2DBC Driver is a Java library that supports reactive Programming with Oracle Database ⚫ It implements the R2DBC Service Provider Interface (SPI) as specified by the Reactive Relational Database Connectivity (R2DBC) ⚫ The R2DBC SPI exposes Reactive Streams as an abstraction for Remote database operations ⚫ The sample code uses Project Reactor. It could use RxJava, Akka, or any RS library ⚫ Runs on Java 11+
- 15. Virtual Threads or Reactive? • Oracle JDBC supports both! • Want Virtual Threads? • Oracle JDBC has been “Virtual Thread Compatible” since 21.1.0.0 • Want Reactive? • Oracle R2DBC 1.0.0 is available now • Consume Flow interfaces directly from Oracle JDBC’s Reactive Extensions Copyright © 2023, Oracle and/or its affiliates
- 16. Virtual Threads or Reactive? • Benefits of Virtual Threads: • Easier to read and write • Easier to debug • Integration with JDK tools • Do not alter the basic concurrency model of Java • Limitations of Virtual Threads: • Some libraries are not compatible • Still not in GA –> JDK 21 • JEP 425: Virtual Threads (Preview) • JEP 436: Virtual Threads (Second Preview) • JEP 444: Virtual Threads Copyright © 2023, Oracle and/or its affiliates Benefits of Reactive: • Reactive Libraries (Reactor, RxJava, Akka, Vert.x) • Stream-like API with a functional style • Low-level concurrency is handled for you (locks, atomics, queues). Limitations of Reactive: • Steep learning curve • Harder to read and write • Harder to debug
- 17. Copyright © 2023, Oracle and/or its affiliates MOM - Message-Oriented Middleware • A middleware component that allows communication and exchanges the data (messages). • It involves passing data between applications using a communication channel that carries self- contained units of information (messages). • In a MOM-based communication environment, messages are sent and received asynchronously. • MOM -> Apache ActiveMQ
- 18. AMQP – Advanced Message Queuing Protocol • The Advanced Message Queuing Protocol (AMQP) is an open standard for passing business messages between applications or organizations. • Realize the savings commoditization brings; remove vendor lock-in • Connect applications on different platforms; choose the right platform for the job • Connect to business partners using a full- featured open standard; remove technical barriers to trade • Position for innovations built upon the foundation of AMQP • Security, Reliability, Interoperability, Standard, Openness Copyright © 2023, Oracle and/or its affiliates
- 19. MQTT – Message Queuing Telemetry Transport • An OASIS standard messaging protocol for the Internet of Things (IoT). It is designed as an extremely lightweight publish/subscribe messaging transport that is ideal for connecting remote devices with a small code footprint and minimal network bandwidth. Copyright © 2023, Oracle and/or its affiliates
- 20. STOMP – Simple (or Streaming) Text Oriented Messaging Protocol • STOMP provides an interoperable wire format so that STOMP clients can communicate with any STOMP message broker to provide easy and widespread messaging interoperability among many languages, platforms and brokers.. Copyright © 2023, Oracle and/or its affiliates
- 21. Copyright © 2023, Oracle and/or its affiliates ActiveMQ & JMS Message Consumer ⚫ ActiveMQ ⚫ ActiveMQ is a Java-based open-source message broker which supports REST API and various wire-level protocols, such as MQTT, AMQP, and STOMP. ⚫ JMS Message Consumer ⚫ Like with any messaging-based application, you need to create a receiver that will handle the messages that have been sent. The sample code snippet creates an AMQP connection that connects to ActiveMQ using the given username, password, hostname, and port number. ⚫ Port number 5672 is the default port that ActiveMQ is listening to over the AMQP protocol when it starts up. It also creates a topic subscriber (consumer) to receive messages that have been published to the "event" topic.
- 22. Copyright © 2023, Oracle and/or its affiliates ActiveMQ & JMS Message Consumer
- 23. Copyright © 2023, Oracle and/or its affiliates Demo #4: Architecture
- 24. Copyright © 2023, Oracle and/or its affiliates Demo #4: Project Structure
- 25. Copyright © 2022, Oracle and/or its affiliates References • JDK 19 / Project Loom • JDK 19 - https://openjdk.org/projects/jdk/19/ • Loom - https://openjdk.org/projects/loom/ • JEP 425 Virtual Threads (Preview) - https://bugs.openjdk.org/browse/JDK-8277131 • Reactive Streams Ingestion Library • Getting Started with the Java library for Reactive Streams Ingestion (RSI) - https://bit.ly/3rEiRnC • High-throughput stream processing with the Java Library for Reactive Streams Ingestion (RSI), Virtual Threads, and the Oracle ATP Database - https://bit.ly/3rATCTd • RSI - https://docs.oracle.com/en/database/oracle/ • R2DBC • Oracle R2DBC Driver – https://github.com/oracle/oracle-r2dbc • Develop Java applications with Oracle Database • JDBC – https://www.oracle.com/database/technologies/appdev/jdbc.html
- 26. Oracle LiveLabs Showcasing how Oracle’s solutions can solve your business problems 500+ free workshops, available or in development 3.5 million people have already visited LiveLabs developer.oracle.com/livelabs learn something new …at your pace! 600+ events run using LiveLabs workshops
- 27. Create your FREE Cloud Account • Go to https://signup.cloud.oracle.c om/ Copyright © 2023, Oracle and/or its affiliates
- 28. 3 membership tiers Connect: @oracleace facebook.com/OracleACEs [email protected] 500+ technical experts & community leaders helping peers globally The Oracle ACE Program recognizes & rewards individuals for their technical & community contributions to the Oracle community Nominate yourself or a candidate: ace.oracle.com/nominate Learn more - ace.oracle.com blogs.oracle.com/ace
- 29. About me • Juarez Barbosa Junior - @juarezjunior • Senior Principal Java Developer Evangelist • Over 20 years of experience in IT • SW Engineering, Developer Relations • Microsoft, Oracle, IBM, Nokia, Unisys, Accenture, and a few startups • Microsoft Azure Developer Relations Lead • IBM Watson Tech Evangelist & Cloud Rockstar • IBM Mobile Tech Evangelist & Global Thought Leader • Nokia Developers Global Champion • Lead Software/DevOps Architect • Expertise • Java, Cloud, DevOps, Cloud-native, Blockchain Copyright © 2023, Oracle and/or its affiliates
- 30. Thank you! 30 Copyright © 2023, Oracle and/or its affiliates | Confidential: Internal