Application modernization patterns with apache kafka, debezium, and kubernetes summit 2021
3 likes830 views
The document discusses application modernization through patterns using Apache Kafka, Debezium, and Kubernetes to transition from monolithic to microservices architectures. It highlights the challenges of monolithic applications, benefits of modernization, and effective strategies like the strangler pattern and outbox/saga patterns for managing data and service communication. Ultimately, it emphasizes the importance of leveraging open-source technologies for gradual and low-risk migration towards cloud-native environments.
Application modernization patterns with apache kafka, debezium, and kubernetes summit 2021
- 1. Application modernization patterns with Apache Kafka, Debezium, and Kubernetes Bilgin Ibryam @bibryam Product Manager Red Hat
- 2. 2 We build our computers the way we build our cities - over time, without a plan, on top of ruins. Ellen Ullman
- 3. About me 3 ▸ Product Manager at Red Hat ▸ Former Principal Architect ▸ Committer at Apache Camel ▸ Author ･ Camel Design Patterns ･ Kubernetes Patterns ▸ @ bibryam ▸ https://www.ofbizian.com
- 4. About this talk 4 Application migration/modernization Rs ▸ Rehosting - lift-and-shift ▸ Replatorming - lift, tinker, and shift ▸ Refactor / Re-architect ･ Debezium ･ Apache Kafka ･ Kubernetes
- 5. Starting point 5 Challenges with monolithic applications ▸ Frequent deployment is difficult ▸ Obstacle to scaling development ▸ Scaling the application can be difficult Expected modernization benefits ▸ Reduce time to market ▸ Greater team autonomy ▸ Improved automation ▸ Increased application performance
- 6. Target state 6 Cloud native microservices’ characteristics ▸ Independently deployable ▸ Modeled around a business domain ▸ Own their data ▸ Emit events (based on EDA) ▸ Built on open source technology such as Kubernetes, Apache Kafka, Debezium Measuring the results ▸ Lead time for change ▸ Deployment frequency ▸ Mean time to recovery
- 8. Strangler Pattern 8 High-level steps ▸ Identify functional boundary ▸ Migrate/reimplement functionality ▸ Synchronize data ▸ Reroute traffic to new service ▸ Decomposition old service Main benefits ▸ No big bang migration ▸ Minimal changes to the monolith ▸ Low risk with the ability to fall back
- 9. Functional boundary 9 Functional boundary considerations ▸ Aggregates, bounded context based on DDD ▸ Event Storming technique by A. Brandolini ▸ Data model coupling ▸ Module dependencies Where to start from? ▸ Easy win ▸ Noticeable improvement ▸ A representative effort
- 10. Interception options 10 Prerequisites ▸ Ability to identify inbound calls ▸ Ability to direct calls to new service (and back) Routing options ▸ Client redirection ▸ Transparent HTTP proxy ▸ Shared protocol translation proxy ▸ Embedded/sidecar proxy
- 11. Interception options 11 Prerequisites ▸ Ability to identify inbound calls ▸ Ability to direct calls to new service (and back) Routing options ▸ Client redirection ▸ Transparent HTTP proxy ▸ Shared protocol translation proxy ▸ Embedded/sidecar custom proxy
- 12. Database strangulation 12 Database first Database and code together Code first
- 13. Data synchronization 13 Cons ▸ Polling delay, might miss updates ▸ Polling overhead ▸ Deletes not captured ▸ Not transparent to writing application Pros ▸ Simple Installation and configuration Cons ▸ Varying capabilities ▸ Limited integration capabilities ▸ Not portable ▸ Not easily testable ▸ Not transparent to source database Pros ▸ No additional application required Cons ▸ Requires specialized tools ▸ Requires database configuration Pros ▸ All changes (deletes) are captured ▸ No polling delay or overhead ▸ Transparent to writing application Triggers Log readers Queries/Polling
- 14. Debezium 14 What is Debezium? ▸ Open source Change Data Capture platform ▸ Snapshotting, filtering, transformations ▸ MySQL, PostgreSQL, MongoDB, SQL Server, Db2, Oracle, Vitess, Cassandra ▸ Use cases: data replication, cache updates, search index updates, audit logs, sync data between services
- 15. Strangler Pattern with Debezium 15 Strangler Pattern with Debezium ▸ Transparent to the writing application ▸ Initial bulk import through snapshotting ▸ Schema, table, column filtering ▸ Single Message Transformations as anti-corruption layer ▸ Schema Registry for schema validation and compatibility enforcement of data models Debezium with Apache Kafka ▸ Guaranteed ordering, message compaction ▸ Pull based - re-read from start, or new changes ▸ Kafka Connect - offset tracking, fail over
- 16. Release steps 16 Steps so far ▸ Identified a functional boundary ▸ Migrated it as a new service ▸ Created a new data model in a new database ▸ Automated and deployed everything ▸ Bulk imported data and kept it in-sync ▸ No traffic routed to the new service
- 17. Release steps 17 Next step ▸ Route READ traffic to the new service
- 18. Release steps 18 Next steps ▸ Route READ and WRITE traffic to the new service ▸ Two-way data synchronization in progress
- 19. Release steps 19 Next steps ▸ Stop WRITE traffic to legacy system ▸ Stop data synchronization from legacy system
- 20. Release steps 20 Final steps ▸ STOP READ traffic to the legacy application ▸ Stop data synchronization to legacy system ▸ Decommission migrated module
- 21. New Challenges
- 22. Modernization challenges 22 Migration challenges ▸ Low-risk changes with demonstrable progress ▸ Deliver enhancements and new “business value” ▸ Simultaneously delivering new services Distributed systems challenges ▸ Automation and operations at scale ▸ Dual-writes and long-running business transactions ▸ Analytical and reporting data needs
- 23. Operating event-driven services at scale 23 Microservices on Kubernetes ▸ Deployment/rollback ▸ Placement/scheduling ▸ Configuration management ▸ Resource/failure isolation ▸ Auto heal/upgrade (through operators) Event-driven applications on Kubernetes ▸ Strimzi - operating Apache Kafka cluster ▸ KEDA - event-driven workload autoscaling ▸ Debezium - change data capture platform ▸ Knative Eventing - event-driven primitives
- 24. Outbox Pattern 24 Offers an approach for services to update their data store and notify other services in a reliable and eventually consistent manner. ▸ Addresses the dual-write problem ▸ Offers “read your own writes" semantics ▸ “at-least-once” semantics for consumers
- 25. Saga Pattern 25 Splits up long-running business transactions into a series of multiple local transactions. When implemented using the Outbox Pattern, it offers the benefits of orchestration and asynchronous communication. ▸ Orchestration based coordination ▸ Asynchronous communication with Apache Kafka
- 26. Summary 26 Modern software systems are like cities - they evolve over time, on top of legacy systems. Using proven patterns and standardized tools help create long-lasting systems that grow with your needs. ▸ Strangler Pattern - a low-risk application migration technique ▸ Outbox/Saga Patterns - reliable inter-service communication approach ▸ Debezium - a non-intrusive toolking for change data capture ▸ Kubernetes / Strimzi / Apache Kafka - de facto standards in their fields
- 27. Resources 27 ▸ Monolith To Microservices ▸ Building Event-Driven Microservices ▸ Designing Data-Intensive Applications ▸ Kubernetes Patterns ▸ Debezium.io ▸ Keda.sh ▸ Strimzi.io ▸ github.com/windup
- 28. 28 Red Hat OpenShift Streams for Apache Kafka a fully managed Apache Kafka service by Red Hat http://red.ht/TryKafka Try Apache Kafka in seconds