Aws based digital_transformation_platform

Enterprise Digital
Transformation Platform
Deep Dive and Demos
Slobodan Sipcic Ph.D.
Executive IT Architect
December 12, 2020 Enterprise Digital Transformation Platform 1

Agenda
• Business’ Ecosystems: Data and Integration Challenges
ü Digital Transformation Drivers
ü Digital Transformation Inhibitors
ü Digital Transformation Platform Northern Star Attributes
• How Platform Computing and EDTP Address Challenges
ü Enterprise Digital Transformation Platform
ü EDTP Layered Architecture
ü EDTP Layered Distributed Architecture
• EDTP Deep Dive
ü EDTP Fully Installed Through Automation on AWS
• Demo 1: Business Agility Through Automation of the
EDTP Full Stack Deployment
ü Demo 1a: ”Core Platform Pipeline”
ü Demo 1b: “Tech Components Pipeline”
ü Demo 1c: “Service Templates Pipeline”
• Demo 2: Accelerated Deployment of Digital Transformation
Solutions Based on the EDTP Data and Integration Services
ü Demo 2a: “Data Quality Through Harmonization”
ü Demo 2b: “Diverse Analytics”
ü Demo 2c: “Microservices Through Templating”
• Sample Use Cases:
ü Enterprise Logging Framework
ü Enterprise Interactions Logging Framework
ü Next Generation Campaign Management
ü Insurance Forms Automation
• Discussion and Next Steps
2
December 12, 2020 Enterprise Digital Transformation Platform

Businesses’ Ecosystem:
Data and Integration Challenges
3

Digital Transformation Drivers
Businesses may take on Digital
Transformation for several reasons. But
by far, the most likely reason is that
they have to: It is a survival issue.
A business’ ability to adapt quickly to
disruptions from incumbents and
startups, to time to market pressures,
and to rapidly changing customer
expectations has become critical.
It is about evolutions and changes
4

Digital Transformation Inhibitors
Data and integration challenges
hindering businesses’ ability to
transform:
1. Data Quality and Consistency,
2. Coupling on Data & BL levels,
3. Point-to-Point Integration n-
square problem,
4. Point-to-Point with imbedded
transformations,
5. Multiple runtime environments.
5

Digital Transformation Platform North Star Attributes
• Decoupling
• Adaptability:
⎯ Service Layer
⎯ Data Layer
⎯ Integration Layer
• Agility
• Elasticity
• Experimenting
• Fast-failing
6

How Platform Computing and
EDTP Address Challenges
7

Enterprise Digital Transformation Platform
To address the challenges we
developed EDTP for accelerated
development and deployment of a
wide range of agile Services and
Data Digital Transformation
Solutions (DTS) that scale.
The solutions are based on the
modern 4-tier architectural styles
including cloud, containers,
microservices, events, streaming,
and sync & async processing.
8

EDTP Layered Architecture
EDTP is built on top of core
technologies including Docker
Kubernetes and OpenShift:
• Docker provides container services,
• Kubernetes run and scale containers
for production,
• OpenShift provides developers with
the features to manage their DevOps,
• EDTP builds on to of it to deliver full-
stack automation and accelerators for
delivery of DTS.
9

EDTP Layered Distributed Architecture
EDTP has a microservices-based architecture of smaller, decoupled
units that work together. It runs on top of OpenShift and Kubernetes
clusters. The services are broken down by function:
• EDTP Technology Components and Services are deployed in
Containers - a virtual boundary of compute and memory resources
assigned to the components. The containers spin-up from Docker
images.
• OpenShift leverages the Kubernetes concept of a POD, which is one or
more Containers deployed together on one Worker Node, and the
smallest compute unit that can be defined, deployed, and managed.
• A Kubernetes Service serves as an internal load balancer. It exposes
an applications running on a set of PODs as a network service.
• An OpenShift Route is a way to expose a Service by giving it an
externally reachable hostname .
• The control plane, which is composed of Master Nodes, manages the
OpenShift cluster.
10

EDTP Deep Dive
11

EDTP Fully Installed Through Automation on AWS
• IaaC – deployed via Terraform.
• Technology Components
(extensible):
⎯ Ambassador,
⎯ Nifi,
⎯ Zookeeper,
⎯ Kafka &Kafka Streams,
⎯ Schema Registry,
⎯ MongoDB,
⎯ ELK.
• Services (configurable):
⎯ Harmonization,
⎯ Materialization,
⎯ Registration & Templates,
⎯ Ingress & Egress,
⎯ Connectors.
12

Demo 1:Business Agility Through
Automation of the EDTP Full
Stack Deployment
13

Demo 1: Scope
• Demo 1a – “Core Platform Pipeline”: deployment of the EDTP core platform components
(cloud agnostic infrastructure, Docker, Kubernetes and OpenShift) using Terraform.
• Demo 1b – “Tech Components Pipelines”: walkthrough the Jenkins pipelines for
deployments of the EDTP pluggable digital technology components (Ambassador, Nifi,
Zookeeper, Kafka, MongoDB, Istio and Elastic Stack).
• Demo 1c – “Service Templates Pipelines”: walkthrough the Jenkins pipelines for
deployments of the EDTP data and integration service templates (microservices, data
services, data streaming services, aggregation services, event processing services and
several integration patterns). The templates are used for accelerated deployments of
digital transformation solutions.
14

Demo 1a: Core Platform Pipeline
15
EDTP Core Components deployed.

Demo 1a: From Empty VPC
16

Demo 1a: To Core EDTP
17

Demo 1b: Tech Components Pipeline
18
Technology Components added to
the Core Platform

19
Demo 1b: From Core EDTP

Demo 1b: To Core EDTP & Tech Components
20

Demo 1c: Service Templates Pipeline
21
Technology Components added to
the Core Platform
Services added to the stack

Demo 1c: Keep Adding Services
22

Demo 2: “Accelerated Deployment of Digital
Transformation Solutions Based on the EDTP
Data and Integration Service Templates”
23

Demo 2: Scope
• Demo 2a – “Data Quality Through Harmonization”: deployment of a
data harmonization solution based on the EDTP data streaming
service template.
• Demo 2b – “Diverse Analytics”: deployment of a diverse analytics
solution based on the EDTP data streaming service template for
loading MongoDB JSON data to Snowflake tables for analytics.
• Demo 2c – “Microservices”: deployment of scalable microservices
from the EDTP templates.
24

Demo 2a: Data Quality Through Harmonization
We demonstrate how EDTP ingress a stream of raw records, harmonizes and materializes records in
MongoDB collections for consumption by others utilizing egress services. The solution is fully
configurable via AVRO schemas (raw, harmonized, and materialize). Several EDTP features are
highlighted during the demo:
• Processing of batch and streaming data including CDC,
• Data quality & master data through harmonization,
• Transforming streaming data in flight via configurable DSL,
• Data-Vault pattern and full audit trail,
• Meta-data management,
• Flexibility of JSON/AVRO schemas as opposed to complex CDM,
• Flexibility of an elastic search index for cataloging, browsing and searching data.
25

Demo 2a: EDTP Component Interactions
26

Demo 2b: Diverse Analytics
We demonstrate a diverse analytics solution based on the
EDTP data streaming service template for loading MongoDB
JSON data to Snowflake tables for analytics. The demo is an
extension of Demo 2a and it highlights several EDTP
integration features including:
• Distributed, evolutionary architecture,
• Extensibility via connectors – Snowflake Connector for Kafka,
• Integration via loose coupling of the right systems for the job.
27

Demo 2: Component Interaction View
1. Raw records
dropped into S3
1
2 3
4
2. Nifi service processes
records and publishes
on Raw topic
3. Harmonization service
processes raw data into
MongoDB and publishes
them on Harmonized
topic
4. Kafka Snowflake
Connector Service loads
data into Snowflake
table
28

Demo 2: From S3 to MongoDB to Snowflake
1. Raw records dropped into
S3
2. Nifi service processes
records and publishes on
Raw topic
3. Harmonization service
processes raw data into
MongoDB and publishes
them on Harmonized topic
4. Kafka Snowflake
Connector Service loads
data into Snowflake table
1
2
3
4
29

Demo 2c: Microservices Through Templating
We demonstrate deployment of scalable microservices from the EDTP templates. The demonstration
highlights several EDTP features including:
• Microservices architectural style,
• Decoupling of software components on both business capability and data levels (smart endpoints dumb pipes),
• Exposing services as REST APIs – EDTP as API Economy Platform,
• Four tier digital transformation platform (services tier, aggregation tier, delivery tier and client tier),
• Simplified integration due to loose coupling of software components including mainframe and legacy applications,
• Processing patterns: synchronous and asynchronous with or without retries,
• Event-Driven Architecture Patterns: event notification, event-carried state transfer, event-sourcing and CQRSP (command
query responsibility segregation pattern),
• Handlers – for implementation of client specific business logic,
• Full-stack CI/CD/CT and DevOps.
30

Demo 2c: Templating Idea
31
In Software Development we develop classes and
use them to instantiate objects.
In EDTP Service Development we develop
Templates and Registration Services for a given
Service Class and use them to instantiate new
services of the same class.

Demo 2c: Service Classes based on Patterns
Templating idea is especially powerful when combined with an idea of
using Patterns to define Service Classes
32
From Pattern to Service Class Template to Class Instances

Demo 2c: Examples of Patterns
EDTP provides templates for many Patterns including:
33
Synchronous Pattern Asynchronous Pattern Asynchronous Pattern with Retries

Demo 2c: Instantiating Service Instances Demo
We demonstrate how to instantiate new Service from Asynchronous Pattern
with Retries
34
Asynchronous Pattern with Retries EDTP Service Class Template
Generic Schema
Demo will create
A Service Instance
Generic schema
includes the required
fields, such as:
• TenantID,
• Version #,
• CorelationID.
• Section for Service
Data.

Use Case Samples
35

Enterprise Logging Framework
36

Enterprise Interactions Logging Framework
37

Next Generation Campaign Management
38

Insurance Forms Automation
39
Business
Problem
Solution:
Call to two Async
Services with
Retries
Instantiate Services
via Templating
Solution Deployed
Time and Money Saved

Next Step: Request Demonstration
Email Request to: ssipcic@gmail.com
40

Request Demonstration
The following EDTP features can be demonstrate on request:
1. Automated Deployment of EDTP in Client’s AWS environment including:
a. Infrastructure,
b. Technology Components,
c. Services.
2. EDTP Data and Integration Capabilities including:
a. Ingestion of streaming and batch data,
b. Harmonization and Materialization of the streaming data,
c. Integration based on the EDTP Connector Services – such integration with Snowflake,
d. Service templating for accelerated deployment of enterprise services.
3. EDTP Support for Pattern Based Service Development including:
a. Event Notification,
b. Event-Carried State Transfer,
c. Event-Sourcing,
d. CQRS,
e. Asynchronous Processing,
f. Service Retries Processing,
g. and many more…
41

Aws based digital_transformation_platform

More Related Content

What's hot (18)

Similar to Aws based digital_transformation_platform (20)

Recently uploaded (20)

Aws based digital_transformation_platform