SlideShare a Scribd company logo
BASEL | BERN | BRUGG | BUCHAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENEVA
HAMBURG | COPENHAGEN | LAUSANNE | MANNHEIM | MUNICH | STUTTGART | VIENNA | ZURICH
http://guidoschmutz.wordpress.com@gschmutz
Location Analytics
Real-Time Geofencing using Kafka
Guido Schmutz
UKOUG Techfest 2019
updated
for ksqlDB
Agenda
1. Introduction & Motivation
2. Implementing Geo Fencing with Kafka
• Using Oracle Stream Analytics
• Using ksqlDB
• Using Kafka Streams
• Using Tile38
3. Visualization using ArcadiaData
4. Summary
BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF
HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH
Guido
Working at Trivadis for more than 22 years
Consultant, Trainer, Platform Architect for Java,
Oracle, SOA and Big Data / Fast Data
Oracle Groundbreaker Ambassador & Oracle ACE
Director
@gschmutz guidoschmutz.wordpress.com
176th
edition
Introduction
Geofencing – What is it?
Use of GPS or RFID technology to create a
virtual geographic boundary, enabling
software to trigger a response when an
object/device enters or leaves a particular
area
Possible Events
• ENTER
• EXIT
• OUTSIDE
• lNSIDE
Source: https://tile38.com
Apache Kafka – A Streaming Platform
Source
Connector
Sink
Connector
trucking_
driver
ksqlDB Engine
Kafka Streams
Kafka Broker
ksqlDB
Apache Kafka
Kafka Cluster
Consumer 1 Consume 2r
Broker 1 Broker 2 Broker 3
Zookeeper
Ensemble
ZK 1 ZK 2ZK 3
Schema
Registry
Service 1
Management
Control Center
Kafka Manager
KAdmin
Producer 1 Producer 2
kafkacat
Data Retention:
• Never
• Time (TTL) or Size-based
• Log-Compacted based
Producer3Producer3
ConsumerConsumer 3
Apache Kafka – How does it scale?
• horizontally scalable, guaranteed order
Streaming Analytics: ksqlDB or Kafka Streams
Stream
• unbounded sequence of structured data
("facts")
• Facts in a stream are immutable
Table
• collected state of a stream
• Latest value for each key in a stream
• Facts in a table are mutable
ksqlDB: Stream Processing with zero
coding using SQL-like language (now
supporting push and pull queries)
Kafka Streams: Java library providing
stream analytics capabilities
trucking_
driver
Kafka Broker
ksqlDB Engine
Kafka Streams
ksqlDB REST
Commands
ksqlDB CLI
push pull
Stream vs. Table
Vehicle Position Stream
id latitude longitude
1 52.3924 13.0514
id latitude longitude
1 52.3924 13.0514
4 38.4847 -90.23345
id latitude longitude
1 52.3924 13.0514
4 38.4847 -90.23345
1 52.39052 13.06455
id name geometry_wkt
10 St. Louis POLYGON ((13.297920227050781
52.56195151687443, …))
11 Berlin POLYGON ((-90.23345947265625
38.484769753492536,…))
id name geometry_wkt
10 St. Louis POLYGON ((13.297920227050781
52.56195151687443, …))
id name geometry_wkt
10 St. Louis
(US)
POLYGON ((13.297920227050781
52.56195151687443, …))
11 Berlin
(GE)
POLYGON ((-90.23345947265625
38.484769753492536,…))
GeoFence Table
First "naïve " idea when answering the CFP
• geofence being an UDF (User Defined Function)
• would need access to the geofences defined somewhere
• what about updates?
• a function should not cause any side-effects
SELECT geofence(latitude, longitude) geo_event
FROM geo_position_stream
Dash
board
High Level Overview of Test Case
geofence
Join Position
& Geofences
Vehicle
Position
vehicle
position
pos &
geofences
Geo
fencing
geofence
status
key=10
{ "id" : "10", "latitude" : 38.35821, "longitude" : -90.15311}
key=3
{"id":3,"name":"Berlin, Germany","geometry_wkt":"POLYGON
((13.297920227050781 52.56195151687443,
…))","last_update":1560607149015}
Geofence
Mgmt
Vehicle
Position
Weather
Service
Geo-Processing
Well-known text (WKT) is a text markup language for representing vector geometry
objects on a map
GeoTools is a free software GIS toolkit for developing
standards compliant solutions
Implementing Geo Fencing -
using Oracle Stream Analytics
Oracle Stream Analytics (OSA)
• Expressive Patterns Library,
including Geo Processing
• Abstracted visual façade to
interrogate live real time
streaming data
• Integrate with Apache Kafka
• Runs on top of Spark Cluster
as a Spark Streaming Job
Implementing Geo Fencing -
using ksqlDB
ksqlDB – Streams and Tables
geofence
Table
vehicle
position
Stream
KSQL
Geofencing
id latitude longitude
1 52.3924 13.0514
4 38.4847 -90.23345
3 .. ..
id name geometry_wkt
10 St. Louis POLYGON ((13.297920227050781
52.56195151687443, …))
11 Berlin POLYGON ((-90.23345947265625
38.484769753492536,…))
12 xxxxx POLYGON ((…))
ksqlDB – Streams and Tables
geofence
Table
vehicle
position
Stream
CREATE STREAM vehicle_position_s
(id VARCHAR,
latitude DOUBLE,
longitude DOUBLE)
WITH (KAFKA_TOPIC='vehicle_position',
VALUE_FORMAT='DELIMITED');
CREATE TABLE geo_fence_t
(id BIGINT,
name VARCHAR,
geometry_wkt VARCHAR)
WITH (KAFKA_TOPIC='geo_fence',
VALUE_FORMAT='JSON',
KEY = 'id');
KSQL
Geofencing
How to determine "inside" or "outside" geofence?
Only one standard UDF for geo processing in KSQL: GEO_DISTANCE
Implement custom UDF using functionality from GeoTools Java library
public String geo_fence(final double latitude, final double longitude,
final String geometryWKT){ .. }
public List<String> geo_fence_bulk(final double latitude
, final double longitude, List<String> idGeometryListWKT) { .. }
ksql> SELECT geo_fence(latitude, longitude, 'POLYGON ((13.297920227050781
52.56195151687443, 13.2440185546875 52.530216577830124, ...))')
FROM test_geo_udf_s;
52.4497 | 13.3096 | OUTSIDE
52.4556 | 13.3178 | INSIDE
Custom UDF to determine if Point is inside a geometry
@Udf(description = "determines if a lat/long is inside or outside the
geometry passed as the 3rd parameter as WKT encoded ...")
public String geo_fence(@UdfParameter(value="latitude") final double latitude,
@UdfParameter(value="longitude") ) final double longitude,
@UdfParameter(value="geometryWKT") ) final String geometryWKT) {
String status = "";
GeometryFactory geometryFactory = JTSFactoryFinder.getGeometryFactory();
WKTReader reader = new WKTReader(geometryFactory);
Polygon polygon = (Polygon) reader.read(geometryWKT);
Coordinate coord = new Coordinate(longitude, latitude);
Point point = geometryFactory.createPoint(coord);
if (point.within(polygon)) {
status = "INSIDE";
} else {
status = "OUTSIDE";
}
return status;
}
1) Using Cross Join
geofence
Table
Join Position
& Geofences
vehicle
position
Stream
Stream
pos &
geofences
CREATE STREAM vp_join_gf_s
AS
SELECT vp.id,
geo_fence(vp.latitude,
vp.longitude,
gf.geometry_wkt) status
FROM vehicle_position_s AS vp
CROSS JOIN geo_fence_t AS gf
There is no Cross Join in ksqlDB!
id latitude longitude
1 52.3924 13.0514
4 38.4847 -90.23345
3 .. ..
id name geometry_wkt
10 St. Louis POLYGON ((13.297920227050781
52.56195151687443, …))
11 Berlin POLYGON ((-90.23345947265625
38.484769753492536,…))
12 xxxxx POLYGON ((…))
X
2) INNER Join
geofence
Stream
Join Position
& Geofences
vehicle
position
Stream
Stream
pos &
geofences
{ "group":1", "name":"Berlin", "geometry_wkt":"POLYGON ((-
90.23345947265625 38.484769753492536,…))",
"last_update":1560607149015}
Enrich Group
Table
geofences
by group 1
Enrich Group
Stream
postion by
group 1
Cannot insert into Table
from Stream
>INSERT INTO geo_fence_t
>SELECT '1' AS group_id, geof.id, …
>FROM geo_fence_s geof;
INSERT INTO can only be used to insert into
a stream. A02_GEO_FENCE_T is a table.
{ "group":"1", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514}
gid id latitude longitude
1 1 52.3924 13.0514
1 4 38.4847 -90.23345
1 3 .. ..
gid id name geometry_wkt
1 10 St. Louis POLYGON ((13.297920227050781
52.56195151687443, …))
1 11 Berlin POLYGON ((-90.23345947265625
38.484769753492536,…))
{ "group":1", "name":"St. Louis", "geometry_wkt":"POLYGON
((13.297920227050781 52.56195151687443, …))",
"last_update":1560607149015}
3) Geofences aggregated in one group
Join Position
& Geofences
Stream
geofence
event
Geofences
aggby group
Table
{ "group":"1", [ "1:POLYGON ((13.297920227050781
52.56195151687443, …))","2:POLYGON ((-
90.23345947265625 38.484769753492536,…))" ] }
geo_fence_bulk
geofence
Stream
vehicle
position
Stream
{ "group":1", "name":"St. Louis",
"geometry_wkt":"POLYGON ((13.297920227050781
52.56195151687443, …))",
"last_update":1560607149015}
Enrich With
Group-1
Stream
geofences
by group 1
Enrich With
Group-1
Stream
postion by
group 1
geofences
by group 1
high low
low high
low high
Scalable
Latency
"Code Smell"
medium
medium
medium
{ "group":"1", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514}
{ "group":1", "name":"Berlin", "geometry_wkt":"POLYGON
((-90.23345947265625 38.484769753492536,…))",
"last_update":1560607149015}
gid id latitude longitude
1 1 52.3924 13.0514
1 4 38.4847 -90.23345
1 3 .. ..
gid geometry_wkt_list
1 1:POLYGON ((13.297920227050781
52.56195151687443, …))
2:POLYGON ((-90.23345947265625
38.484769753492536,…))
X:POLGON ((…))
always only one row
3) Geofences aggregated in one group
CREATE TABLE a03_geo_fence_aggby_group_t
AS
SELECT group_id
, collect_set(id + ':' + geometry_wkt) AS id_geometry_wkt_list
FROM a03_geo_fence_by_group_s geof
GROUP BY group_id;
CREATE STREAM a03_vehicle_position_by_group_s
AS
SELECT '1' group_id, vehp.id, vehp.latitude, vehp.longitude
FROM vehicle_position_s vehp
PARTITION BY group_id;
3) Geofences aggregated in one group
CREATE STREAM a03_geo_fence_status_s
AS
SELECT vehp.id, vehp.latitude, vehp.longitude,
geo_fence_bulk(vehp.latitude, vehp.longitude,
geofaggid_geometry_wkt_list) AS geofence_status
FROM a03_vehicle_position_by_group_s veh
LEFT JOIN a03_geo_fence_aggby_group_t geofagg
ON vehp.group_id = geofagg.group_id;
ksql> SELECT * FROM a03_geo_fence_status_s;
46 | 52.47546 | 13.34851 | [1:OUTSIDE, 3:INSIDE]
46 | 52.47521 | 13.34881 | [1:OUTSIDE, 3:INSIDE]
...
As many as there are geo-fences
Geo Hash for a better distribution
Geohash is a geocoding which
encodes a geographic location
into a short string of letters and
digits
Length Area width x height
1 5,009.4km x 4,992.6km
2 1,252.3km x 624.1km
3 156.5km x 156km
4 39.1km x 19.5km
12 3.7cm x 1.9cm
http://geohash.gofreerange.com/
Geo Hash Custom UDF
ksql> SELECT latitude, longitude, geo_hash(latitude, longitude, 3)
>FROM vehicle_position_s;
38.484769753492536 | -90.23345947265625 | 9yz
public String geohash(final double latitude,
final double longitude, int length)
public List<String> neighbours(String geohash)
public String adjacentHash(String geohash, String directionString)
public List<String> coverBoundingBox(String geometryWKT, int length)
ksql> SELECT name, wkt, geo_hash(geometry_wkt, 3) FROM a04_geo_fence_s;
St. Louis, POLYGON ((-90.25749206542969 38.71551876930462, -90.31723022460938
38.69301319283493, ...)) | [9yz]
ksql> SELECT name, wkt, geo_hash(geometry_wkt, 4) FROM a04_geo_fence_s;
St. Louis, POLYGON ((-90.25749206542969 38.71551876930462, -90.31723022460938
38.69301319283493, ...)) | [9yzg, 9yzu]
4) Geofences aggregated by GeoHash
Join Position
& Geofences
Stream
geofence
event
Geofences
gpby geohash
Table
{ "geohash":"u33", ["2:POLYGON ((-
90.23345947265625 38.484769753492536,…))"], …}
geo_fence()
geofence
Table
vehicle
position
Stream
{ "geohash":"9yz", "name":"St. Louis", "geometry_wkt":"POLYGON
((13.297920227050781 52.56195151687443, …))",
"last_update":1560607149015}}
Enrich with
GeoHash
Stream
geofences
& geohash
Enrich with
GeoHash
Stream
position &
geohash
geofences
by geohash
geo_hash()
geo_hash()
{ "geohash":"u33", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514}
{ "group":"u33", "name":"Berlin", "geometry_wkt":"POLYGON ((-
90.23345947265625 38.484769753492536,…))",
"last_update":1560607149015}
{ "geohash":"dnb", "name":"St. Louis", "geometry_wkt":"POLYGON
((13.297920227050781 52.56195151687443, …))",
"last_update":1560607149015}}
{ "geohash":"9yz", [ "1:POLYGON
((13.297920227050781 52.56195151687443, …))", ..]}
{ "geohash":"dnb", [ "1:POLYGON
((13.297920227050781 52.56195151687443, …))",..]}
geohash id latitude longitude
u33 1 52.3924 13.0514
9yz 4 38.4847 -90.23345
u34 3 .. ..
geohash geometry_wkt_list
9yz 1:POLYGON ((13.297920227050781 52.56195151687443, …))
N:POLYGON (( …. ))
u33 2:POLYGON ((-90.23345947265625 38.484769753492536,…))
dnb 1:POLYGON ((13.297920227050781 52.56195151687443, …))
high low
low high
low high
Scalable
Latency
"Code Smell"
medium
medium
medium
4) Geofences aggregated by GeoHash
CREATE STREAM a04_geo_fence_by_geohash_s
AS SELECT EXPLODE(geo_hash(geometry_wkt, 3)) geo_hash,
id,
name,
geometry_wkt
FROM a04_geo_fence_s
PARTITION by geo_hash;
There was no explode() / unpivot() functionality in KSQL but now
ksqlDB provides it!
geofence
Table
Enrich with
GeoHash
Stream
geofence &
geohash
ksql> SELECT name, geo_hash FROM a04_geo_fence_and_geohash_s EMIT CHANGES;
| Colombia, Missouri | 9yz
| Berlin, Germany | u33
| St. Louis, Missouri | 9yz
4) Geofences aggregated by GeoHash
CREATE TABLE a04_geo_fence_by_geohash_t
AS
SELECT geohash,
COLLECT_SET(geometry_wkt) AS id_geometry_wkt_list,
COLLECT_SET(id) AS id_list
FROM a04_geo_fence_and_geohash_s
GROUP BY geo_hash;
Geofences
gpby geohash
Table
Stream
geofences
& geohash
geofences
by geohash
ksql> SELECT * FROM a04_geo_fence_by_geohash_t EMIT CHANGES;
| 9yz | [POLYGON ((13.297920227050781 52.56195151687443, …)), POLYGON((…))] |[1,N]
| u33 | [POLYGON ((-90.23345947265625 38.484769753492536] |[2]
| dnb | [POLYGON ((13.297920227050781 52.56195151687443, …)), POLYGON((…))] |[1]
...
4) Geofences aggregated by GeoHash
CREATE STREAM a04_vehicle_position_by_geohash_s
AS
SELECT vp.id, vp.latitude, vp.longitude,
geo_hash(vp.latitude, vp.longitude, 3) geo_hash
FROM vehicle_position_s vp
PARTITION BY geo_hash;
vehicle
position
Stream
Enrich with
GeoHash
Stream
position &
geohash
ksql> SELECT * FROM a04_vehicle_position_by_geohash_s EMIT CHANGES;
| 10 | 52.4497 | 13.3096 | u33
| 11 | 38.521846880854966 | -90.19912719726561 | 9yz
...
4) Geofences aggregated by GeoHash
geohash id latitude longitude
u33 1 52.3924 13.0514
9yz 4 38.4847 -90.23345
nnn 3 .. ..
geohash geometry_wkt_list
9yz 1:POLYGON ((13.297920227050781 52.56195151687443, …))
N:POLYGON (( …. ))
u33 2:POLYGON ((-90.23345947265625 38.484769753492536,…))
dnb 1:POLYGON ((13.297920227050781 52.56195151687443, …))
geoh id latitude longitude geometry_wkt_list
u33 1 52.3924 13.0514 1:POLYGON ((13.297920227050781 52.56195151687443, …))
9yz 4 38.4847 -90.23345 1:POLYGON ((13.297920227050781 52.56195151687443, …))
N:POLYGON (( …. ))
nnn 3 .. ..
4) Geofences aggregated by GeoHash
CREATE STREAM a04_geo_fence_status_s
AS
SELECT vp.id, vp.latitude, vp.longitude, vp.geo_hash,
explode (gf.id_list) AS geofenceId,
geo_fence (vp.latitude,
vp.longitude,
explode (gf.wkt_list)) AS geo_event
FROM a04_vehicle_position_by_geohash_s vp
LEFT JOIN a04_geo_fence_by_geohash_t gf
ON (vp.geo_hash = gf.geo_hash);
ksql> SELECT * FROM a04_geo_fence_status_s;
| 10 | 52.4497 | 13.3096 | u33 | 3 | OUTSIDE
| 11 | 38.521846880854966 | -90.19912719726561 | 9yz | 2 | OUTSIDE
| 11 | 38.521846880854966 | -90.19912719726561 | 9yz | 1 | OUTSIDE
...
Join Position
& Geofences
Stream
geofence
event
Berne
Fribourg
It works …. but ….
• Becaue of re-partitioning by
geohash we lose the guaranteed
order for a given vehicle
• Can be problematic, if there is a
backlog in one of the
topics/partitions
u0m5
u0m4
u0m7
u0m6
Consumer 1 Consumer 2
Implementing Geo Fencing -
using Kafka Streams
Geo-Fencing with Kafka Streams and Global KTable
Enrich Position with
GeoHash & Join
with Geofences
Global
KTable
geofence
KTable
vehicle
position
{ "geohash":u33", "name":"Potsdam",
"geometry_wkt":"POLYGON ((5.668945 51.416016, …))",
"last_update":1560607149015}
Enrich and Group
by GeoHash
matched
geofences
Detect Geo
Event
geofence_
status
high low
low high
low high
Scalable
Latency
"Code Smell"
medium
medium
medium
geofence
by geohash
{"id":"10", "latitude" : 52.3924,
"longitude" : 13.0514, [
{"name":"Berlin"} ] }
{ "geohash":"u33", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514}
{"id":"10", "status" : "ENTER", "geofenceName":"Berlin"} }
position &
geohash
{ "geohash":u33", "name":"Berlin",
"geometry_wkt":"POLYGON ((-90.23345947265625
38.484769753492536,…))",
"last_update":1560607149015}
{ "geohash":"u33", [ {"name":"Berlin",
"geometry_wkt":"POLYGON ((-90.23345947265625
38.484769753492536,…))"},{"name":"Potsdam",
"geometry_wkt":"POLYGON ((5.668945 51.416016,
…))"} ] }
{ "geohash":"9yz", [ {"name":"St. Louis",
"geometry_wkt":"POLYGON ((13.297920227050781
52.56195151687443, …))"} ] }{ "geohash":"9yz", "name":"St. Louis",
"geometry_wkt":"POLYGON ((13.297920227050781
52.56195151687443, …))",
"last_update":1560607149015}}
Geo-Fencing with Kafka Streams and Global KTable
KStream<String, GeoFence> geoFence = builder.stream(GEO_FENCE);
KStream<String, GeoFence> geoFenceByGeoHash =
geoFence.map((k,v) -> KeyValue.<GeoFence, List<String>> pair(v,
GeoHashUtil.coverBoundingBox(v.getWkt().toString(), 5)))
.flatMapValues(v -> v)
.map((k,v) -> KeyValue.<String,GeoFence>pair(v, createFrom(k, v)));
KTable<String, GeoFenceList> geofencesByGeohash =
geoFenceByGeoHash.groupByKey().aggregate(
() -> new GeoFenceList(new ArrayList<GeoFenceItem>()),
(aggKey, newValue, aggValue) -> {
GeoFenceItem geoFenceItem = new
GeoFenceItem(newValue.getId(), newValue.getName(),
newValue.getWkt(), "");
if (!aggValue.getGeoFences().contains(geoFenceItem))
aggValue.getGeoFences().add(geoFenceItem);
return aggValue;
},
Materialized.<String, GeoFenceList,
KeyValueStore<Bytes,byte[]>>as("geofences-by-geohash-store"));
geofencesByGeohash.toStream().to(GEO_FENCES_KEYEDBY_GEOHASH,
Produced.<String, GeoFenceList> keySerde(stringSerde));
Geo-Fencing with Kafka Streams and Global KTable
final GlobalKTable<String, GeoFenceList> geofences =
builder.globalTable(GEO_FENCES_KEYEDBY_GEOHASH);
KStream<String, VehiclePositionWithMatchedGeoFences> positionWithMatchedGeoFences =
vehiclePositionsWithGeoHash.leftJoin(geofences,
(k, pos) -> pos.getGeohash().toString(),
(pos, geofenceList) -> {
List<MatchedGeoFence> matchedGeofences = new ArrayList<MatchedGeoFence>();
if(geofenceList != null) {
for (GeoFenceItem geoFenceItem : geofenceList.getGeoFences()) {
boolean geofenceStatus =
GeoFenceUtil.geofence(pos.getLatitude(), pos.getLongitude(),
geoFenceItem.getWkt().toString());
if(geofenceStatus)
matchedGeofences.add(new MatchedGeoFence(geoFenceItem.getId(),
geoFenceItem.getName(), null));
}
}
return new VehiclePositionWithMatchedGeoFences(pos.getVehicleId(), 0L,
pos.getLatitude(), pos.getLongitude(),
pos.getEventTime(), matchedGeofences);
});
Implementing Geo Fencing -
using Tile38
Tile38
• https://tile38.com
• Open Source Geospatial Database & Geofencing Server
• Real Time Geofencing
• Roaming Geofencing
• Fast Spatial Indices
• Pluggable Event Notifications
Tile38 – How does it work?
> SETCHAN berlin WITHIN vehicle FENCE OBJECT
{"type":"Polygon","coordinates":[[[13.297920227050781,52.56195151687443],[1
3.2440185546875,52.530216577830124],[13.267364501953125,52.45998421679598],
[13.35113525390625,52.44826791583386],[13.405036926269531,52.44952338289473
],[13.501167297363281,52.47148826410652], ...]]}
> SUBSCRIBE berlin
{"ok":true,"command":"subscribe","channel":"berlin","num":1,"elapsed":"5.85
µs"}
.
.
.
{"command":"set","group":"5d07581689807d000193ac33","detect":"outside","hoo
k":"berlin","key":"vehicle","time":"2019-06-
17T09:06:30.624923584Z","id":"10","object":{"type":"Point","coordinates":[1
3.3096,52.4497]}}
SET vehicle 10 POINT 52.4497 13.3096
Tile38 – How does it work?
> SETHOOK berlin_hook kafka://broker-1:9092/tile38_geofence_status WITHIN
vehicle FENCE OBJECT
{"type":"Polygon","coordinates":[[[13.297920227050781,52.56195151687443],[1
3.2440185546875,52.530216577830124],[13.267364501953125,52.45998421679598],
[13.35113525390625,52.44826791583386],[13.405036926269531,52.44952338289473
],[13.501167297363281,52.47148826410652], ...]]}
bigdata@bigdata:~$ kafkacat -b localhost -t tile38_geofence_status
% Auto-selecting Consumer mode (use -P or -C to override)
{"command":"set","group":"5d07581689807d000193ac34","detect":"outside","hoo
k":"berlin_hook","key":"vehicle","time":"2019-06-
17T09:12:00.488599119Z","id":"10","object":{"type":"Point","coordinates":[1
3.3096,52.4497]}}
SET vehicle 10 POINT 52.4497 13.3096
1) Enrich with GeoFences – aggregated by geohash
geofence
Stream
vehicle
position
Stream
Invoke UDF
{ "id" : "10", "latitude" : 38.35821, "longitude" : -90.15311}
Invoke UDF
Geofence
Service
geofence
status
set_pos()
set_fence()
Stream
null
high low
low high
low high
Scalable
Latency
"Code Smell"
medium
medium
medium
{ "id":11", "name":"Berlin", "geometry_wkt":"POLYGON ((-
90.23345947265625 38.484769753492536,…))",
"last_update":1560607149015}
{ "id":10", "name":"St. Louis", "geometry_wkt":"POLYGON
((13.297920227050781 52.56195151687443, …))",
"last_update":1560607149015}
2) Using Custom Kafka Connector for Tile38
geofence
vehicle
position
Geofence
Service
kafka-to-
tile38
kafka-to-
tile38
geofence
status
high low
low high
low high
Scalable
Latency
"Code Smell"
medium
medium
medium
{ "id":11", "name":"Berlin", "geometry_wkt":"POLYGON ((-
90.23345947265625 38.484769753492536,…))",
"last_update":1560607149015}
{ "id":10", "name":"St. Louis", "geometry_wkt":"POLYGON
((13.297920227050781 52.56195151687443, …))",
"last_update":1560607149015}
{ "id" : "10", "latitude" : 38.35821, "longitude" : -90.15311}
2) Using Custom Kafka Connector for Tile38
curl -X PUT 
/api/kafka-connect-1/connectors/Tile38SinkConnector/config 
-H 'Content-Type: application/json' 
-H 'Accept: application/json' 
-d '{
"connector.class":
"com.trivadis.geofence.kafka.connect.Tile38SinkConnector",
"topics": "vehicle_position",
"tasks.max": "1",
"tile38.key": "vehicle",
"tile38.operation": "SET",
"tile38.hosts": "tile38:9851"
}'
Currently only supports SET command
Visualization using Arcadia
Data
Arcadia Data https://www.arcadiadata.com/
Location Analytics - Real-Time Geofencing using Apache Kafka

More Related Content

PPTX
Modeling Data and Queries for Wide Column NoSQL
PDF
Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd
PDF
Santander Stream Processing with Apache Flink
PDF
Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드
PDF
Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...
PDF
From airflow to google cloud composer
PDF
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
PPTX
PayPal merchant ecosystem using Apache Spark, Hive, Druid, and HBase
Modeling Data and Queries for Wide Column NoSQL
Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd
Santander Stream Processing with Apache Flink
Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드
Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...
From airflow to google cloud composer
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
PayPal merchant ecosystem using Apache Spark, Hive, Druid, and HBase

What's hot (20)

PPTX
Anton Grishko "Multi-cloud with Google Anthos, Kubernetes and Istio. How to s...
PDF
KSQL Intro
PDF
SQream DB, GPU-accelerated data warehouse
PDF
State of the Trino Project
PDF
Building robust CDC pipeline with Apache Hudi and Debezium
PDF
Kubernetes internals (Kubernetes 해부하기)
PPTX
공간정보거점대학 1.geo server_고급과정
PDF
Apache Kafka in the Airline, Aviation and Travel Industry
PPTX
Technical overview of Azure Cosmos DB
PDF
Postgres Vision 2018: WAL: Everything You Want to Know
 
PDF
Patroni: Kubernetes-native PostgreSQL companion
PPTX
Cloud & GCP 101
PDF
Kubernetes Security with Calico and Open Policy Agent
PDF
MariaDB MaxScale
PPTX
Building an Analytic Extension to MySQL with ClickHouse and Open Source
PDF
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안
PDF
ksqlDB - Stream Processing simplified!
PDF
Seamless scaling of Kubernetes nodes
PPTX
검색엔진이 데이터를 다루는 법 김종민
PPTX
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...
Anton Grishko "Multi-cloud with Google Anthos, Kubernetes and Istio. How to s...
KSQL Intro
SQream DB, GPU-accelerated data warehouse
State of the Trino Project
Building robust CDC pipeline with Apache Hudi and Debezium
Kubernetes internals (Kubernetes 해부하기)
공간정보거점대학 1.geo server_고급과정
Apache Kafka in the Airline, Aviation and Travel Industry
Technical overview of Azure Cosmos DB
Postgres Vision 2018: WAL: Everything You Want to Know
 
Patroni: Kubernetes-native PostgreSQL companion
Cloud & GCP 101
Kubernetes Security with Calico and Open Policy Agent
MariaDB MaxScale
Building an Analytic Extension to MySQL with ClickHouse and Open Source
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안
ksqlDB - Stream Processing simplified!
Seamless scaling of Kubernetes nodes
검색엔진이 데이터를 다루는 법 김종민
Cassandra on Mesos Across Multiple Datacenters at Uber (Abhishek Verma) | C* ...
Ad

Similar to Location Analytics - Real-Time Geofencing using Apache Kafka (20)

PDF
Location Analytics Real-Time Geofencing using Kafka
PDF
Location Analytics - Real Time Geofencing using Apache Kafka
PDF
Location Analytics - Real-Time Geofencing using Kafka
PDF
Geospatial Enhancements in MongoDB 2.4
PPTX
LocationTech Projects
PDF
Scripting GeoServer
ODP
Introduction To PostGIS
KEY
Handling Real-time Geostreams
KEY
Handling Real-time Geostreams
PDF
LocationTech Projects
PDF
Geolocation on Rails
KEY
OSCON july 2011
PDF
CouchDB Mobile - From Couch to 5K in 1 Hour
PPT
Stockage, manipulation et analyse de données matricielles avec PostGIS Raster
PDF
Representing and Querying Geospatial Information in the Semantic Web
PDF
State of the Art Web Mapping with Open Source
PDF
The Ring programming language version 1.2 book - Part 41 of 84
PPTX
Tim Hall [InfluxData] | InfluxDB Roadmap | InfluxDays Virtual Experience Lond...
PDF
GeoServer for Spatio-temporal Data Handling With Examples For MetOc And Remot...
PDF
GeoDjango & HTML5 Geolocation
Location Analytics Real-Time Geofencing using Kafka
Location Analytics - Real Time Geofencing using Apache Kafka
Location Analytics - Real-Time Geofencing using Kafka
Geospatial Enhancements in MongoDB 2.4
LocationTech Projects
Scripting GeoServer
Introduction To PostGIS
Handling Real-time Geostreams
Handling Real-time Geostreams
LocationTech Projects
Geolocation on Rails
OSCON july 2011
CouchDB Mobile - From Couch to 5K in 1 Hour
Stockage, manipulation et analyse de données matricielles avec PostGIS Raster
Representing and Querying Geospatial Information in the Semantic Web
State of the Art Web Mapping with Open Source
The Ring programming language version 1.2 book - Part 41 of 84
Tim Hall [InfluxData] | InfluxDB Roadmap | InfluxDays Virtual Experience Lond...
GeoServer for Spatio-temporal Data Handling With Examples For MetOc And Remot...
GeoDjango & HTML5 Geolocation
Ad

More from Guido Schmutz (20)

PDF
30 Minutes to the Analytics Platform with Infrastructure as Code
PDF
Event Broker (Kafka) in a Modern Data Architecture
PDF
Big Data, Data Lake, Fast Data - Dataserialiation-Formats
PDF
Kafka as your Data Lake - is it Feasible?
PDF
Event Hub (i.e. Kafka) in Modern Data Architecture
PDF
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
PDF
Event Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
PDF
Building Event Driven (Micro)services with Apache Kafka
PDF
Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka
PDF
What is Apache Kafka? Why is it so popular? Should I use it?
PDF
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
PDF
Streaming Visualisation
PDF
Kafka as an event store - is it good enough?
PDF
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
PDF
Fundamentals Big Data and AI Architecture
PDF
Streaming Visualization
PDF
Streaming Visualization
PDF
Building Event-Driven (Micro) Services with Apache Kafka
PDF
Introduction to Stream Processing
PDF
Stream Processing – Concepts and Frameworks
30 Minutes to the Analytics Platform with Infrastructure as Code
Event Broker (Kafka) in a Modern Data Architecture
Big Data, Data Lake, Fast Data - Dataserialiation-Formats
Kafka as your Data Lake - is it Feasible?
Event Hub (i.e. Kafka) in Modern Data Architecture
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Event Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
Building Event Driven (Micro)services with Apache Kafka
Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka
What is Apache Kafka? Why is it so popular? Should I use it?
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Streaming Visualisation
Kafka as an event store - is it good enough?
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
Fundamentals Big Data and AI Architecture
Streaming Visualization
Streaming Visualization
Building Event-Driven (Micro) Services with Apache Kafka
Introduction to Stream Processing
Stream Processing – Concepts and Frameworks

Recently uploaded (20)

PPTX
CEE 2 REPORT G7.pptxbdbshjdgsgjgsjfiuhsd
PPTX
climate analysis of Dhaka ,Banglades.pptx
PDF
Capcut Pro Crack For PC Latest Version {Fully Unlocked 2025}
PPT
DATA COLLECTION METHODS-ppt for nursing research
PPTX
Market Analysis -202507- Wind-Solar+Hybrid+Street+Lights+for+the+North+Amer...
PPTX
mbdjdhjjodule 5-1 rhfhhfjtjjhafbrhfnfbbfnb
PDF
REAL ILLUMINATI AGENT IN KAMPALA UGANDA CALL ON+256765750853/0705037305
PDF
168300704-gasification-ppt.pdfhghhhsjsjhsuxush
PPTX
The THESIS FINAL-DEFENSE-PRESENTATION.pptx
PDF
Galatica Smart Energy Infrastructure Startup Pitch Deck
PDF
22.Patil - Early prediction of Alzheimer’s disease using convolutional neural...
PDF
Introduction to Data Science and Data Analysis
PPTX
01_intro xxxxxxxxxxfffffffffffaaaaaaaaaaafg
PDF
Optimise Shopper Experiences with a Strong Data Estate.pdf
PPTX
importance of Data-Visualization-in-Data-Science. for mba studnts
PPTX
Pilar Kemerdekaan dan Identi Bangsa.pptx
PPTX
iec ppt-1 pptx icmr ppt on rehabilitation.pptx
PDF
[EN] Industrial Machine Downtime Prediction
PPTX
modul_python (1).pptx for professional and student
CEE 2 REPORT G7.pptxbdbshjdgsgjgsjfiuhsd
climate analysis of Dhaka ,Banglades.pptx
Capcut Pro Crack For PC Latest Version {Fully Unlocked 2025}
DATA COLLECTION METHODS-ppt for nursing research
Market Analysis -202507- Wind-Solar+Hybrid+Street+Lights+for+the+North+Amer...
mbdjdhjjodule 5-1 rhfhhfjtjjhafbrhfnfbbfnb
REAL ILLUMINATI AGENT IN KAMPALA UGANDA CALL ON+256765750853/0705037305
168300704-gasification-ppt.pdfhghhhsjsjhsuxush
The THESIS FINAL-DEFENSE-PRESENTATION.pptx
Galatica Smart Energy Infrastructure Startup Pitch Deck
22.Patil - Early prediction of Alzheimer’s disease using convolutional neural...
Introduction to Data Science and Data Analysis
01_intro xxxxxxxxxxfffffffffffaaaaaaaaaaafg
Optimise Shopper Experiences with a Strong Data Estate.pdf
importance of Data-Visualization-in-Data-Science. for mba studnts
Pilar Kemerdekaan dan Identi Bangsa.pptx
iec ppt-1 pptx icmr ppt on rehabilitation.pptx
[EN] Industrial Machine Downtime Prediction
modul_python (1).pptx for professional and student

Location Analytics - Real-Time Geofencing using Apache Kafka

  • 1. BASEL | BERN | BRUGG | BUCHAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENEVA HAMBURG | COPENHAGEN | LAUSANNE | MANNHEIM | MUNICH | STUTTGART | VIENNA | ZURICH http://guidoschmutz.wordpress.com@gschmutz Location Analytics Real-Time Geofencing using Kafka Guido Schmutz UKOUG Techfest 2019 updated for ksqlDB
  • 2. Agenda 1. Introduction & Motivation 2. Implementing Geo Fencing with Kafka • Using Oracle Stream Analytics • Using ksqlDB • Using Kafka Streams • Using Tile38 3. Visualization using ArcadiaData 4. Summary
  • 3. BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 22 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 176th edition
  • 5. Geofencing – What is it? Use of GPS or RFID technology to create a virtual geographic boundary, enabling software to trigger a response when an object/device enters or leaves a particular area Possible Events • ENTER • EXIT • OUTSIDE • lNSIDE Source: https://tile38.com
  • 6. Apache Kafka – A Streaming Platform Source Connector Sink Connector trucking_ driver ksqlDB Engine Kafka Streams Kafka Broker ksqlDB
  • 7. Apache Kafka Kafka Cluster Consumer 1 Consume 2r Broker 1 Broker 2 Broker 3 Zookeeper Ensemble ZK 1 ZK 2ZK 3 Schema Registry Service 1 Management Control Center Kafka Manager KAdmin Producer 1 Producer 2 kafkacat Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based Producer3Producer3 ConsumerConsumer 3
  • 8. Apache Kafka – How does it scale? • horizontally scalable, guaranteed order
  • 9. Streaming Analytics: ksqlDB or Kafka Streams Stream • unbounded sequence of structured data ("facts") • Facts in a stream are immutable Table • collected state of a stream • Latest value for each key in a stream • Facts in a table are mutable ksqlDB: Stream Processing with zero coding using SQL-like language (now supporting push and pull queries) Kafka Streams: Java library providing stream analytics capabilities trucking_ driver Kafka Broker ksqlDB Engine Kafka Streams ksqlDB REST Commands ksqlDB CLI push pull
  • 10. Stream vs. Table Vehicle Position Stream id latitude longitude 1 52.3924 13.0514 id latitude longitude 1 52.3924 13.0514 4 38.4847 -90.23345 id latitude longitude 1 52.3924 13.0514 4 38.4847 -90.23345 1 52.39052 13.06455 id name geometry_wkt 10 St. Louis POLYGON ((13.297920227050781 52.56195151687443, …)) 11 Berlin POLYGON ((-90.23345947265625 38.484769753492536,…)) id name geometry_wkt 10 St. Louis POLYGON ((13.297920227050781 52.56195151687443, …)) id name geometry_wkt 10 St. Louis (US) POLYGON ((13.297920227050781 52.56195151687443, …)) 11 Berlin (GE) POLYGON ((-90.23345947265625 38.484769753492536,…)) GeoFence Table
  • 11. First "naïve " idea when answering the CFP • geofence being an UDF (User Defined Function) • would need access to the geofences defined somewhere • what about updates? • a function should not cause any side-effects SELECT geofence(latitude, longitude) geo_event FROM geo_position_stream
  • 12. Dash board High Level Overview of Test Case geofence Join Position & Geofences Vehicle Position vehicle position pos & geofences Geo fencing geofence status key=10 { "id" : "10", "latitude" : 38.35821, "longitude" : -90.15311} key=3 {"id":3,"name":"Berlin, Germany","geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))","last_update":1560607149015} Geofence Mgmt Vehicle Position Weather Service
  • 13. Geo-Processing Well-known text (WKT) is a text markup language for representing vector geometry objects on a map GeoTools is a free software GIS toolkit for developing standards compliant solutions
  • 14. Implementing Geo Fencing - using Oracle Stream Analytics
  • 15. Oracle Stream Analytics (OSA) • Expressive Patterns Library, including Geo Processing • Abstracted visual façade to interrogate live real time streaming data • Integrate with Apache Kafka • Runs on top of Spark Cluster as a Spark Streaming Job
  • 16. Implementing Geo Fencing - using ksqlDB
  • 17. ksqlDB – Streams and Tables geofence Table vehicle position Stream KSQL Geofencing id latitude longitude 1 52.3924 13.0514 4 38.4847 -90.23345 3 .. .. id name geometry_wkt 10 St. Louis POLYGON ((13.297920227050781 52.56195151687443, …)) 11 Berlin POLYGON ((-90.23345947265625 38.484769753492536,…)) 12 xxxxx POLYGON ((…))
  • 18. ksqlDB – Streams and Tables geofence Table vehicle position Stream CREATE STREAM vehicle_position_s (id VARCHAR, latitude DOUBLE, longitude DOUBLE) WITH (KAFKA_TOPIC='vehicle_position', VALUE_FORMAT='DELIMITED'); CREATE TABLE geo_fence_t (id BIGINT, name VARCHAR, geometry_wkt VARCHAR) WITH (KAFKA_TOPIC='geo_fence', VALUE_FORMAT='JSON', KEY = 'id'); KSQL Geofencing
  • 19. How to determine "inside" or "outside" geofence? Only one standard UDF for geo processing in KSQL: GEO_DISTANCE Implement custom UDF using functionality from GeoTools Java library public String geo_fence(final double latitude, final double longitude, final String geometryWKT){ .. } public List<String> geo_fence_bulk(final double latitude , final double longitude, List<String> idGeometryListWKT) { .. } ksql> SELECT geo_fence(latitude, longitude, 'POLYGON ((13.297920227050781 52.56195151687443, 13.2440185546875 52.530216577830124, ...))') FROM test_geo_udf_s; 52.4497 | 13.3096 | OUTSIDE 52.4556 | 13.3178 | INSIDE
  • 20. Custom UDF to determine if Point is inside a geometry @Udf(description = "determines if a lat/long is inside or outside the geometry passed as the 3rd parameter as WKT encoded ...") public String geo_fence(@UdfParameter(value="latitude") final double latitude, @UdfParameter(value="longitude") ) final double longitude, @UdfParameter(value="geometryWKT") ) final String geometryWKT) { String status = ""; GeometryFactory geometryFactory = JTSFactoryFinder.getGeometryFactory(); WKTReader reader = new WKTReader(geometryFactory); Polygon polygon = (Polygon) reader.read(geometryWKT); Coordinate coord = new Coordinate(longitude, latitude); Point point = geometryFactory.createPoint(coord); if (point.within(polygon)) { status = "INSIDE"; } else { status = "OUTSIDE"; } return status; }
  • 21. 1) Using Cross Join geofence Table Join Position & Geofences vehicle position Stream Stream pos & geofences CREATE STREAM vp_join_gf_s AS SELECT vp.id, geo_fence(vp.latitude, vp.longitude, gf.geometry_wkt) status FROM vehicle_position_s AS vp CROSS JOIN geo_fence_t AS gf There is no Cross Join in ksqlDB! id latitude longitude 1 52.3924 13.0514 4 38.4847 -90.23345 3 .. .. id name geometry_wkt 10 St. Louis POLYGON ((13.297920227050781 52.56195151687443, …)) 11 Berlin POLYGON ((-90.23345947265625 38.484769753492536,…)) 12 xxxxx POLYGON ((…)) X
  • 22. 2) INNER Join geofence Stream Join Position & Geofences vehicle position Stream Stream pos & geofences { "group":1", "name":"Berlin", "geometry_wkt":"POLYGON ((- 90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} Enrich Group Table geofences by group 1 Enrich Group Stream postion by group 1 Cannot insert into Table from Stream >INSERT INTO geo_fence_t >SELECT '1' AS group_id, geof.id, … >FROM geo_fence_s geof; INSERT INTO can only be used to insert into a stream. A02_GEO_FENCE_T is a table. { "group":"1", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514} gid id latitude longitude 1 1 52.3924 13.0514 1 4 38.4847 -90.23345 1 3 .. .. gid id name geometry_wkt 1 10 St. Louis POLYGON ((13.297920227050781 52.56195151687443, …)) 1 11 Berlin POLYGON ((-90.23345947265625 38.484769753492536,…)) { "group":1", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015}
  • 23. 3) Geofences aggregated in one group Join Position & Geofences Stream geofence event Geofences aggby group Table { "group":"1", [ "1:POLYGON ((13.297920227050781 52.56195151687443, …))","2:POLYGON ((- 90.23345947265625 38.484769753492536,…))" ] } geo_fence_bulk geofence Stream vehicle position Stream { "group":1", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015} Enrich With Group-1 Stream geofences by group 1 Enrich With Group-1 Stream postion by group 1 geofences by group 1 high low low high low high Scalable Latency "Code Smell" medium medium medium { "group":"1", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514} { "group":1", "name":"Berlin", "geometry_wkt":"POLYGON ((-90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} gid id latitude longitude 1 1 52.3924 13.0514 1 4 38.4847 -90.23345 1 3 .. .. gid geometry_wkt_list 1 1:POLYGON ((13.297920227050781 52.56195151687443, …)) 2:POLYGON ((-90.23345947265625 38.484769753492536,…)) X:POLGON ((…)) always only one row
  • 24. 3) Geofences aggregated in one group CREATE TABLE a03_geo_fence_aggby_group_t AS SELECT group_id , collect_set(id + ':' + geometry_wkt) AS id_geometry_wkt_list FROM a03_geo_fence_by_group_s geof GROUP BY group_id; CREATE STREAM a03_vehicle_position_by_group_s AS SELECT '1' group_id, vehp.id, vehp.latitude, vehp.longitude FROM vehicle_position_s vehp PARTITION BY group_id;
  • 25. 3) Geofences aggregated in one group CREATE STREAM a03_geo_fence_status_s AS SELECT vehp.id, vehp.latitude, vehp.longitude, geo_fence_bulk(vehp.latitude, vehp.longitude, geofaggid_geometry_wkt_list) AS geofence_status FROM a03_vehicle_position_by_group_s veh LEFT JOIN a03_geo_fence_aggby_group_t geofagg ON vehp.group_id = geofagg.group_id; ksql> SELECT * FROM a03_geo_fence_status_s; 46 | 52.47546 | 13.34851 | [1:OUTSIDE, 3:INSIDE] 46 | 52.47521 | 13.34881 | [1:OUTSIDE, 3:INSIDE] ... As many as there are geo-fences
  • 26. Geo Hash for a better distribution Geohash is a geocoding which encodes a geographic location into a short string of letters and digits Length Area width x height 1 5,009.4km x 4,992.6km 2 1,252.3km x 624.1km 3 156.5km x 156km 4 39.1km x 19.5km 12 3.7cm x 1.9cm http://geohash.gofreerange.com/
  • 27. Geo Hash Custom UDF ksql> SELECT latitude, longitude, geo_hash(latitude, longitude, 3) >FROM vehicle_position_s; 38.484769753492536 | -90.23345947265625 | 9yz public String geohash(final double latitude, final double longitude, int length) public List<String> neighbours(String geohash) public String adjacentHash(String geohash, String directionString) public List<String> coverBoundingBox(String geometryWKT, int length) ksql> SELECT name, wkt, geo_hash(geometry_wkt, 3) FROM a04_geo_fence_s; St. Louis, POLYGON ((-90.25749206542969 38.71551876930462, -90.31723022460938 38.69301319283493, ...)) | [9yz] ksql> SELECT name, wkt, geo_hash(geometry_wkt, 4) FROM a04_geo_fence_s; St. Louis, POLYGON ((-90.25749206542969 38.71551876930462, -90.31723022460938 38.69301319283493, ...)) | [9yzg, 9yzu]
  • 28. 4) Geofences aggregated by GeoHash Join Position & Geofences Stream geofence event Geofences gpby geohash Table { "geohash":"u33", ["2:POLYGON ((- 90.23345947265625 38.484769753492536,…))"], …} geo_fence() geofence Table vehicle position Stream { "geohash":"9yz", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015}} Enrich with GeoHash Stream geofences & geohash Enrich with GeoHash Stream position & geohash geofences by geohash geo_hash() geo_hash() { "geohash":"u33", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514} { "group":"u33", "name":"Berlin", "geometry_wkt":"POLYGON ((- 90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} { "geohash":"dnb", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015}} { "geohash":"9yz", [ "1:POLYGON ((13.297920227050781 52.56195151687443, …))", ..]} { "geohash":"dnb", [ "1:POLYGON ((13.297920227050781 52.56195151687443, …))",..]} geohash id latitude longitude u33 1 52.3924 13.0514 9yz 4 38.4847 -90.23345 u34 3 .. .. geohash geometry_wkt_list 9yz 1:POLYGON ((13.297920227050781 52.56195151687443, …)) N:POLYGON (( …. )) u33 2:POLYGON ((-90.23345947265625 38.484769753492536,…)) dnb 1:POLYGON ((13.297920227050781 52.56195151687443, …)) high low low high low high Scalable Latency "Code Smell" medium medium medium
  • 29. 4) Geofences aggregated by GeoHash CREATE STREAM a04_geo_fence_by_geohash_s AS SELECT EXPLODE(geo_hash(geometry_wkt, 3)) geo_hash, id, name, geometry_wkt FROM a04_geo_fence_s PARTITION by geo_hash; There was no explode() / unpivot() functionality in KSQL but now ksqlDB provides it! geofence Table Enrich with GeoHash Stream geofence & geohash ksql> SELECT name, geo_hash FROM a04_geo_fence_and_geohash_s EMIT CHANGES; | Colombia, Missouri | 9yz | Berlin, Germany | u33 | St. Louis, Missouri | 9yz
  • 30. 4) Geofences aggregated by GeoHash CREATE TABLE a04_geo_fence_by_geohash_t AS SELECT geohash, COLLECT_SET(geometry_wkt) AS id_geometry_wkt_list, COLLECT_SET(id) AS id_list FROM a04_geo_fence_and_geohash_s GROUP BY geo_hash; Geofences gpby geohash Table Stream geofences & geohash geofences by geohash ksql> SELECT * FROM a04_geo_fence_by_geohash_t EMIT CHANGES; | 9yz | [POLYGON ((13.297920227050781 52.56195151687443, …)), POLYGON((…))] |[1,N] | u33 | [POLYGON ((-90.23345947265625 38.484769753492536] |[2] | dnb | [POLYGON ((13.297920227050781 52.56195151687443, …)), POLYGON((…))] |[1] ...
  • 31. 4) Geofences aggregated by GeoHash CREATE STREAM a04_vehicle_position_by_geohash_s AS SELECT vp.id, vp.latitude, vp.longitude, geo_hash(vp.latitude, vp.longitude, 3) geo_hash FROM vehicle_position_s vp PARTITION BY geo_hash; vehicle position Stream Enrich with GeoHash Stream position & geohash ksql> SELECT * FROM a04_vehicle_position_by_geohash_s EMIT CHANGES; | 10 | 52.4497 | 13.3096 | u33 | 11 | 38.521846880854966 | -90.19912719726561 | 9yz ...
  • 32. 4) Geofences aggregated by GeoHash geohash id latitude longitude u33 1 52.3924 13.0514 9yz 4 38.4847 -90.23345 nnn 3 .. .. geohash geometry_wkt_list 9yz 1:POLYGON ((13.297920227050781 52.56195151687443, …)) N:POLYGON (( …. )) u33 2:POLYGON ((-90.23345947265625 38.484769753492536,…)) dnb 1:POLYGON ((13.297920227050781 52.56195151687443, …)) geoh id latitude longitude geometry_wkt_list u33 1 52.3924 13.0514 1:POLYGON ((13.297920227050781 52.56195151687443, …)) 9yz 4 38.4847 -90.23345 1:POLYGON ((13.297920227050781 52.56195151687443, …)) N:POLYGON (( …. )) nnn 3 .. ..
  • 33. 4) Geofences aggregated by GeoHash CREATE STREAM a04_geo_fence_status_s AS SELECT vp.id, vp.latitude, vp.longitude, vp.geo_hash, explode (gf.id_list) AS geofenceId, geo_fence (vp.latitude, vp.longitude, explode (gf.wkt_list)) AS geo_event FROM a04_vehicle_position_by_geohash_s vp LEFT JOIN a04_geo_fence_by_geohash_t gf ON (vp.geo_hash = gf.geo_hash); ksql> SELECT * FROM a04_geo_fence_status_s; | 10 | 52.4497 | 13.3096 | u33 | 3 | OUTSIDE | 11 | 38.521846880854966 | -90.19912719726561 | 9yz | 2 | OUTSIDE | 11 | 38.521846880854966 | -90.19912719726561 | 9yz | 1 | OUTSIDE ... Join Position & Geofences Stream geofence event
  • 34. Berne Fribourg It works …. but …. • Becaue of re-partitioning by geohash we lose the guaranteed order for a given vehicle • Can be problematic, if there is a backlog in one of the topics/partitions u0m5 u0m4 u0m7 u0m6 Consumer 1 Consumer 2
  • 35. Implementing Geo Fencing - using Kafka Streams
  • 36. Geo-Fencing with Kafka Streams and Global KTable Enrich Position with GeoHash & Join with Geofences Global KTable geofence KTable vehicle position { "geohash":u33", "name":"Potsdam", "geometry_wkt":"POLYGON ((5.668945 51.416016, …))", "last_update":1560607149015} Enrich and Group by GeoHash matched geofences Detect Geo Event geofence_ status high low low high low high Scalable Latency "Code Smell" medium medium medium geofence by geohash {"id":"10", "latitude" : 52.3924, "longitude" : 13.0514, [ {"name":"Berlin"} ] } { "geohash":"u33", "id" : "10", "latitude" : 52.3924, "longitude" : 13.0514} {"id":"10", "status" : "ENTER", "geofenceName":"Berlin"} } position & geohash { "geohash":u33", "name":"Berlin", "geometry_wkt":"POLYGON ((-90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} { "geohash":"u33", [ {"name":"Berlin", "geometry_wkt":"POLYGON ((-90.23345947265625 38.484769753492536,…))"},{"name":"Potsdam", "geometry_wkt":"POLYGON ((5.668945 51.416016, …))"} ] } { "geohash":"9yz", [ {"name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))"} ] }{ "geohash":"9yz", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015}}
  • 37. Geo-Fencing with Kafka Streams and Global KTable KStream<String, GeoFence> geoFence = builder.stream(GEO_FENCE); KStream<String, GeoFence> geoFenceByGeoHash = geoFence.map((k,v) -> KeyValue.<GeoFence, List<String>> pair(v, GeoHashUtil.coverBoundingBox(v.getWkt().toString(), 5))) .flatMapValues(v -> v) .map((k,v) -> KeyValue.<String,GeoFence>pair(v, createFrom(k, v))); KTable<String, GeoFenceList> geofencesByGeohash = geoFenceByGeoHash.groupByKey().aggregate( () -> new GeoFenceList(new ArrayList<GeoFenceItem>()), (aggKey, newValue, aggValue) -> { GeoFenceItem geoFenceItem = new GeoFenceItem(newValue.getId(), newValue.getName(), newValue.getWkt(), ""); if (!aggValue.getGeoFences().contains(geoFenceItem)) aggValue.getGeoFences().add(geoFenceItem); return aggValue; }, Materialized.<String, GeoFenceList, KeyValueStore<Bytes,byte[]>>as("geofences-by-geohash-store")); geofencesByGeohash.toStream().to(GEO_FENCES_KEYEDBY_GEOHASH, Produced.<String, GeoFenceList> keySerde(stringSerde));
  • 38. Geo-Fencing with Kafka Streams and Global KTable final GlobalKTable<String, GeoFenceList> geofences = builder.globalTable(GEO_FENCES_KEYEDBY_GEOHASH); KStream<String, VehiclePositionWithMatchedGeoFences> positionWithMatchedGeoFences = vehiclePositionsWithGeoHash.leftJoin(geofences, (k, pos) -> pos.getGeohash().toString(), (pos, geofenceList) -> { List<MatchedGeoFence> matchedGeofences = new ArrayList<MatchedGeoFence>(); if(geofenceList != null) { for (GeoFenceItem geoFenceItem : geofenceList.getGeoFences()) { boolean geofenceStatus = GeoFenceUtil.geofence(pos.getLatitude(), pos.getLongitude(), geoFenceItem.getWkt().toString()); if(geofenceStatus) matchedGeofences.add(new MatchedGeoFence(geoFenceItem.getId(), geoFenceItem.getName(), null)); } } return new VehiclePositionWithMatchedGeoFences(pos.getVehicleId(), 0L, pos.getLatitude(), pos.getLongitude(), pos.getEventTime(), matchedGeofences); });
  • 39. Implementing Geo Fencing - using Tile38
  • 40. Tile38 • https://tile38.com • Open Source Geospatial Database & Geofencing Server • Real Time Geofencing • Roaming Geofencing • Fast Spatial Indices • Pluggable Event Notifications
  • 41. Tile38 – How does it work? > SETCHAN berlin WITHIN vehicle FENCE OBJECT {"type":"Polygon","coordinates":[[[13.297920227050781,52.56195151687443],[1 3.2440185546875,52.530216577830124],[13.267364501953125,52.45998421679598], [13.35113525390625,52.44826791583386],[13.405036926269531,52.44952338289473 ],[13.501167297363281,52.47148826410652], ...]]} > SUBSCRIBE berlin {"ok":true,"command":"subscribe","channel":"berlin","num":1,"elapsed":"5.85 µs"} . . . {"command":"set","group":"5d07581689807d000193ac33","detect":"outside","hoo k":"berlin","key":"vehicle","time":"2019-06- 17T09:06:30.624923584Z","id":"10","object":{"type":"Point","coordinates":[1 3.3096,52.4497]}} SET vehicle 10 POINT 52.4497 13.3096
  • 42. Tile38 – How does it work? > SETHOOK berlin_hook kafka://broker-1:9092/tile38_geofence_status WITHIN vehicle FENCE OBJECT {"type":"Polygon","coordinates":[[[13.297920227050781,52.56195151687443],[1 3.2440185546875,52.530216577830124],[13.267364501953125,52.45998421679598], [13.35113525390625,52.44826791583386],[13.405036926269531,52.44952338289473 ],[13.501167297363281,52.47148826410652], ...]]} bigdata@bigdata:~$ kafkacat -b localhost -t tile38_geofence_status % Auto-selecting Consumer mode (use -P or -C to override) {"command":"set","group":"5d07581689807d000193ac34","detect":"outside","hoo k":"berlin_hook","key":"vehicle","time":"2019-06- 17T09:12:00.488599119Z","id":"10","object":{"type":"Point","coordinates":[1 3.3096,52.4497]}} SET vehicle 10 POINT 52.4497 13.3096
  • 43. 1) Enrich with GeoFences – aggregated by geohash geofence Stream vehicle position Stream Invoke UDF { "id" : "10", "latitude" : 38.35821, "longitude" : -90.15311} Invoke UDF Geofence Service geofence status set_pos() set_fence() Stream null high low low high low high Scalable Latency "Code Smell" medium medium medium { "id":11", "name":"Berlin", "geometry_wkt":"POLYGON ((- 90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} { "id":10", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015}
  • 44. 2) Using Custom Kafka Connector for Tile38 geofence vehicle position Geofence Service kafka-to- tile38 kafka-to- tile38 geofence status high low low high low high Scalable Latency "Code Smell" medium medium medium { "id":11", "name":"Berlin", "geometry_wkt":"POLYGON ((- 90.23345947265625 38.484769753492536,…))", "last_update":1560607149015} { "id":10", "name":"St. Louis", "geometry_wkt":"POLYGON ((13.297920227050781 52.56195151687443, …))", "last_update":1560607149015} { "id" : "10", "latitude" : 38.35821, "longitude" : -90.15311}
  • 45. 2) Using Custom Kafka Connector for Tile38 curl -X PUT /api/kafka-connect-1/connectors/Tile38SinkConnector/config -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{ "connector.class": "com.trivadis.geofence.kafka.connect.Tile38SinkConnector", "topics": "vehicle_position", "tasks.max": "1", "tile38.key": "vehicle", "tile38.operation": "SET", "tile38.hosts": "tile38:9851" }' Currently only supports SET command