Creating a Context-Aware solution, Complex Event Processing with FIWARE Perseo
1 like574 views
The document discusses complex event processing (CEP) utilizing the Perseo framework, focusing on integrating it with the Orion Context Broker and defining events through Event Processing Language (EPL). It outlines how CE is designed for high throughput and low latency applications, enabling real-time event detection and action triggers. Additionally, it provides examples of event rules, actions, and the structure of event processing within the FIWARE ecosystem.
![Notice vs. Event
▪ Notifications or notices from Context Broker
are processed by Perseo (perseo-fe) before
being sent to its core (perseo-core + Esper)
as events.
▪ They correspond to changes in the values of
an attribute of an entity to which Perseo is
subscribed to.
{
"subscriptionId":"5bfaf6ab27124e3d9b6c0b20",
"data":[{
"id":"streetlight:gothenburg:3",
"type":"Streetlight",
"status":{
"type":"Text",
"value":"ok",
"metadata":{}
},
"dateLastLampChange":{
"type":"DateTime",
"value":"2018-11-20T11:00:00.00Z",
"metadata":{}
},
"address":{
"type":"Address",
"value":{
"addressLocality":"Gothenburg",
"addressCountry":"Sweden",
"postalCode":"425 30",
"streetAddress":"Norra Ligården 1”
},
"metadata":{}
},
"areaServed":{
"type":"Text",
"value":"Karra-Sodra",
"metadata":{}
},
"illuminanceLevel":{
"type":”Number",
"value":”95",
"metadata":{}
},
"powerState":{
"type":"Text",
"value":"off",
"metadata":{}
}
}],
"subservice":"/demo/fiwaresummit19",
"service":”carthage”
}
Notice
8](https://image.slidesharecdn.com/session7-creatingacontext-awaresolutioncomplexeventprocessingwithperseo-191127083908/85/Creating-a-Context-Aware-solution-Complex-Event-Processing-with-FIWARE-Perseo-9-320.jpg)
![Rules
{
"name":“streetlight_column_alarm",
"text":"select ev.id? as id, ev.status? as laststatus, ev.dateLastLampChange__iso? as lastchange
pattern [every ev=iotEvent(type="Streetlight" and ev.status? = ‘columnIssue’)]",
"action”:{
"type":"update",
"parameters":{
"id":"${id}_alarm_column",
"type”:"StreetlightAlarm",
"version":"2",
"attributes": [
{
"name":"status",
"type":"Text",
"value":"${laststatus}"
},
{
"name":”lastChange",
"type":"DateTime",
"value":"${lastchange}"
}
]
}
}
Rule example. (create/update a StreetlightAlarm entity when a problem is detected in the column/pole)
▪ Perseo rules have three mandatory sections: name, text and action
11](https://image.slidesharecdn.com/session7-creatingacontext-awaresolutioncomplexeventprocessingwithperseo-191127083908/85/Creating-a-Context-Aware-solution-Complex-Event-Processing-with-FIWARE-Perseo-12-320.jpg)
![Rules: text
▪ Based on Esper EPL SQL-like
rules
▪ Pattern matching:
• from pattern
• match_recognize
▪ Event stream called iotEvent
and instanciated as ev
▪ We can use predefined
functions, such as cast(),
avg(), etc…
▪ It is no longer necessary to add
the RuleName in the "text" field
SELECT ev.id? as id, ev.status? as laststatus, ev.areaServed?
as areaServed
FROM pattern
[every ev=iotEvent(type=‘Streetlight’ and ev.status? != ‘ok’)]
EPL using paGern clause
SELECT *
FROM iotEvent(type=‘Streetlight’).win:time(1 minute)
MATCH_RECOGNIZE (
measures A.status? as status
pattern (A B)
define(
A as A.status? != ‘ok’,
B as B.status? != ‘ok’ and
B.id? != A.id)
EPL using match_recognize
13](https://image.slidesharecdn.com/session7-creatingacontext-awaresolutioncomplexeventprocessingwithperseo-191127083908/85/Creating-a-Context-Aware-solution-Complex-Event-Processing-with-FIWARE-Perseo-14-320.jpg)
![Rules: actions
Rule actions
▪ Options:
• Update context broker entity
• SMS
• Email
• HTTP invocation
• Send a tweet
▪ We can use placeholders, ${X},
to insert data from the events
"action":”:{
"type":"update",
"parameters":{
"id":"${id}_alarm_column",
"type”:"StreetlightAlarm",
"version":"2",
"attributes": [
{
"name":"status",
"type":"Text",
"value":"${laststatus}"
},
{
"name":”lastChange",
"type":"DateTime",
"value":"${lastchange}"
}
]
}
Update CB entity action
"action": {
"type": "sms",
"template": "${id} status has change to ${laststatus}",
"parameters": {
"to": "123456789"
}}
SMS action
14](https://image.slidesharecdn.com/session7-creatingacontext-awaresolutioncomplexeventprocessingwithperseo-191127083908/85/Creating-a-Context-Aware-solution-Complex-Event-Processing-with-FIWARE-Perseo-15-320.jpg)
![Timed Rules
{
"name":”10_sec_timer_rule",
"text":"select * from pattern [every timer:interval(10 sec)]",
"action":{...}
}
Rule example: fires each 10 seconds
{
"name":”2_week_timer_rule",
"text":"select * from pattern [every timer:interval(2 weeks)]",
"action":{...}
}
▪ timer:interval(<number|variable_name> <time-period>)
Rule example: fires each 2 weeks
Time Periods:
• " years" | "year"
• " months" | "month “
• " weeks" | "week“
• "days" | "day“
• "hours" | "hour“
• "minutes" | "minute" | "min“
• "seconds" | "second" | "sec“
• "milliseconds" | "millisecond" | "msec"
18](https://image.slidesharecdn.com/session7-creatingacontext-awaresolutioncomplexeventprocessingwithperseo-191127083908/85/Creating-a-Context-Aware-solution-Complex-Event-Processing-with-FIWARE-Perseo-19-320.jpg)
![Timed Rules
▪ timer:at (minutes, hours, days of month, months, days of week [, seconds [, time zone]]).
{
"name":”5_min_past_hour",
"text":"select * from pattern [every timer:at(5, *, *, *, *)]",
"action":{...}
}
Rule example: fires once an hour, at minute five
{
"name":”17_PST",
"text":"select * from pattern [timer:at (0, 17, *, *, *, *, 'PST')]",
"action":{...}
}
Rule example: only fires 1 time at 5:00 pm Pacific Standard Time
19](https://image.slidesharecdn.com/session7-creatingacontext-awaresolutioncomplexeventprocessingwithperseo-191127083908/85/Creating-a-Context-Aware-solution-Complex-Event-Processing-with-FIWARE-Perseo-20-320.jpg)
![Timed Rules
▪ timer:schedule(repetitions, date, period)
{
"name":"specific_date",
"text":"select * from pattern[every timer:schedule(repetitions: 2, date: '2018-03-01T13:00:00Z',
period: 1 year 2 month 10 days 2 hours 30 minutes)]",
"action":{...}
}
Rule example: fires 2 times, first callback (2018-03-01 at 13:00:00 UTC) second callback (2019-05-11 at 15:30:00 UTC)
{
"name":"next_minute",
"text":"select * from pattern[every timer:schedule(date: '2019-10-01T05:52:00Z')]",
"action":{...}
}
Rule example: fires on 2019-10-01T05:52:00Z
Note: There are many other kind of timed Esper rules, such as time_batch , ext_timed_batch, match_recognize intervals, etc.
20](https://image.slidesharecdn.com/session7-creatingacontext-awaresolutioncomplexeventprocessingwithperseo-191127083908/85/Creating-a-Context-Aware-solution-Complex-Event-Processing-with-FIWARE-Perseo-21-320.jpg)
Creating a Context-Aware solution, Complex Event Processing with FIWARE Perseo
- 1. Session 7 - Creating a Context-Aware Solution: Complex Event Processing with Perseo Fernando López, Cloud & Platform Senior Expert [email protected] @flopezaguilar FIWARE Foundation, e.V.
- 2. Learning Goals 1 ● What is a Complex Event Processing? ● How to connect Perseo with Orion Context Broker? ● How can I define an event with EPL? ● What are the available acBons?
- 3. Overview 2 ▪ Context-aware CEP GE • Listen to events from context information to identify patterns described by rules, and immediately react by triggering actions. • Designed to be NGSI-compliant, and thus, to implement systems that works seamless and jointly with Orion CB. □ Uses NGSI as communication protocol for events □ Follows a PUSH model • Based on Esper CEP ▪ It is focused on Complex Event Processing, not on Stream Processing. • It defines a rule (query) engine, not a graph engine. • Rules are set using SQL-like queries (EPL: Event Processing Language)
- 4. Introduction to Complex Event Processing 3 ▪ Requirement to process events (or messages) in real-time or near real-time. ▪ Key considerations for these types of applications are throughput, latency and the complexity of the logic required: • High throughput - applications that process large volumes of messages (between 1,000 to 100k messages per second) • Low latency - applications that react in real-time to conditions that occur (from a few milliseconds to a few seconds) • Complex computations - applications that detect patterns among events (event correlation), filter events, aggregate time or length windows of events, join event series, trigger based on absence of events etc.
- 5. Introduction to the Esper architecture ▪ Esper is a language, a language compiler and a runtime environment. ▪ Language • The Esper language is the Event Processing Language (EPL). • It is a declarative, data-oriented language for dealing with high frequency time-based event data. • EPL is compliant to the SQL-92 standard and extended for analyzing series of events and in respect to time. ▪ Language Compiler • Compiles EPL source code into Java Virtual Machine (JVM) bytecode. • The resulting executable code runs on a JVM within the Esper runtime environment. 4
- 6. Introduction to the Esper architecture ▪ Runtime • The Esper runtime runs on top of a JVM. • You can run byte code produced by the Esper compiler using the Esper runtime. ▪ The Esper architecture is similar to Scala, Clojure and Kotlin. 5
- 7. Overview: CEP in FIWARE with Perseo 6 PERSEO Context-Aware CEP ACTIONS RULES ORION Context Broker EVENTS NGSI subscription NGSI Context notification Pattern matching REST API ACTION: NGSI Update Context ACTION: HTTP ACTION: SMS ACTION: EMAIL ACTION: TWEET Perseo-fe Perseo-core Events Alarms Actuate Sensors SMART IoT PLATFORM
- 8. ▪ Repositories • Perseo Core (https://github.com/telefonicaid/perseo-core) • Perseo Front-End (https://github.com/telefonicaid/perseo-fe) ▪ Docker images • Perseo Core (https://hub.docker.com/r/fiware/perseo-core/) • Perseo Front-End (https://hub.docker.com/r/fiware/perseo/) Overview: CEP in FIWARE with Perseo 7
- 9. Notice vs. Event ▪ Notifications or notices from Context Broker are processed by Perseo (perseo-fe) before being sent to its core (perseo-core + Esper) as events. ▪ They correspond to changes in the values of an attribute of an entity to which Perseo is subscribed to. { "subscriptionId":"5bfaf6ab27124e3d9b6c0b20", "data":[{ "id":"streetlight:gothenburg:3", "type":"Streetlight", "status":{ "type":"Text", "value":"ok", "metadata":{} }, "dateLastLampChange":{ "type":"DateTime", "value":"2018-11-20T11:00:00.00Z", "metadata":{} }, "address":{ "type":"Address", "value":{ "addressLocality":"Gothenburg", "addressCountry":"Sweden", "postalCode":"425 30", "streetAddress":"Norra Ligården 1” }, "metadata":{} }, "areaServed":{ "type":"Text", "value":"Karra-Sodra", "metadata":{} }, "illuminanceLevel":{ "type":”Number", "value":”95", "metadata":{} }, "powerState":{ "type":"Text", "value":"off", "metadata":{} } }], "subservice":"/demo/fiwaresummit19", "service":”carthage” } Notice 8
- 10. Notice vs. Event ▪ Notifications or notices from Context Broker are processed by Perseo (perseo-fe) before being sent to its core (perseo-core + Esper) as events. ▪ They correspond to changes in the values of an attribute of an entity to which Perseo is subscribed to. { "noticeId":"710e8dd0-f15a-11e8-90d5-7fa905f73247", "noticeTS":1543223138094, "id":"streetlight:gothenburg:3", "type":"Streetlight", "isPattern":false, "subservice":"/demo/fiwaresummit19", "service":"carthage", "status__type":"Text", "status":"ok", "address__type":"Address", "address__addressLocality":"Gothenburg", "address__addressCountry":"Sweden", "address__postalCode":"425 30", "address__streetAddress":"Norra Ligården 1", "areaServed__type":"Text", "areaServed":"Karra-Sodra", "illuminanceLevel __type":"Number", "illuminanceLevel": 95 "powerState__type":"Text", "powerState":"off” } Event Restriction: ▪ An attribute cannot be named id or type 9
- 11. Events, datatypes and NGSIv2 ▪ Perseo flattens the data structures to access the internal values of each data type, and add the next parsed values in the event. This attributes can be used in the rules individually. "location__type":"geo:point", "location__lat":12.0011546, "location__lon":57.7934324, "location__x":586374.5004314554, "location__y":1326805.6699024632, Location (geo:point, geo:line, geo:box, geo:polygon and geo:json): "location": { "type": "geo:point", "value": "12.0011546, 57.7934324", "metadata": {} } "dateLastLampChange__type": "DateTime", "dateLastLampChange__iso": 2017-09-10T11:00:00.00Z, "dateLastLampChange__ts": 1505041200000, "dateLastLampChange__day": 10, "dateLastLampChange__month": 9, "dateLastLampChange__year": 2017, "dateLastLampChange__hour": 13, "dateLastLampChange__minute": 0, "dateLastLampChange__second": 0, "dateLastLampChange__millisecond": 0, "dateLastLampChange__dayUTC": 10, "dateLastLampChange__monthUTC": 9, "dateLastLampChange__yearUTC": 2017, "dateLastLampChange__hourUTC": 11, "dateLastLampChange__minuteUTC": 0, "dateLastLampChange__secondUTC": 0, "dateLastLampChange__millisecondUTC": 0, DateTime: "dateLastLampChange": { "type": "DateTime", "value": "2017-09-10T11:00:00.00Z", "metadata": {} } 10
- 12. Rules { "name":“streetlight_column_alarm", "text":"select ev.id? as id, ev.status? as laststatus, ev.dateLastLampChange__iso? as lastchange pattern [every ev=iotEvent(type="Streetlight" and ev.status? = ‘columnIssue’)]", "action”:{ "type":"update", "parameters":{ "id":"${id}_alarm_column", "type”:"StreetlightAlarm", "version":"2", "attributes": [ { "name":"status", "type":"Text", "value":"${laststatus}" }, { "name":”lastChange", "type":"DateTime", "value":"${lastchange}" } ] } } Rule example. (create/update a StreetlightAlarm entity when a problem is detected in the column/pole) ▪ Perseo rules have three mandatory sections: name, text and action 11
- 13. Rules: names ▪ Simple description of the rule. 12 "name":“streetlight_column_alarm” Name
- 14. Rules: text ▪ Based on Esper EPL SQL-like rules ▪ Pattern matching: • from pattern • match_recognize ▪ Event stream called iotEvent and instanciated as ev ▪ We can use predefined functions, such as cast(), avg(), etc… ▪ It is no longer necessary to add the RuleName in the "text" field SELECT ev.id? as id, ev.status? as laststatus, ev.areaServed? as areaServed FROM pattern [every ev=iotEvent(type=‘Streetlight’ and ev.status? != ‘ok’)] EPL using paGern clause SELECT * FROM iotEvent(type=‘Streetlight’).win:time(1 minute) MATCH_RECOGNIZE ( measures A.status? as status pattern (A B) define( A as A.status? != ‘ok’, B as B.status? != ‘ok’ and B.id? != A.id) EPL using match_recognize 13
- 15. Rules: actions Rule actions ▪ Options: • Update context broker entity • SMS • Email • HTTP invocation • Send a tweet ▪ We can use placeholders, ${X}, to insert data from the events "action":”:{ "type":"update", "parameters":{ "id":"${id}_alarm_column", "type”:"StreetlightAlarm", "version":"2", "attributes": [ { "name":"status", "type":"Text", "value":"${laststatus}" }, { "name":”lastChange", "type":"DateTime", "value":"${lastchange}" } ] } Update CB entity action "action": { "type": "sms", "template": "${id} status has change to ${laststatus}", "parameters": { "to": "123456789" }} SMS action 14
- 16. Rules: actions Rule actions ▪ Options: • Update context broker entity • SMS • Email • HTTP invocation • Send a tweet ▪ We can use placeholders, ${X}, to insert data from the events "action": { "type": "email", "template": "${id} status has change to ${laststatus}”, "parameters": { "to": "[email protected]", "from": "[email protected]", "subject": ”Problems with StreetLamp ${id}. The status has change to ${laststatus}" } } Email action 15
- 17. Rules: actions Rule actions ▪ Options: • Update context broker entity • SMS • Email • HTTP invocation • Send a tweet ▪ We can use placeholders, ${X}, to insert data from the events "action":{ "type": "post", "template": "${id} status has change to ${laststatus}", "parameters":{ "url": "http://localhost:9182/${type}/${id}", "method": "PUT", "headers": { "Content-type": "text/plain", "X-${type}-status": "${laststatus}" }, "qs": { ”status": "${laststatus}", } } } HTTP acLon 16
- 18. Rules: actions Rule actions ▪ Options: • Update context broker entity • SMS • Email • HTTP invocation • Send a tweet ▪ We can use placeholders, ${X}, to insert data from the events "action": { "type": "twitter", "template": "${id} status has change to ${laststatus}" "parameters": { "consumer_key": "xvz1evFS4wEEPTGEFPHBog", "consumer_secret": "L8qq9PZyRg6ieKGEKhZolGC0vJWLw8iEJ88DRdyOg", "access_token_key": "xvz1evFS4wEEPTGEFPHBog", "access_token_secret": "L8qq9PZyRg6ieKGEKhZolGC0vJWLw8iEJ88DRdyOg" } } Tweet acLon 17
- 19. Timed Rules { "name":”10_sec_timer_rule", "text":"select * from pattern [every timer:interval(10 sec)]", "action":{...} } Rule example: fires each 10 seconds { "name":”2_week_timer_rule", "text":"select * from pattern [every timer:interval(2 weeks)]", "action":{...} } ▪ timer:interval(<number|variable_name> <time-period>) Rule example: fires each 2 weeks Time Periods: • " years" | "year" • " months" | "month “ • " weeks" | "week“ • "days" | "day“ • "hours" | "hour“ • "minutes" | "minute" | "min“ • "seconds" | "second" | "sec“ • "milliseconds" | "millisecond" | "msec" 18
- 20. Timed Rules ▪ timer:at (minutes, hours, days of month, months, days of week [, seconds [, time zone]]). { "name":”5_min_past_hour", "text":"select * from pattern [every timer:at(5, *, *, *, *)]", "action":{...} } Rule example: fires once an hour, at minute five { "name":”17_PST", "text":"select * from pattern [timer:at (0, 17, *, *, *, *, 'PST')]", "action":{...} } Rule example: only fires 1 time at 5:00 pm Pacific Standard Time 19
- 21. Timed Rules ▪ timer:schedule(repetitions, date, period) { "name":"specific_date", "text":"select * from pattern[every timer:schedule(repetitions: 2, date: '2018-03-01T13:00:00Z', period: 1 year 2 month 10 days 2 hours 30 minutes)]", "action":{...} } Rule example: fires 2 times, first callback (2018-03-01 at 13:00:00 UTC) second callback (2019-05-11 at 15:30:00 UTC) { "name":"next_minute", "text":"select * from pattern[every timer:schedule(date: '2019-10-01T05:52:00Z')]", "action":{...} } Rule example: fires on 2019-10-01T05:52:00Z Note: There are many other kind of timed Esper rules, such as time_batch , ext_timed_batch, match_recognize intervals, etc. 20
- 22. Summary: Terms 21 ● CEP, Complex Event processing is event processing that combines data from mulBple sources to infer events or paYerns that suggest more complicated circumstances. (source: Wikipedia). ● Esper is an event stream processor that aims to enable a short development cycle from incepBon to producBon for these types of applicaBons. (source: EsperTech). ● EPL, Event Processing Language (EPL) defined to be the Esper language and designed for Complex Event Processing and Streaming AnalyBcs. (source: EsperTech).
- 23. References 22 ● FIWARE Catalogue o https://www.fiware.org/developers/catalogue ● FIWARE Academy o https://fiware-academy.readthedocs.io/en/latest/index.html ● Perseo documentation o https://perseo.readthedocs.io/en/latest/
- 24. References 23 ● GitHub repos o Perseo Core (https://github.com/telefonicaid/perseo-core) o Perseo Front-End (https://github.com/telefonicaid/perseo-fe) ● Docker images o Perseo Core (https://hub.docker.com/r/fiware/perseo-core) o Perseo Front-End (https://hub.docker.com/r/fiware/perseo) ● Esper Doc o http://esper.espertech.com/release-7.1.0/esper-reference/html/index.html
- 27. 2 6