zenoh: zero overhead pub/sub store/query compute
1 like1,254 views
The document discusses Zenoh, an innovative networking protocol that integrates data in motion, at rest, and computations to address challenges in data management for IoT, edge, and fog computing. It emphasizes the need for reactive, event-driven architectures and presents Zenoh's capabilities for efficient pub/sub and distributed queries while maintaining low overhead. The document highlights Zenoh's support for geo-distributed storage and various back-end integrations, positioning it as a leading solution for modern data management needs.
![fn data_handler(res_name: &str, payload: RBuf, _data_info: DataInfo) {
println!("FUNCTION >> [Subscription listener] Received ('{}': '{:02x?}')",
res_name, payload);
}
fn main() {
task::block_on( async {
let kexp =“/demo/example/**".to_string();
let z = open().await.unwrap();
let sub_info = SubInfo { reliability: Reliability::Reliable,
mode: SubMode::Push, period: None };
let sub = z.declare_subscriber(kexp.into(), &sub_info, data_handler).await.unwrap();
let mut r = std::io::stdin();
let mut input = [0u8];
while input[0] != 'q' as u8 { r.read_exact(&mut input).unwrap(); }
z.undeclare_subscriber(sub).await.unwrap();
z.undeclare_subscriber(sub2).await.unwrap();
z.close().await.unwrap();
})
}
Rust Subscriber
zenoh.net](https://image.slidesharecdn.com/2020-200528151733/85/zenoh-zero-overhead-pub-sub-store-query-compute-30-320.jpg)
zenoh: zero overhead pub/sub store/query compute
- 2. Context
- 3. Data Management Today Technologies for dealing with data in motion and data at rest have belonged historically to different families Publish/Subscribe is today the leading paradigm for dealing with with data in motion Databases (SQL and NoSQL) are the leading paradigm to deal with data at rest
- 4. Data Management Today Historically, data at rest was predominant in IT systems and data in motion everywhere else, e.g. OT systems. This is however changing. Some more…
- 5. Reactive / Event-Driven The IT world is embracing reactive, event riven architectures and thus in introducing the challenge of dealing with data at rest and in motion in a unified manner.
- 6. IT/OT Convergence The convergence between IT and OT is creating an increasing need to integrate the world of data-at- rest (queries) with that of data- in-motion (pub/sub)
- 7. (I)IoT, Edge and Fog Computing IoT, Edge and Fog Computing are creating new challenges with respect to decentralisation, scalability, efficiency, and location transparent access to geographically- distributed data
- 8. Cloud-Based Solution One common approach is to use the cloud as the place to store and retrieve information. But what about : - Latency ? - Privacy ? - Connectivity ?
- 9. Decentralisation What if we want to keep some of the data locally? That would make sense from energy, processing ands privacy perspectives But if we keep data locally, how can we still provide global access to it?
- 11. Technological Gap Embrace the cloud-centric paradigm, if you can afford the privacy, performance and efficiency implications Get into patchwork design in which multiple protocols and storages are stitched together to provide some meaningful end-to-end semantics. 1 2 Today you really have two choices:
- 12. zenoh
- 13. Unifies data in motion, data in-use, data at rest and computations. It carefully blends traditional pub/sub with distributed queries, while retaining a level of time and space efficiency that is well beyond any of the mainstream stacks. It provides built-in support for geo- distributed storages and distributed computations
- 14. Implements a networking layer capable of running above a Data Link, Network or Transport Layer. This protocol provides primitives for efficient pub/sub and distributed queries. It supports fragmentation and ordered reliable delivery. zenoh.net Provides a high level API for pub/sub and distributed queries, data representation transcoding, an implementation of geo-distributed storage and distributed computed values zenoh Data Link Network Transport Physical zenoh zenoh.net
- 15. Implements a networking layer capable of running above a Data Link, Network or Transport Layer. This protocol provides primitives for efficient pub/sub and distributed queries. It supports fragmentation and ordered reliable delivery. zenoh.net Data Link Network Transport Physical zenoh.net scouting session It provides a pluggable scouting abstraction for discovery Defines and builds upon a session protocol that provides abstractions for ordered best effort and reliable channels with unlimited MTU that are independent of the underlying layer.
- 16. zenoh.net
- 17. Naming Data Following the tradition of Named Data Networking protocols, data is named by a sequence of byte arrays — called key — such as: /home/kitchen/sensors/temp /home/kitchen/sensors/C202 Data interest and intents are expressed by means of keys regular expressions, such as: /home/*/sensors/temp /home/**/C202 zenoh.net
- 18. Selecting Data Uses selector to defines data sets. A selector is composed by a key expression, and optionally a predicate, a projection and a set of properties /myhome/*/sensor/temp?value>25 /mycar/dynamics?speed>25#acceleration uses the key-expression to route the query, but does not interpret the predicate nor the properties. It also provide different policies to control query consolidation and completeness and potentially quorums zenoh.net zenoh.net
- 19. Key Abstractions Resource. A named data, in other term a (key,value) Publisher. A spring of values for a key expression Subscriber. A sink of values for a key expression Queryable. A well of values for a key expression zenoh.net
- 20. Key Primitives zenoh.net open/close — Open/Close a zenoh.net session scout — Looks for zenoh entities, the kinds of relevant nodes, e.g. peers, router, etc., is specified by a bit-mask. declare/undeclare — Declare/Undeclare resource, publisher, subscriber and queryable. Declarations are used for discovery and various optimisations. For subscribers the declare primitive registers a user provided call-back that will be triggered when data is available. For queryable, the declare primitive register a user provided call-back triggered whenever a query needs to be asnwered.
- 21. Key Primitives zenoh.net write — Writes data for a key expression query — Issues a distributed query and returns a stream of results. The query target, coverage and consolidation depends on policies pull — Pulls data for a pull subscriber.
- 23. Push and Pull Push as well as pull subscriber are supported. A push subscriber, will receive data as produced For a pull subscriber, the infrastructure caches the data, as close as possible to allow the subscriber to consume it effectively when ready to pull Time schedules can be negotiated for both push and pull subscribers zenoh.net
- 24. Push, Pull and Query zZ
- 25. Reliability & Ordering Z1 Z2 Z6 Z3 Z5 Z4A1 A2 application-to-application reliability first-to-last-broker Three levels of reliability : • Hop to hop reliability. Ensures reliability and ordering when NO failures. • App-to-app reliability. • First-to-last-broker reliability. More scalable than app-to-app reliability. zenoh.net
- 26. Routing Adaptative, fault tolerant, brokering and routing. zenoh.net
- 27. Protocol Summary Highlights Most wire/power/memory efficient protocol in the market to provide connectivity to extremely constrained targets Supports push and pull pub/sub along with distributed queries Resource keys are represented as integers on the wire, these integer are local to a session => good for wire efficiency Supports for peer-to-peer and routed communication. Ordered reliable data delivery and fragmentation. Minimal wire overhead for user data is 4 bytes User provided attachments can be associated with every protocol message zenoh.net Data Link Network Transport Physical zenoh zenoh.net
- 29. Rust Publisher fn main() { task::block_on( async { let z = open().await.unwrap(); let rid = z.declare_resource(“/demo/example/zenoh-rs-write”.into()).await.unwrap(); let pub = z.declare_publisher(rid).await.unwrap(); let value = "Write from Rust!”.to_bytes(); z.write(&rid, value).await.unwrap(); z.undeclare_publisher(pub).await.unwrap(); z.session.close().await.unwrap(); }) } zenoh.net
- 30. fn data_handler(res_name: &str, payload: RBuf, _data_info: DataInfo) { println!("FUNCTION >> [Subscription listener] Received ('{}': '{:02x?}')", res_name, payload); } fn main() { task::block_on( async { let kexp =“/demo/example/**".to_string(); let z = open().await.unwrap(); let sub_info = SubInfo { reliability: Reliability::Reliable, mode: SubMode::Push, period: None }; let sub = z.declare_subscriber(kexp.into(), &sub_info, data_handler).await.unwrap(); let mut r = std::io::stdin(); let mut input = [0u8]; while input[0] != 'q' as u8 { r.read_exact(&mut input).unwrap(); } z.undeclare_subscriber(sub).await.unwrap(); z.undeclare_subscriber(sub2).await.unwrap(); z.close().await.unwrap(); }) } Rust Subscriber zenoh.net
- 32. P2P Throughput zenoh.net Pub Sub Payload (bytes) Msgs/sec 0 300000 600000 900000 1200000 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 zenoh p2p Mosquito MQTT
- 33. Routed Throughput zenoh.net Pub Sub Payload (bytes) Msgs/sec 0 250000 500000 750000 1000000 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 zenoh brokered Mosquito MQTT
- 34. More Perf… Msgs/sec 0 300000 600000 900000 1200000 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 zenoh p2p Mosquito MQTT zenoh Rust (Broker) msgs/sec Mbps 0 7500 15000 22500 30000 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 zenoh p2p zenoh brokered Mbps 1 100 10000 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 zenoh p2p zenoh brokered
- 35. zenoh
- 36. Provides a high level API for pub/sub and distributed queries, data representation transcoding, an implementation of geo-distributed storage and distributed computed values zenoh Defines a series of supported data encoding, such as JSON, Properties, Relational, Raw, etc., along with transcoding. Defines a canonical query syntax based on URIs syntax. Data Link Network Transport Physical zenoh zenoh.net Distributed Computations Geo-Distributed Storage Pub/ Sub Data / Query Encoding and Transcoding Session Scouting
- 37. Provides a Geo-Distributed Storage and a storage back-end plug-in API. Currently supported storage back-ends are Memory, MySQL, MariaDB, PostgreSQL, SQLite and InfluxDB zenoh Built-in support for transcoding deals with data and query format adaptation across back-ends. By default the Geo-Distributed Storages work under eventual consistency. Stronger consistency can be implemented by user leveraging Quorum Mechanism Storage alignement is provided by the infrastructure. Data Link Network Transport Physical zenoh zenoh.net Distributed Computations Geo-Distributed Storage Pub/ Sub Data / Query Encoding and Transcoding Session Scouting
- 38. Geo Distributed Storage /region01/house01/** /region01/house02/** /region02/house01/** /region02/*/public/** /region01/*/public/** Query: /**/temperature The ownership of data is specified through key expressions. Query are able to resolve data in a location transparent manner zenoh
- 39. A form of distributed computation, called eval, is supported through zenoh.net queryable. zenoh evals allow for representing anything from computed values, to map-reduce and voting. This is provided as a high-level abstraction and an example of the power of zenoh.net queryable Data Link Network Transport Physical zenoh zenoh.net Distributed Computations Geo-Distributed Storage Pub/ Sub Data / Query Encoding and Transcoding Session Scouting
- 40. A form of distributed computation is supported through zenoh.net queryable. zenoh The provided abstraction allows for representing anything from computed values, to map-reduce and voting. This is provided as a high-level abstraction and an example of the power of zenoh.net queryable Data Link Network Transport Physical zenoh zenoh.net Distributed Computations Geo-Distributed Storage Pub/ Sub Data / Query Encoding and Transcoding Session Scouting
- 41. A zero overhead networking protocol that provides composable primitives for pub/sub and generalised distributed queries. zenoh A framework that leverages zenoh.net primitives to provide an opinionated implementation of pub/ sub, geo-distributed storage and computations. zenoh.netvs zenoh.net zenoh Data Link Network Transport Physical zenoh zenoh.net Distributed Computations Geo-Distributed Storage Pub/ Sub Data / Query Encoding and Transcoding Session Scouting
- 43. Coding Lab zenoh
- 44. Soo Incredibly Easy!!! Publish : ws = Zenoh.login().workspace() ws.put('/demo/hello','Hello world’) ws.put(‘/house/livingroom/temp’, 25.5) Subscribe : ws = Zenoh.login().workspace() ws.subscribe(‘/demo/**', lambda data: print('received {}'.format(data))) Query : ws = Zenoh.login().workspace() result = ws.get(‘/demo/hello?(name=World)’) zenoh
- 45. Live Demo zenoh
- 46. us-west.zenoh.io /demo/us-west/** us-east.zenoh.io /demo/us-east/** eu.zenoh.io /demo/eu/** ap.zenoh.io /demo/ap/** Example: • Put data: curl -X PUT -d 'Hello World!' http://us-west.zenoh.io:8000/demo/eu/test • Get data: curl http://ap-southeast.zenoh.io:8000/demo/*/test
- 47. Concluding Remarks zenoh.net is arguably the most innovative and efficient networking protocol available on the market zenoh is the only framework that makes it incredibly easy and ubiquitous to deal with data at rest, data in movement and computations. Data Link Network Transport Physical zenoh zenoh.net Distributed Computations Geo-Distributed Storage Pub/ Sub Data / Query Encoding and Transcoding Session Scouting