Persistent Memory Programming with Pmemkv
1 like687 views
The document outlines the features and technical goals of pmemkv, a key/value store designed for persistent memory, highlighting its simple API, language bindings, and flexibility in engine configurations. It emphasizes community involvement, open-source development, and support for multiple programming languages including C++, Java, Node.js, Ruby, and upcoming Python. A roadmap for future releases and additional functionality, including various storage engines and continuous integration, is also provided.
 { LOG(" visited: " << k); });
s = kv->Remove("key1");
assert(s == OK && !kv->Exists("key1"));
delete kv;
return 0;
}](https://image.slidesharecdn.com/persistentmemoryprogrammingmadeeasywithpmemkv-robdickinson-190617165942/85/Persistent-Memory-Programming-with-Pmemkv-6-320.jpg)
![SPDK, PMDK & Vtune™ Summit 8
pmemkvsampleprogram -- JAVA
public static void main(String[] args)
{
KVEngine kv = new KVEngine("vsmap", "{"path":"/dev/shm/"}");
kv.put("key1", "value1");
assert kv.count() == 1;
assert kv.get("key1").equals("value1");
kv.all((String k) -> System.out.println(" visited: " + k));
kv.remove("key1");
assert !kv.exists("key1");
kv.stop();
}
NOTE:
Binding based on JNI
Throw exception on error
Supports String, byte[] and ByteBuffer
Overloaded iterator methods/types](https://image.slidesharecdn.com/persistentmemoryprogrammingmadeeasywithpmemkv-robdickinson-190617165942/85/Persistent-Memory-Programming-with-Pmemkv-8-320.jpg)
![SPDK, PMDK & Vtune™ Summit 11
sourcesoflatencyWithinHIGH-LEVELBINDINGS
• # of round trips between high-level language & native code
• pmemkv language bindings (one round trip per operation)
• Copy key/value data between persistent memory and high-level object
• Create high-level object (string, byte[], reference type, callback/closure)
• Depending on the type of high-level object…
• Translate bytes to UTF-8
• String interning, reference counting or GC
• pmemkv core (native code)
• Searching indexes in DRAM
• Updating indexes in DRAM
• Managing transactions
• Allocating persistent memory
• Persistent Memory (read & write latencies)](https://image.slidesharecdn.com/persistentmemoryprogrammingmadeeasywithpmemkv-robdickinson-190617165942/85/Persistent-Memory-Programming-with-Pmemkv-11-320.jpg)
Persistent Memory Programming with Pmemkv
- 1. Key/Value Store for Persistent Memory Rob Dickinson -- pmemkv Technical Lead
- 2. SPDK, PMDK & Vtune™ Summit Agenda • Why pmemkv? • Key/Value API • Language Bindings • Storage Engines • Roadmap 2
- 3. SPDK, PMDK & Vtune™ Summit 3 Persistent memory programming is difficult There are many languages in the cloud Key/value datastores are increasingly popular → Large addressable market of cloud developers for an easy KV store for persistent memory whypmemkv?
- 4. SPDK, PMDK & Vtune™ Summit 4 Technical: Local key/value store (no networking) Idiomatic language bindings Simple, familiar, bulletproof API Easily extended with new engines Optimized for persistent memory (limit copying to/from DRAM) Flexible configuration, not limited to a single storage algorithm Community: Open source, developed in the open Friendly licensing Outside contributions are welcome Collaboration alternative for folks frustrated with other OS projects Intel provides stewardship, validation on real hardware, and code reviews Standard/comparable benchmarks goalsforpmemkv
- 5. SPDK, PMDK & Vtune™ Summit 5 Engine Lifecycle: Start ( engine_name, json_config ) → engine Stop ( engine ) Engine Operations: engine.Put ( key, value ) engine.Get ( key, value_or_callback ) engine.Exists ( key ) → true/false engine.Remove ( key ) Iteration Operations: (optional) engine.Count ( ) → count engine.All ( key_callback ) engine.Each ( key_and_value_callback ) plus range iteration versions of these! simple&portablekey/valueAPI
- 6. SPDK, PMDK & Vtune™ Summit 6 pmemkvsampleprogram -- C++ int main() { KVEngine* kv = KVEngine::Start("vsmap", "{"path":"/dev/shm/"}"); KVStatus s = kv->Put("key1", "value1"); assert(s == OK && kv->Count() == 1); string value; s = kv->Get("key1", &value); assert(s == OK && value == "value1"); kv->All([](const string& k) { LOG(" visited: " << k); }); s = kv->Remove("key1"); assert(s == OK && !kv->Exists("key1")); delete kv; return 0; }
- 7. SPDK, PMDK & Vtune™ Summit 7 Implementations: C++ (core) C (extern "C" on C++ core) Java (v8 or higher) NodeJS (v6.10 or higher) Ruby (v2.2 or higher) Python (coming soon!) Goals: No modifications to languages or runtime environments Designed for multi-language use Familiar object model Familiar error handling Language-specific types Heap offloading (where applicable) Consistent unit tests IdiomaticLANGUAGEBINDINGS
- 8. SPDK, PMDK & Vtune™ Summit 8 pmemkvsampleprogram -- JAVA public static void main(String[] args) { KVEngine kv = new KVEngine("vsmap", "{"path":"/dev/shm/"}"); kv.put("key1", "value1"); assert kv.count() == 1; assert kv.get("key1").equals("value1"); kv.all((String k) -> System.out.println(" visited: " + k)); kv.remove("key1"); assert !kv.exists("key1"); kv.stop(); } NOTE: Binding based on JNI Throw exception on error Supports String, byte[] and ByteBuffer Overloaded iterator methods/types
- 9. SPDK, PMDK & Vtune™ Summit 9 pmemkvsampleprogram -- nodejs const kv = new KVEngine('vsmap', '{"path":"/dev/shm/"}'); kv.put('key1', 'value1'); assert(kv.count === 1); assert(kv.get('key1') === 'value1'); kv.all((k) => console.log(` visited: ${k}`)); kv.remove('key1'); assert(!kv.exists('key1')); kv.stop(); NOTE: Maintained by Intel JSTC Binding based on NAPI Throw exception on error Buffer support coming soon!
- 10. SPDK, PMDK & Vtune™ Summit 10 pmemkvsampleprogram -- RUBY kv = KVEngine.new('vsmap', '{"path":"/dev/shm/"}') kv.put('key1', 'value1') assert kv.count == 1 assert kv.get('key1').eql?('value1') kv.all_strings {|k| puts " visited: #{k}"} kv.remove('key1') assert !kv.exists('key1') kv.stop NOTE: Binding based on FFI Throw exception on error Supports string & byte array Iterator methods with type names
- 11. SPDK, PMDK & Vtune™ Summit 11 sourcesoflatencyWithinHIGH-LEVELBINDINGS • # of round trips between high-level language & native code • pmemkv language bindings (one round trip per operation) • Copy key/value data between persistent memory and high-level object • Create high-level object (string, byte[], reference type, callback/closure) • Depending on the type of high-level object… • Translate bytes to UTF-8 • String interning, reference counting or GC • pmemkv core (native code) • Searching indexes in DRAM • Updating indexes in DRAM • Managing transactions • Allocating persistent memory • Persistent Memory (read & write latencies)
- 12. SPDK, PMDK & Vtune™ Summit 12 relativeperformanceofBINDINGS NOTE: Single-threaded benchmarks on emulated persistent memory, 16 byte keys & values, C++ all/each pass pointers only, C++ & Java benchmarks use pre-generated keys, Ruby & NodeJS bindings use UTF-8 strings built during the benchmark
- 13. SPDK, PMDK & Vtune™ Summit 13 relativeperformanceofBINDINGS NOTE: Single-threaded benchmarks on emulated persistent memory, 16 byte keys & values, C++ all/each pass pointers only, C++ & Java benchmarks use pre-generated keys, Ruby & NodeJS bindings use UTF-8 strings built during the benchmark
- 14. SPDK, PMDK & Vtune™ Summit 14 Implementations: cmap (PMDK concurrent map) vsmap (memkind) vcmap (memkind) tree3 / stree (experimental) caching (experimental) Goals: Applications can use multiple engines Engines are optimized for different workloads & capabilities (threading, persistence, iterating, sorting) All engines work with all utilities and language bindings Engines can use other engines Engines from the community storageengines
- 15. SPDK, PMDK & Vtune™ Summit 15 AVAILABLEENGINES
- 16. SPDK, PMDK & Vtune™ Summit 16 contributingENGINESiseasy https://github.com/pmem/pmemkv/blob/master/CONTRIBUTING.md#engines
- 17. SPDK, PMDK & Vtune™ Summit 17 1.0 Release: C/C++, Java, JavaScript, Ruby bindings cmap, vmap, vcmap engines Automated tests & CI environment pmemkv_bench (port of db_bench) Q2 2019 (estimated) 1.1 Release: Python bindings csmap, caching engines Third-party engines? Pool partitioning Q4 2019 (estimated) pmemkvroadmap
- 18. SPDK, PMDK & Vtune™ Summit 18 calltoaction Star/watch us on GitHub! https://github.com/pmem/pmemkv