Efficient Shared Data in Perl
- 1. Efficient Shared Data in Perl Perrin Harkins
- 2. What’s your problem? Apache is multi-process Process assignment is random Information wants to be shared Inter-process data sharing is ad hoc
- 3. Sharing is good for Sessions Caching Usually transient data Otherwise, use a RDBMS
- 4. Approaches Files One big file One file per record DBM Shared memory Seems like the obvious choice, but… RDBMS
- 5. Playing well together Atomic updates Prevents corruption Exclusive Locking Prevents lost updates Without this, last save wins Perl Fund Blossom Buttercup $100 $105 $2100 $100
- 6. Cache::Cache Consistent interface to multiple storage methods File system Shared memory via IPC::ShareLite Many cache-related features built in Expiration times Size limit Multiple namespaces
- 7. Cache::Cache, continued Atomic updates Easy to install No compiler needed for file-based storage Benchmarks are on backend storage classes Cache::FileBackend not Cache::FileCache
- 8. Cache::Mmap Uses one big mmap’ed file Many tuning options Size of blocks Size of locking regions Optimization for scalar data Uses locks internally Requires compiler
- 9. MLDBM::Sync Extension of MLDBM Originally developed for Apache::ASP Uses lock file, tie/untie Choice of DBM types SDBM is fastest, but limited Tied interface Locks on entire database Explicit locking in API Can run with standard library
- 10. BerkeleyDB Not DB_File, BerkeleyDB.pm Requires Berkeley DB library from sleepycat.com Tricky to install on some systems Tied or OO interface No built-in support for complex data structures Locks on entire database or on pages Supports transactions Shared memory cache Tests are on BTree
- 11. IPC::MM Interface for Engelschall’s mm Implements shared BTree and Hash in C Tied interface Data is not persistent Only shares between related processes
- 12. Tie::TextDir Dirt-simple: one record per file Keys must be legal file names No compiler needed Doesn’t handle complex data structures
- 13. IPC::Shareable Very Perlish and transparent Shared memory Lots going on under the hood Explicit locking supported Tied interface Requires a compiler
- 14. DBD::SQLite Fast, single-file SQL engine in a DBD Full transaction support! Locking between processes at database level
- 15. DBD::MySQL Adds network capabilities Atomic updates or transactions More work than most to set up
- 16. memcached Networked daemon Intended for clusters Non-blocking I/O Clients for Perl, PHP, Java Requires a Linux kernel patch, until 2.6 is out
- 17. Testing Methodology P4 2.53 Ghz, 512MB RAM, Red Hat 9, ext3, Perl 5.8.0 Abstraction layer IPC::SharedHash Implements new(), fetch(), store() Handles serialization where necessary Calls FETCH() and STORE() instead of using tied interface mod_perl handler ab (Apache Bench)
- 18. Variables Number of parallel clients Percentage of writes Sessions can have a lot of writes Caches are mostly read, by definition Locality of access Scalars vs. complex data
- 20. Effect of Increasing Clients
- 21. Effect of Read/Write Ratio
- 22. Scalars vs. Complex Data Structures
- 23. Latest Results
- 24. Analysis Why is shared memory so slow? Still has to serialize Moving too much data at once What about IPC::MM? Moves one at a time Moving parts are in C Why is the file system so fast? Modern VM system Kernel-managed caching
- 25. Analysis Why is Tie::TextDir faster than Cache::FileBackend? Digest::SHA1 Splitting into multiple directories not normally necessary on modern filesystems: /mu/lt/ip/ledirs
- 26. Problems with this test Size of values not considered Size of overall hash not considered correctly BerkeleyDB should be tested with fancier lock mode Needs a real network test for memchached and MySQL Should try harder to reduce margin of error
- 27. A Word About Clustering Shared filesystems NFS Samba/CIFS RDBMS Most reliable, well understood, easy integration Replicated data Multicast Spread
- 28. What about threads? Apache 2/mod_perl 2/Perl 5.8 bring threads to the table Still not clear how this will work with complex data structures and objects Threaded performance is mostly bad in 5.8
- 29. Questions to help you choose Do you need to store complex data? BerkeleyDB, Tie::TextDir, and IPC::MM require a wrapper for this Are your keys valid filenames? Tie::TextDir does not hash the keys Do you need persistence? IPC::MM is not persistent Do you need explicit locking? MLDBM::Sync, MySQL, BerkeleyDB
- 30. Questions to help you choose No compiler? Cache::FileBackend, Tie::TextDir, MLDBM::Sync if you have Storable Need clustering? DBD::MySQL, memcached