Symas Lightning Memory-Mapped Database (LMDB)
This page contains technical information about Symas's LMDB database library.
LMDB is an ultra-fast, ultra-compact, crash-proof key-value embedded data store developed by Symas for
the OpenLDAP Project. It uses memory-mapped
files, so it has the read performance of a pure in-memory database while still
offering the persistence of standard disk-based databases, and is only limited
to the size of the virtual address space, (it is not limited to the size of
physical RAM).
Note: LMDB was originally called MDB, but was renamed to avoid confusion
with other software associated with the name MDB
Quick Overview
LMDB is a tiny database with some great capabilities:
- Ordered-map interface (keys are always sorted, supports range lookups)
- Fully transactional, full ACID semantics with MVCC.
- Reader/writer transactions: readers don't block writers and writers don't block readers.
Writers are fully serialized, so writes are always deadlock-free.
- Read transactions are extremely cheap, and can be performed using no mallocs or
any other blocking calls.
- Supports multi-thread and multi-process concurrency, environments may be
opened by multiple processes on the same host.
- Multiple sub-databases may be created with transactions covering all sub-databases.
- Memory-mapped, allowing for zero-copy lookup and iteration.
- Maintenance-free, no external process or background cleanup/compaction required.
- Crash-proof, no logs or crash recovery procedures required.
- No application-level caching. LMDB fully exploits the operating system's buffer cache.
- 32KB of object code and 6KLOC of C.
- Licensed under the OpenLDAP Public License
It is a read-optimized design and performs reads several times faster than other DB engines,
several orders of magnitude faster in many cases. It is not a write-optimized design; for heavy
random write workloads other DB designs may be more suitable.
Feature Comparison
A quick comparison of popular embedded key/value stores.
| LMDB | BDB | LevelDB | Kyoto TreeDB |
| ACID Transactions | x | x | |
|
| Nested Transactions | x | x | |
|
| Multiple Namespaces | x | x | |
|
| Sorted Keys | x | x | x | x
|
| Sorted Duplicate Keys | x | x | |
|
| Multi-thread concurrency | x | x | x | x
|
| Multi-process concurrency | x | x | |
|
| No cache tuning: zero-config | x | | |
|
| Instantaneous crash recovery | x | | |
|
| Zero-copy reads | x | | |
|
| Zero-copy writes | x | | |
|
| Atomic hot backup | x | | |
|
Documentation
The full docs (generated by doxygen) are here.
Source
Development occurs in
the OpenLDAP Project's git repo in the mdb.master branch.
A clone of just the LMDB code is also available on
Github.
Support
Commercial Support
While not required, Symas offers fixed-price commercial support to those using LMDB in mission-critical applications. Send email inquiries to sales@symas.com or call us at (650) 963-7601.
Porting and Professional Services
Thinking about switching your application to LMDB? Call us. We will be happy to provide developer support or porting services to help you transition your application to LMDB.
Community
Questions and discussions of LMDB should be directed to the
OpenLDAP mailing lists.
Published Reports on LMDB
-
This library was first presented at the LDAPCon in October 2011.
Paper, Slides.
-
It was also presented to the BayLISA User Group in February 2012.
-
An updated presentation was given at the UKUUG Spring 2012 Conference in March 2012.
The paper and presentation are also available here:
Paper, Slides.
-
It was presented again at the Ubuntu Developer Summit in May 2012 with more adoption.
- The talk at LinuxCon North America in August 2012 is now online here: Paper, Slides.
- The talk at LinuxCon Europe on November 7 2012 is now online here: Slides. This talk goes into greater detail
about how LMDB works, and gives examples so developers can quickly get going with LMDB.
- The LinuxCon talk was presented again at the Dublin University Internet Society on March 20, 2013.
- The talk at Devoxx France in Paris March 29, 2013 is online here: Slides,
Video.
- The talk at LOADAYS in Antwerp April 6-7 2013 is online here: Slides.
- Meetings were held at the Linux Foundation Collaboration Summit in San Francisco April 15-17 2013 to work with other projects on adopting LMDB.
- Researchers from the University of Wisconsin studied application crash-consistency of a number of
systems including LMDB. Their preliminary report from Spring 2014 is here and the final report was presented at the Usenix OSDI conference October 2014. They report finding a single
reliability flaw in LMDB but further discussion shows the flaw is not real.
- Researchers from HP Labs and Ohio State University also conducted reliability tests of LMDB and presented
their results at Usenix OSDI. Again, they report a single flaw in LMDB, but it's confirmed to be a filesystem bug.
- The talk at BuildStuffLT in Vilnius November 20 2014 is online here: Slides
- LMDB was presented at QCon London on March 5 2015, that talk is online here InfoQ.com
- A technical introduction to LMDB was presented at the SNIA Storage Developer Conference in Santa Clara September 21 2015 SNIA SDC
- LMDB was presented as part of the Databaseology lecture series at Carnegie-Mellon University October 8 2015 CMU
Benchmarks
- 2012-09
-
Some microbenchmarks comparing it to other embedded databases are here.
The tests compare OpenLDAP LMDB against Google LevelDB, SQLite 3, Kyoto Cabinet, and BerkeleyDB.
The tests were conducted against multiple filesystems, including btrfs, ext2, ext3, ext4,
jfs, ntfs, xfs, and zfs.
LMDB is several orders of magnitude faster than everything else for reads, while also
being several times smaller. Surprisingly, while LMDB's focus is read speed, it is also
decently fast for writes, and unmatched for bulk load speed.
- 2013-05
-
Further testing using Memcached can be seen here. The
tests compare LMDB against BerkeleyDB, MySQL 5.6 InnoDB, and pure-memory Memcached.
Again, LMDB is multiple orders of magnitude faster than other disk-based storage engines,
and is even faster than the pure-memory Memcached in multi-threaded workloads.
- 2013-05
-
Tests comparing the read performance of LMDB to BerkeleyDB in OpenLDAP here.
- 2013-08
-
A benchmark with HyperDex, using both its original LevelDB-based backend,
and with LMDB, is now available here.
- 2013-09
- Additional tests with HyperDex are available here. These
tests are longer duration and with smaller records, showing some areas where LMDB is weaker.
- 2013-11
- LDAP server benchmarks showing OpenLDAP using LMDB and BDB, as well as most of the major
open source and proprietary directory servers were presented at LDAPCon 2013.
Products tested included OpenDJ, 389DS, ApacheDS, and servers from Microsoft, CA, Oracle, and Novell.
None of the others are anywhere near as efficient as OpenLDAP on LMDB.
The slides are available on Lanyrd.
- 2014-06
-
A benchmark with RocksDB and other LevelDB-based engines on In-Memory workloads
is available here.
- 2014-09
-
Scalability testing for In-Memory workloads, building on the previous tests here.
- 2014-11
-
Scalability testing for On-Disk workloads, across a varying range of data sizes here.
- 2015-02
-
In-memory testing using different malloc libraries here.
- 2015-02
-
In-memory testing using different compression libraries here.
Many other databases claim to be lightweight, but none of them can beat LMDB's compact 32K of object code.
Many other databases also claim to be world's fastest, but none of them are in LMDB's league...
LMDB In Other Projects
Besides the LMDB backend for OpenLDAP slapd (back-mdb), support for LMDB is also available in
these open source projects:
- Cyrus SASL
- Patches posted to the SASL mailing list
- Heimdal Kerberos
- Support is available in the development branch
- SQLite3
- Source code available on Github
- OpenDKIM
- Domain Keys Identified Mail, integrated since version 2.6.0
- MemcacheDB
- Memcached with persistent storage using LMDB, on Github.
Tests with memcachetest shows that LMDB is several times faster than BerkeleyDB, and faster even than the pure-memory memcached.
- Mapkeeper
- Thrift-based storage server.
Test results are available using the Yahoo! Cloud Serving Benchmark (YCSB).
- Replicant
- A replicated state machine manager for HyperDex.
Source code available on Github
- HyperDex
- A reliable and fast NoSQL server (distributed key/value store).
Source code available on Github
- SQLite4
- (Coming soon)
- Postfix
- Fast and secure email server. Support integrated since 2.11
- CfEngine
- IT infrastructure automation framework. Default backend as of 3.6.0
- Redis
- Near-realtime disk persistence Second strike with Lightning, code on
Github
- Ardb
- A redis-protocol compatible persistent storage server. On Github
- SkyDB
- High performance analysis of behavioral data. Formerly used LevelDB but switched to
LMDB due to concurrency and performance issues in LevelDB.
- FineDB
- A new high-performance noSQL database built on LMDB.
- PowerDNS
- An Authoritative DNS server written in C++; delivering Very High Performance using LMDB.
- Hustle
- A column oriented, embarrassingly distributed relational NoSQL database.
- UrBackup
- A client/server file backup system.
- Consul
- Distributed service discovery and configuration.
- BAREOS
- Backup Archiving REcovery Open Sourced. Integrated in 14.2
- Caffe
- Deep learning / computer vision / neural network
- Monero
- Crypto currency
- CHI-Driver
- Cache driver for perl
- ledisdb
- Another redis clone, written in go
- DVID
- Distributed, versioned, image-oriented datastore
- sph-dg
- guile graph database
- StoreMate
- Building block for distributed storage systems
- InfluxDB
- A distributed time series database
- thundergraph
- Another graph DB on LMDB
- Armory
- A Bitcoin client on LMDB
- Cryptonite
- Another Bitcoin blockchain implementation built on LMDB
- mod_authn_lmdb
- An Apache authentication module
- Extenium
- Another time series database built on LMDB
- LightningQueues
- Fast persistent queues for .Net
- rotz
- Another graph database on LMDB
- Whoosh
- Full text indexer in python, using LMDB
- grapheekdb
- A fast and lightweight graph database in python
- ngraph-lmdb
- Persistence for the ngraph graphing library
- Slumber DB
- Key value store for JSON / REST
- ueberDB
- A key value store abstraction layer for node.js
- Similaria
- Scala library implementing an item-based recommandation engine
- AngstromDB
- A tutorial/demo NoSQL database on LMDB
- predis
- A redis clone written in perl
- raft-mdb
- An LMDB backend for the Raft consensus protocol
- LibPaxos
- The core of the Paxos consensus protocol
- CryptonorDB
- privacy aware cloud-mobile database
- Minerva
- a fast and flexible system for deep learning
- KnotDNS
- High-performance authoritative-only DNS server
- Haskell VCache
- large, persistent, memcached values and structure sharing for Haskell
- Flotilla
- Raft-based consensus, embedded and programmable database for building distributed databases
- PearlDB
- A Durable HTTP Key-Value Pair Database
- ActorDB
- Distributed SQL database with linear scalability
- FastoNoSQL
- GUI manager for various NoSQL databases
Wrappers for Other Languages
The compact LMDB code can easily be called from other languages using their respective wrappers.
All of these wrappers were developed by 3rd parties. If you need assistance with them, we can
work with you and the wrapper authors as needed.
- Erlang wrapper
- Source code available on Github,
developed for use with Riak but suitable for other Erlang apps as well.
- C++ wrapper
- Source code available here
- C++11 wrapper
- (Newer than the above) on Github
- Python wrapper
- Source code available on Github
- Python wrapper
- (Another one) available on Github
- Lua wrapper
- Source code available on Github
- Ruby wrapper
- Source code available on Github
- Go wrapper
- Source code available on Github
- Objective C wrapper
- Source code available on Github
- Node.js wrapper
- Source code available on Github
- Java wrapper
- Using JNI, available on Github
- .NET wrapper
- Source code and binaries available on Github
- Perl wrapper
- Source code available on CPAN
- Crack
- Bindings and simplified wrappers in the main Crack source tree.
- Rust wrapper
- Source code available on rust-ci.org
- OCaml wrapper
- Source code available on Github
- PHP wrapper
- Source code available on Github
- PHP wrapper
- Newer more extensive version available on Github
- Haskell wrapper
- Project page here
- Julia driver
- Source code available on Github
- Matlab wrapper
- Source code available on Github
- mruby wrapper
- Source code available on Github
- Nim wrapper
- Source code available on Github
- Common Lisp wrapper
- Guide available here
- TCL wrapper
- Project page here
- Clojure wrapper
- Blog post here
Since the LMDB API is similar to BerkeleyDB, porting existing BDB-based code to LMDB is usually quite simple
and fast. More projects will be adapted as they come to our attention.
Page last updated 2016-04-18