Metastore Interface

Although Apache BookKeeper provides LedgerManager and Hedwig Metadata Managers for users to plugin different metadata storages for both BookKeeper and Hedwig, it is quite difficult to implement a correct and efficient manager version based on the knowledge for both projects. The MetaStore interface extracts the commonality of the metadata storage interfaces and is provided for users to focus on adapting the underlying storage itself w/o having to worry about the detailed logic for BookKeeper and Hedwig.


The MetaStore interface provide users with access to MetastoreTable__s used for BookKeeper and Hedwig metadata management. There are two kinds of table defined in a MetaStore, MetastoreTable which provides basic PUT,__GET,__REMOVE__,__SCAN__ operations and which does not assume any ordering requirements from the underlying storage; and MetastoreScannableTable which is derived from MetastoreTable, but does assume that data is stored in key order in the underlying storage.

MetaStore Table

MetastoreTable is a basic unit in a MetaStore, which is used to handle different types of metadata, i.e. A MetastoreTable is used to store metadata for ledgers, while the other MetastoreTable is used to store metadata for topic persistence info. The interface for a MetastoreTable is quite simple:

MetaStore Scannable Table

MetastoreScannableTable is identical to a MetastoreTable except that it provides an addition interface to iterate over entries in the table in key order.

How to organize your metadata.

Some metadata in Hedwig and BookKeeper does not need to be stored in the order of the ledger id or the topic. You could use kind of hash table to store metadata for them. These metadata are topic ownership and topic persistence info. Besides that, subscription state and ledger metadata must be stored in key order due to the current logic in Hedwig/BookKeeper.