This document contains information about deploying, administering and maintaining BookKeeper. It also discusses best practices and common problems.

Running a BookKeeper instance

System requirements

A typical BookKeeper installation comprises a set of bookies and a set of ZooKeeper replicas. The exact number of bookies depends on the quorum mode, desired throughput, and number of clients using this installation simultaneously. The minimum number of bookies is three for self-verifying (stores a message authentication code along with each entry) and four for generic (does not store a message authentication code with each entry), and there is no upper limit on the number of bookies. Increasing the number of bookies will, in fact, enable higher throughput.

For performance, we require each server to have at least two disks. It is possible to run a bookie with a single disk, but performance will be significantly lower in this case.

For ZooKeeper, there is no constraint with respect to the number of replicas. Having a single machine running ZooKeeper in standalone mode is sufficient for BookKeeper. For resilience purposes, it might be a good idea to run ZooKeeper in quorum mode with multiple servers. Please refer to the ZooKeeper documentation for detail on how to configure ZooKeeper with multiple replicas.

Running bookies

To run a bookie, we execute the following command:

bookkeeper-server/bin/bookkeeper bookie

The configuration parameters can be set in bookkeeper-server/conf/bk_server.conf.

The important parameters are:

Ideally, journalDir and ledgerDir are each in a different device. See Bookie Configuration Parameters for a full list of configuration parameters.


From time to time, we may make changes to the filesystem layout of the bookie, which are incompatible with previous versions of bookkeeper and require that directories used with previous versions are upgraded. If you upgrade your bookkeeper software, and an upgrade is required, then the bookie will fail to start and print an error such as:

2012-05-25 10:41:50,494 - ERROR - [main:Bookie@246] - Directory layout version is less than 3, upgrade needed

BookKeeper provides a utility for upgrading the filesystem.
bookkeeper-server/bin/bookkeeper upgrade

The upgrade application takes 3 possible switches, --upgrade, --rollback or --finalize. A normal upgrade process looks like.

  1. bookkeeper-server/bin/bookkeeper upgrade --upgrade
  2. bookkeeper-server/bin/bookkeeper bookie
  3. Check everything is working. Kill bookie, ^C
  4. If everything is ok, bookkeeper-server/bin/bookkeeper upgrade --finalize
  5. Start bookie again bookkeeper-server/bin/bookkeeper bookie
  6. If something is amiss, you can roll back the upgrade bookkeeper-server/bin/bookkeeper upgrade --rollback


BookKeeper uses slf4j for logging, with the log4j bindings enabled by default. To enable logging from a bookie, create a log4j.properties file and point the environment variable BOOKIE_LOG_CONF to the configuration file. The path to the log4j.properties file must be absolute.

export BOOKIE_LOG_CONF=/tmp/log4j.properties
bookkeeper-server/bin/bookkeeper bookie

Missing disks or directories

Replacing disks or removing directories accidentally can cause a bookie to fail while trying to read a ledger fragment which the ledger metadata has claimed exists on the bookie. For this reason, when a bookie is started for the first time, it's disk configuration is fixed for the lifetime of that bookie. Any change to the disk configuration of the bookie, such as a crashed disk or an accidental configuration change, will result in the bookie being unable to start with the following error:

2012-05-29 18:19:13,790 - ERROR - [main:BookieServer@314] - Exception running bookie server :
.......at org.apache.bookkeeper.bookie.Cookie.verify(Cookie.java:82)
.......at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:275)
.......at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:351)

If the change was the result of an accidental configuration change, the change can be reverted and the bookie can be restarted. However, if the change cannot be reverted, such as is the case when you want to add a new disk or replace a disk, the bookie must be wiped and then all its data re-replicated onto it. To do this, do the following:

  1. Increment the bookiePort in bk_server.conf.
  2. Ensure that all directories specified by journalDirectory and ledgerDirectories are empty.
  3. Start the bookie.
  4. Run bin/bookkeeper org.apache.bookkeeper.tools.BookKeeperTools <zkserver> <oldbookie> <newbookie> to re-replicate data. and are identified by their external IP and bookiePort. For example if this process is being run on a bookie with an external IP of, with an old bookiePort of 3181 and a new bookiePort of 3182, and with zookeeper running on zk1.example.com, the command to run would be
    bin/bookkeeper org.apache.bookkeeper.tools.BookKeeperTools zk1.example.com See Bookie Recovery for more details on the re-replication process.

The mechanism to prevent the bookie from starting up in the case of configuration changes exists to prevent the following silent failures:

  1. A strict subset of the ledger devices (among multiple ledger devices) has been replaced, consequently making the content of the replaced devices unavailable;
  2. A strict subset of the ledger directories has been accidentally deleted.

Setting up a test ensemble

Sometimes it is useful to run a ensemble of bookies on your local machine for testing. We provide a utility for doing this. It will set up N bookies, and a zookeeper instance locally. The data on these bookies and of the zookeeper instance are not persisted over restarts, so obviously this should never be used in a production environment. To run a test ensemble of 10 bookies, do the following:

bookkeeper-server/bin/bookkeeper localbookie 10