This document is a guide to deploying, administering, and maintaining BookKeeper. It also discusses best practices and common problems.
A typical BookKeeper installation consists of an ensemble of bookies and a ZooKeeper quorum. The exact number of bookies depends on the quorum mode that you choose, desired throughput, and the number of clients using the installation simultaneously.
The minimum number of bookies depends on the type of installation:
- For self-verifying entries you should run at least three bookies. In this mode, clients store a message authentication code along with each entry.
- For generic entries you should run at least four
There is no upper limit on the number of bookies that you can run in a single ensemble.
To achieve optimal performance, BookKeeper requires each server to have at least two disks. It's possible to run a bookie with a single disk but performance will be significantly degraded.
There is no constraint on the number of ZooKeeper nodes you can run with BookKeeper. A single machine running ZooKeeper in standalone mode is sufficient for BookKeeper, although for the sake of higher resilience we recommend running ZooKeeper in quorum mode with multiple servers.
Starting and stopping bookies
$ bin/bookkeeper bookie
To start a bookie in the background, use the
bookkeeper-daemon.sh script and run
$ bin/bookkeeper-daemon.sh start bookie
The instructions above showed you how to run bookies intended for production use. If you'd like to experiment with ensembles of bookies locally, you can use the
localbookie command of the
bookkeeper CLI tool and specify the number of bookies you'd like to run.
This would spin up a local ensemble of 6 bookies:
$ bin/bookkeeper localbookie 6
When you run a local bookie ensemble, all bookies run in a single JVM process.
There's a wide variety of parameters that you can set in the bookie configuration file in
bookkeeper-server/conf/bk_server.conf of your BookKeeper installation. A full listing can be found in Bookie configuration.
Some of the more important parameters to be aware of:
|The TCP port that the bookie listens on|
|A comma-separated list of ZooKeeper servers in |
|The directory where the log device stores the bookie's write-ahead log (WAL)|
|The directories where the ledger device stores the bookie's ledger entries (as a comma-separated list)|
Ideally, the directories specified
ledgerDirectoriesshould be on difference devices.
To enable logging for a bookie, create a
log4j.properties file and point the
BOOKIE_LOG_CONF environment variable to the configuration file. Here's an example:
$ export BOOKIE_LOG_CONF=/some/path/log4j.properties
$ bin/bookkeeper bookie
From time to time you may need to make changes to the filesystem layout of bookies---changes that are incompatible with previous versions of BookKeeper and require that directories used with previous versions are upgraded. If a filesystem upgrade is required when updating BookKeeper, the bookie will fail to start and return an error like this:
2017-05-25 10:41:50,494 - ERROR - [main:Bookie@246] - Directory layout version is less than 3, upgrade needed
BookKeeper provides a utility for upgrading the filesystem. You can perform an upgrade using the
upgrade command of the
bookkeeper CLI tool. When running
bookkeeper upgrade you need to specify one of three flags:
|Performs an upgrade|
|Performs a rollback to the initial filesystem version|
|Marks the upgrade as complete|
A standard upgrade pattern is to run an upgrade...
$ bin/bookkeeper upgrade --upgrade
...then check that everything is working normally, then kill the bookie. If everything is okay, finalize the upgrade...
$ bin/bookkeeper upgrade --finalize
...and then restart the server:
$ bin/bookkeeper bookie
If something has gone wrong, you can always perform a rollback:
$ bin/bookkeeper upgrade --rollback
By default, formatting is done in interactive mode, which prompts you to confirm the format operation if old data exists. You can disable confirmation using the
-nonInteractive flag. If old data does exist, the format operation will abort unless you set the
-force flag. Here's an example:
$ bin/bookkeeper shell metaformat
You can format the local filesystem data on a bookie using the
bookieformat command on each bookie. Here's an example:
$ bin/bookkeeper shell bookieformat
-nonInteractiveflags are also available for the
For a guide to AutoRecovery in BookKeeper, see this doc.
Missing disks or directories
Accidentally replacing disks or removing directories can cause a bookie to fail while trying to read a ledger fragment that, according to the ledger metadata, exists on the bookie. For this reason, when a bookie is started for the first time, its disk configuration is fixed for the lifetime of that bookie. Any change to its disk configuration, such as a crashed disk or an accidental configuration change, will result in the bookie being unable to start. That will throw an error like this:
2017-05-29 18:19:13,790 - ERROR - [main:BookieServer314] – Exception running bookie server : @
If the change was the result of an accidental configuration change, the change can be reverted and the bookie can be restarted. However, if the change cannot be reverted, such as is the case when you want to add a new disk or replace a disk, the bookie must be wiped and then all its data re-replicated onto it.
Run the following command to re-replicate the data:
$ bin/bookkeeper shell recover <oldbookie>
The ZooKeeper server, old bookie, and new bookie, are all identified by their external IP and
bookiePort(3181 by default). Here's an example:
$ bin/bookkeeper shell recover 192.168.1.10:3181
See the AutoRecovery documentation for more info on the re-replication process.