This is the 16th release of Apache BookKeeper!
The 4.9.0 release incorporates hundreds of bug fixes, improvements, and features since previous major release, 4.8.0, which was released four months ago. It is a new milestone in Apache BookKeeper community.
Apache BookKeeper/DistributedLog users are encouraged to upgrade to 4.9.0. The technical details of this release are summarized below.
The main features in 4.9.0 cover are around following areas:
- Dependencies Changes
- Public API
- Table Service
- Builds & Testing
- Bug Fixes
Here is a list of dependencies changed in 4.9.0:
- Upgrade Jackson from
- Upgrade Jline to
- Upgrade Netty from
- Upgrade TestContainers from
There are multiple new client features introduced in 4.9.0. Here are two highlighted features:
- LedgerHandleAdv exposes
asyncAddEntryvariant that takes ByteBuf
There are bunch of new settings introduced in both bookie and client in 4.9.0. Here are those settings:
serverNumIOThreads: configures the number of IO threads for bookies (see #1612)
- The default value of
fileInfoFormatVersionToWriteis bumped from
1. (see #1689)
- The default value of
journalFormatVersionToWriteis bumped from
6. (see #1689)
numIOThreads: configures the number of IO threads for client (see #1612)
There are a few big changes around metadata in 4.9.0. They are:
- Refactor ledger metadata in LedgerHandle to make ledger metadata instance immutable (see #281)
- Store ledger metadata in binary protobuf format (see details at #723)
- Etcd based metadata driver implementation is in BETA release (see details at #1639)
Additionally, there are bunch of new interfaces introduced in the metadata driver API.
There are a lot of improvements and features introduced into the table service. The maturity of table service is moving from alpha to beta, and has started to be used as the state storage for Pulsar Functions. More table service usage will come in Pulsar’s future releases.
Starting from 4.9.0, bookkeeper will release a python client for table service. See details at #1691
HTTP Admin REST Endpoint
/api/v1/bookie/gc_detailsis introduced to retrieve the GC details.
/api/v1/bookie/gcis introduced to trigger GC through HTTP REST endpoint.
There are are multiple new commands are added in BookieShell. Here are a few highlighted:
regenerate-interleaved-storage-index-filecommand is introduced for rebuilding the index files for interleaved based ledger storage. (#1642)
ledgermetadatacommand now supports dumping/restoring ledger metadata to/from file.
localconsistencycheckcommand is introduce for running consistency check on bookies locally. (#1819)
- a new
bk-perfscript is introduced for running performance benchmark on bookkeeper. (1697)
A new BookKeeper CLI package is released as
bkctl package includes both the existing bookie shell and the new
Mapped Diagnostic Context (MDC) is now supported at both bookie and client sides. Application request context can be passed as context and being logged through slf4j/log4j. This simplifies throubleshooting of request-level failures/errors. See details at #1672.
StatsDoc annotation is introduced in BP-36. The
StatsDoc annotation is
used for documenting stats added across the project.
Builds & Testing
- Java 11 is supported for building bookkeeper.
- Issue 1791: Read Submission should bypass OSE Threads
- A new module is introduced for enabling CPU affinity #1641
- Issue 1682: Added BlockingQueue implementation based on JCtools
- Issue 1813: Set default sizes of DbLedgerStorage read and write cache to be proportional to JVM direct memory
- Issue 1808: Allow to configure sticky reads
- Issue 1754: Netty allocator wrapper
- Issue #1414: Ensure BufferedChannel instance is properly closed
- Issue #1805: Fixed Auth with V2 protocol
- Issue #1769: prevent race between flush and delete from recreating index
- Issue #1807: Fix sorted ledger storage rotating entry log files too frequent
- Issue #1843: DbLedgerStorage should do periodical flush
- Issue #1578: Fixed deadlock in auditor blocking ZK thread
- Issue #1834: Only publish suspect ledgers if they have missing fragments
- Issue #1762: Don’t cache Bookie hostname DNS resolution forever
- Issue #1788: Fix bugs in DefaultEnsemblePlacementPolicy
- Issue #1862: Fix selectFromNetworkLocation in RackawareEnsemblePlacementPolicyImpl
- Issue #1857: changingEnsemble should be negated before calling unset success