Release Notes - Bookkeeper - Version 4.3.0

NOTE: After support protobuf in [BOOKKEEPER-582] in 4.3.0, a 4.3.0 client will not work with a 4.2.3 server, while a 4.3.0 server still keeps backward compatability with 4.2.3 clients. Please upgrade servers in prior to clients.

Sub-task

• [BOOKKEEPER-312] - Implementation of JMS provider
• [BOOKKEEPER-506] - Provide better topic release algorithm
• [BOOKKEEPER-555] - Make BookieServer use Netty rather than a custom IO server
• [BOOKKEEPER-565] - Make an option to separate storing entry log files from index files.
• [BOOKKEEPER-574] - Extend the bookkeeper shell to get a list of available bookies
• [BOOKKEEPER-576] - Bookie client should use netty Decoder/Encoder
• [BOOKKEEPER-582] - Make bookie and client use protobuf for requests (non-wire part)
• [BOOKKEEPER-595] - Crash of inprocess autorecovery daemon should not take down the bookie
• [BOOKKEEPER-608] - Make SyncThread a reusable component
• [BOOKKEEPER-615] - Twitter stats implementation of stats interface
• [BOOKKEEPER-629] - Support hostname based ledger metadata to help users to change IP with existing installation
• [BOOKKEEPER-643] - Improve concurrency of entry logger
• [BOOKKEEPER-644] - Provide a bookie address wrapper
• [BOOKKEEPER-657] - Journal Improvement
• [BOOKKEEPER-658] - ledger cache refactor
• [BOOKKEEPER-659] - LRU page management in ledger cache.
• [BOOKKEEPER-696] - stats collection on bookkeeper client
• [BOOKKEEPER-697] - stats collection on bookkeeper server
• [BOOKKEEPER-703] - Document all the settings added in BOOKKEEPER-656
• [BOOKKEEPER-704] - reconnectable zookeeper client wrapper
• [BOOKKEEPER-715] - bookie: delay dropping journal cached pages
• [BOOKKEEPER-716] - padding writes for bookie journal
• [BOOKKEEPER-717] - journal should look forward to group time-out entries
• [BOOKKEEPER-773] - Provide admin tool to rename bookie identifier in Cookies
• [BOOKKEEPER-782] - Use builder pattern for Cookie
• [BOOKKEEPER-787] - Modify Cookie by removing 'znodeVersion' state/field

Bug

• [BOOKKEEPER-313] - Bookkeeper shutdown call from Bookie thread is not shutting down server
• [BOOKKEEPER-363] - Re-distributing topics among newly added hubs.
• [BOOKKEEPER-446] - BookKeeper.createLedger(..) should not mask the error with ZKException
• [BOOKKEEPER-544] - Modify hedwig server tests to allow client testcases to start/stop them as part of their tests
• [BOOKKEEPER-548] - Document about periodic ledger checker configuration
• [BOOKKEEPER-549] - Documentation missed for readOnlyMode support
• [BOOKKEEPER-554] - fd leaking when move ledger index file
• [BOOKKEEPER-556] - BookieServerMXBean#getServerState makes no sense
• [BOOKKEEPER-557] - Compiler error showing up badly with jdk 7
• [BOOKKEEPER-559] - Fix occasional failure in AuditorBookieTest
• [BOOKKEEPER-561] - Findbugs reports errors with openjdk
• [BOOKKEEPER-567] - ReadOnlyBookieTest hangs on shutdown
• [BOOKKEEPER-568] - NPE during GC with HierarchicalLedgerManager
• [BOOKKEEPER-569] - Critical performance bug in InterleavedLedgerStorage
• [BOOKKEEPER-577] - BookieFailureTest uses sync/wait()/notify() incorrectly
• [BOOKKEEPER-579] - TestSubAfterCloseSub was put in a wrong package
• [BOOKKEEPER-580] - improve close logic
• [BOOKKEEPER-581] - Ledger recovery doesn't work correctly when recovery adds force changing ensembles.
• [BOOKKEEPER-583] - Read from a ReadOnlyBookie fails if index fileinfo is not in ledger cache
• [BOOKKEEPER-584] - Data loss when ledger metadata is overwritten
• [BOOKKEEPER-585] - Auditor logs noisily when a ledger has been deleted
• [BOOKKEEPER-586] - Remove recursive call in delivery manager
• [BOOKKEEPER-596] - Ledgers are gc'ed by mistake in MSLedgerManagerFactory.
• [BOOKKEEPER-597] - Add flag to output test logs to stdout
• [BOOKKEEPER-598] - Fails to compile - RESUBSCRIBE_EXCEPTION conflict
• [BOOKKEEPER-599] - NPE in PerChannelBookieClient
• [BOOKKEEPER-600] - shouldClaim flag isn't cleared for hedwig multiplex java client
• [BOOKKEEPER-601] - readahead cache size isn't updated correctly
• [BOOKKEEPER-602] - we should have request timeouts rather than channel timeout in PerChannelBookieClient
• [BOOKKEEPER-603] - Support Boost 1.53 for Hedwig Cpp Client
• [BOOKKEEPER-604] - Ledger storage can log an exception if GC happens concurrently.
• [BOOKKEEPER-607] - Filtered Messages Require ACK from Client Causes User Being Throttled Incorrectly Forever
• [BOOKKEEPER-610] - Make SyncThread use an executor
• [BOOKKEEPER-617] - BOOKKEEPER-544 breaks hedwig-server/bin/hedwig script
• [BOOKKEEPER-619] - Bookie should not create local cookie files if zookeeper is uninitialized
• [BOOKKEEPER-620] - PerChannelBookieClient race during channel disconnect
• [BOOKKEEPER-623] - LedgerChecker should avoid segments of closed ledger with higher start entryId than closed entry.
• [BOOKKEEPER-624] - Reduce logs generated by ReplicationWorker
• [BOOKKEEPER-626] - BOOKIE_EXTRA_OPTS are added twice
• [BOOKKEEPER-632] - AutoRecovery should consider read only bookies
• [BOOKKEEPER-633] - ConcurrentModificationException in RackawareEnsemblePlacementPolicy when a bookie is removed from available list
• [BOOKKEEPER-635] - jenkins build should highlight which lines of the patch cause raw analysis errors
• [BOOKKEEPER-636] - Latest txn logs might be deleted in a race condition which is not recoverable if BK goes down before next txn log created.
• [BOOKKEEPER-638] - Two bookies could start at the same time to access bookie data.
• [BOOKKEEPER-641] - DeathWatcher thread is unnecessarily running even after bookie shutdown
• [BOOKKEEPER-642] - Bookie returns incorrect exitcode, ExitCode.ZK_REG_FAIL is getting overridden
• [BOOKKEEPER-646] - BookieShell readjournal command is throwing BufferUnderflowException
• [BOOKKEEPER-649] - Race condition in sync ZKUtils.createFullPathOptimistic()
• [BOOKKEEPER-652] - Logger class name is wrong in LedgerCacheImpl.java
• [BOOKKEEPER-654] - Bookkeeper client operations are allowed even after its closure, bk#close()
• [BOOKKEEPER-663] - HierarchicalLedgerManager iterator is missing some ranges and the last ledger in the range
• [BOOKKEEPER-664] - Compaction increases latency on journal writes
• [BOOKKEEPER-667] - Client write will fail with BadMetadataVersion in case of multiple Bookie failures with AutoRecovery enabled
• [BOOKKEEPER-668] - Race between PerChannelBookieClient#channelDisconnected() and disconnect() calls can make clients hang while add/reading entries in case of multiple bookie failures
• [BOOKKEEPER-673] - Ledger length can be inaccurate in failure case
• [BOOKKEEPER-676] - Make add asynchrounous in ledger recovery
• [BOOKKEEPER-678] - BookieServer shutdown hangs indefinitely at NioServerSocketChannelFactory.releaseExternalResources
• [BOOKKEEPER-683] - TestSubAfterCloseSub fails on 4.2
• [BOOKKEEPER-686] - Bookie startup will fail if one of the configured ledgerDir is full and the same is used for replaying the journal
• [BOOKKEEPER-688] - NPE exception in PerChannelBookieClient
• [BOOKKEEPER-698] - Bookie client closure is not considering timeoutExecutor
• [BOOKKEEPER-700] - GarbageCollectorThread exsiting with ArrayIndexOutOfBoundsException
• [BOOKKEEPER-708] - Shade protobuf library to avoid incompatible versions
• [BOOKKEEPER-709] - SlowBookieTest#testSlowBookie fails intermittently
• [BOOKKEEPER-710] - OpenLedgerNoRecovery should watch ensemble change.
• [BOOKKEEPER-711] - bookkeeper-daemon.sh will not remove the pid file on successful stop
• [BOOKKEEPER-712] - bookkeeper script should use 'java' from JAVA_HOME
• [BOOKKEEPER-714] - Logging channel exceptions in PerChannelBookieClient
• [BOOKKEEPER-718] - AuditorLedgerCheckerTest is flakey
• [BOOKKEEPER-719] - Inconsistent synchronization of org.apache.bookkeeper.stats.CodahaleMetricsProvider.metrics
• [BOOKKEEPER-720] - CheckpointSource.MIN#compareTo does exactly the opposite of what it should
• [BOOKKEEPER-724] - Shade introduces RAT error
• [BOOKKEEPER-725] - AutoRecoveryMain should exit with error code if deathwatcher finds dead thread
• [BOOKKEEPER-726] - PerChannelBookieClient should print address that it failed to connect to when it fails to correct
• [BOOKKEEPER-727] - Names of bookie write/read threads are backwards
• [BOOKKEEPER-728] - Bookkeeper#Builder is not public, so can't be used outside of client package
• [BOOKKEEPER-729] - Bookie shouldn't exit with 0, if exiting from deathwatcher and thread death was caused by OOM
• [BOOKKEEPER-730] - Shade pom file missing apache license header
• [BOOKKEEPER-731] - Missing BOOKIE_PID_DIR and BOOKIE_STOP_TIMEOUT in env variables list
• [BOOKKEEPER-739] - Test timeouts mostly ignored
• [BOOKKEEPER-740] - AutoRecoveryMainTest#testAutoRecoverySessionLoss is failing
• [BOOKKEEPER-742] - Fix for empty ledgers losing quorum.
• [BOOKKEEPER-743] - Periodic ledger check running too often as doc doesn't match implementation.
• [BOOKKEEPER-744] - Run the auditor bookie check periodically
• [BOOKKEEPER-745] - Fix for false reports of ledger unreplication during rolling restarts.
• [BOOKKEEPER-746] - 5 new shell commands. List ledgers, list metadata, list underreplicated, show auditor and simpletest
• [BOOKKEEPER-750] - Flake in BookieAutoRecoveryTest#testEmptyLedgerLosesQuorumEventually
• [BOOKKEEPER-751] - Ensure all the bookkeeper callbacks not run under ledger handle lock
• [BOOKKEEPER-755] - Incorrect number of seconds specified in a day
• [BOOKKEEPER-763] - findbugs fails to run on jenkins
• [BOOKKEEPER-765] - bookkeeper script should fall back to java in path if JAVA_HOME is not set
• [BOOKKEEPER-766] - Update notice.txt files to include 2014
• [BOOKKEEPER-767] - Allow loopback in tests
• [BOOKKEEPER-768] - fix typo 'seconds' to milliseconds in benchmark output
• [BOOKKEEPER-774] - Flaky test org.apache.bookkeeper.test.ReadOnlyBookieTest.testBookieShouldTurnWritableFromReadOnly
• [BOOKKEEPER-777] - Flake in LedgerCloseTest
• [BOOKKEEPER-778] - Flake in TestTryReadLastConfirmed
• [BOOKKEEPER-779] - jmx reporter for codahale metrics provider
• [BOOKKEEPER-780] - Findbug issue in trunk
• [BOOKKEEPER-783] - Avoid running out of fds in MutlipleThreadReadTest
• [BOOKKEEPER-784] - BookKeeperCloseTest#testLedgerCheck is failing intermittently
• [BOOKKEEPER-785] - Fix javadoc warnings in trunk
• [BOOKKEEPER-786] - Fix Findbugs Error In Codahale Stats Provider

Improvement

• [BOOKKEEPER-429] - Provide separate read and write threads in the bookkeeper server
• [BOOKKEEPER-432] - Improve performance of entry log range read per ledger entries
• [BOOKKEEPER-526] - multiple threads for delivery manager
• [BOOKKEEPER-563] - Avoid Journal polluting page cache
• [BOOKKEEPER-564] - Better checkpoint mechanism
• [BOOKKEEPER-573] - Script to start a bookkeeper cluster
• [BOOKKEEPER-592] - allow application to recommend ledger data locality
• [BOOKKEEPER-605] - Use static Logger objects everywhere for bookkeeper
• [BOOKKEEPER-611] - Speed up bookkeeper tests
• [BOOKKEEPER-614] - Generic stats interface, which multiple providers can be plugged into
• [BOOKKEEPER-618] - Better resolution of bookie address
• [BOOKKEEPER-627] - LedgerDirsMonitor is missing thread name
• [BOOKKEEPER-630] - Add tag to o.a.b.net.* to indict which release of hadoop they came from, move DNS to o.a.b.net.* and indent
• [BOOKKEEPER-637] - NoSuchEntry exception when reading an entry from a bookie should not print ERROR level message
• [BOOKKEEPER-640] - Log improvement - add shutdown/exit log message for the bookie services
• [BOOKKEEPER-645] - Bookkeeper shell command to get a list of readonly bookies
• [BOOKKEEPER-661] - Turn readonly back to writable if spaces are reclaimed.
• [BOOKKEEPER-662] - Major GC should kick in immediately if remaining space reaches a warning threshold
• [BOOKKEEPER-666] - Naming threads of ExecutorService
• [BOOKKEEPER-699] - Codahale metrics implementation of stats API
• [BOOKKEEPER-701] - Improve exception handling of Bookkeeper threads
• [BOOKKEEPER-732] - Add env variable ENTRY_FORMATTER_CLASS to the bkenv.sh
• [BOOKKEEPER-747] - Implement register/unregister LedgerMetadataListener in MSLedgerManagerFactory
• [BOOKKEEPER-756] - Use HashedwheelTimer for request timeouts for PCBC
• [BOOKKEEPER-758] - Add TryReadLastAddConfirmed API
• [BOOKKEEPER-781] - Fix OOM on Hedwig Tests

New Feature

• [BOOKKEEPER-257] - Ability to list all ledgers
• [BOOKKEEPER-562] - Ability to tell if a ledger is closed or not
• [BOOKKEEPER-736] - Stats for AutoRecovery

Task

• [BOOKKEEPER-789] - Update README to reflect bookkeeper modules
• [BOOKKEEPER-790] - Add JNA license in NOTICE files

Test

• [BOOKKEEPER-653] - Timeout option is missing in few testcases
• [BOOKKEEPER-775] - Improve MultipleThreadReadTest to reduce flakiness
• [BOOKKEEPER-776] - Flaky test BookieRecoveryTest

Wish

• [BOOKKEEPER-675] - Log noise fixup before cutting 4.2.2