Release Notes - Bookkeeper - Version 4.1.0

Sub-task

• [BOOKKEEPER-95] - extends zookeeper JMX to monitor and manage bookie server
• [BOOKKEEPER-96] - extends zookeeper JMX to monitor and manage hedwig server
• [BOOKKEEPER-97] - collect pub/sub/consume statistics on hub server
• [BOOKKEEPER-98] - collect add/read statistics on bookie server
• [BOOKKEEPER-187] - Create well defined interface for LedgerCache
• [BOOKKEEPER-224] - Fix findbugs in bookkeeper-server component
• [BOOKKEEPER-228] - Fix the bugs in BK benchmark
• [BOOKKEEPER-265] - Review JMX documentation
• [BOOKKEEPER-266] - Review versioning documentation
• [BOOKKEEPER-267] - Review documentation for thread pool configuration
• [BOOKKEEPER-269] - Review documentation for hedwig console client
• [BOOKKEEPER-270] - Review documentation on bookie cookie
• [BOOKKEEPER-271] - Review documentation for message bounding

Bug

• [BOOKKEEPER-40] - BookieClientTest fails intermittantly
• [BOOKKEEPER-56] - Race condition of message handler in connection recovery in Hedwig client
• [BOOKKEEPER-72] - Fix warnings issued by FindBugs
• [BOOKKEEPER-74] - Bookkeeper Persistence Manager should give up topic on error
• [BOOKKEEPER-112] - Bookie Recovery on an open ledger will cause LedgerHandle#close on that ledger to fail
• [BOOKKEEPER-133] - Hub server should update subscription state to zookeeper when losing topic or shutting down
• [BOOKKEEPER-135] - Fencing does not check the ledger masterPasswd
• [BOOKKEEPER-137] - Do not create Ledger index files until absolutely necessary.
• [BOOKKEEPER-140] - Hub server doesn't subscribe remote region correctly when a region is down.
• [BOOKKEEPER-142] - Parsing last log id is wrong, which may make entry log files overwritten
• [BOOKKEEPER-146] - TestConcurrentTopicAcquisition sometimes hangs
• [BOOKKEEPER-148] - Jenkins build is failing
• [BOOKKEEPER-150] - Entry is lost when recovering a ledger with not enough bookies.
• [BOOKKEEPER-152] - Can't recover a ledger whose current ensemble contain failed bookie.
• [BOOKKEEPER-153] - Ledger can't be opened or closed due to zero-length metadata
• [BOOKKEEPER-156] - BookieJournalRollingTest failing
• [BOOKKEEPER-161] - PerChannelBookieClient tries to reuse HashedWheelTimer, throws Exception
• [BOOKKEEPER-162] - LedgerHandle.readLastConfirmed does not work
• [BOOKKEEPER-163] - Prevent incorrect NoSuchLedgerException for readLastConfirmed.
• [BOOKKEEPER-166] - Bookie will not recover its journal if the length prefix of an entry is truncated
• [BOOKKEEPER-167] - PerChannelBookieClient doesn't use ClientConfiguration
• [BOOKKEEPER-169] - bookie hangs on reading header when encountering partial header index file
• [BOOKKEEPER-170] - Bookie constructor starts a number of threads
• [BOOKKEEPER-171] - ServerConfiguration can't use more than one directory for ledgers
• [BOOKKEEPER-173] - Uncontrolled number of threads in bookkeeper
• [BOOKKEEPER-174] - Bookie can't start when replaying entries whose ledger were deleted and garbage collected.
• [BOOKKEEPER-176] - HierarchicalBookieFailureTest Hung
• [BOOKKEEPER-177] - Index file is lost or some index pages aren't flushed.
• [BOOKKEEPER-180] - bookie server doesn't quit when running out of disk space
• [BOOKKEEPER-182] - Entry log file is overwritten when fail to read lastLogId.
• [BOOKKEEPER-184] - CompactionTest failing on Jenkins
• [BOOKKEEPER-186] - Bookkeeper throttling - permits is not released when read has failed from all replicas
• [BOOKKEEPER-188] - Garbage collection code is in the wrong place
• [BOOKKEEPER-189] - AbstractZkLedgerManager doesn't disregard cookies
• [BOOKKEEPER-190] - Add entries would fail when number of open ledgers reaches more than openFileLimit.
• [BOOKKEEPER-193] - Ledger is garbage collected by mistake.
• [BOOKKEEPER-194] - Get correct latency for addEntry operations for JMX.
• [BOOKKEEPER-195] - HierarchicalLedgerManager doesn't consider idgen as a "specialNode"
• [BOOKKEEPER-197] - HedwigConsole uses the same file to load bookkeeper client config and hub server config
• [BOOKKEEPER-198] - replaying entries of deleted ledgers would exhaust ledger cache.
• [BOOKKEEPER-207] - BenchBookie doesn't run correctly
• [BOOKKEEPER-209] - Typo in ServerConfiguration for READAHEAD_ENABLED
• [BOOKKEEPER-210] - Spurious GC error message
• [BOOKKEEPER-211] - Bookie fails to to start
• [BOOKKEEPER-212] - Bookie stops responding when creating and deleting many ledgers
• [BOOKKEEPER-213] - PerChannelBookieClient calls the wrong errorOut function when encountering an exception
• [BOOKKEEPER-215] - Deadlock occurs under high load
• [BOOKKEEPER-216] - Bookie doesn't exit with right exit code
• [BOOKKEEPER-217] - NPE in hedwig client when enable DEBUG
• [BOOKKEEPER-229] - Deleted entry log files would be garbage collected again and again.
• [BOOKKEEPER-231] - ZKUtil.killServer not closing the FileTxnSnapLog from ZK.
• [BOOKKEEPER-232] - AsyncBK tests failing
• [BOOKKEEPER-234] - EntryLogger will throw NPE, if any dir does not exist or IO Errors.
• [BOOKKEEPER-235] - Bad syncing in entrylogger degrades performance for many concurrent ledgers
• [BOOKKEEPER-242] - Bookkeeper not able to connect other zookeeper when shutdown the zookeeper server where the BK has connected.
• [BOOKKEEPER-245] - Intermittent failures in PersistanceManager tests
• [BOOKKEEPER-258] - CompactionTest failed
• [BOOKKEEPER-260] - Define constant for -1 (invalid entry id)
• [BOOKKEEPER-263] - ZK ledgers root path is hard coded
• [BOOKKEEPER-273] - LedgerHandle.deleteLedger() should be idempotent
• [BOOKKEEPER-279] - LocalBookKeeper is failing intermittently due to zkclient connection establishment delay
• [BOOKKEEPER-281] - BKClient is failing when zkclient connection delays
• [BOOKKEEPER-285] - TestZkSubscriptionManager quits due to NPE, so other tests are not run in hedwig server.
• [BOOKKEEPER-286] - Compilation warning
• [BOOKKEEPER-287] - NoSuchElementException in LedgerCacheImpl
• [BOOKKEEPER-288] - NOTICE files don't have the correct year

Improvement

• [BOOKKEEPER-23] - Timeout requests
• [BOOKKEEPER-73] - Add monitoring support
• [BOOKKEEPER-141] - Run extracting ledger id from entry log files in GC thread to speed up bookie restart
• [BOOKKEEPER-145] - Put notice and license file for distributed binaries in SVN
• [BOOKKEEPER-157] - For small packets, increasing number of bookies actually degrades performance.
• [BOOKKEEPER-158] - Move latest benchmarking code into trunk
• [BOOKKEEPER-160] - bookie server needs to do compaction over entry log files to reclaim disk space
• [BOOKKEEPER-172] - Upgrade framework for filesystem layouts
• [BOOKKEEPER-175] - Bookie code is very coupled
• [BOOKKEEPER-178] - Delay ledger directory creation until the ledger index file was created
• [BOOKKEEPER-185] - Remove bookkeeper-server dependency on hadoop-common
• [BOOKKEEPER-196] - Define interface between bookie and ledger storage
• [BOOKKEEPER-200] - Fix format and comments
• [BOOKKEEPER-218] - Provide journal manager to manage journal related operations
• [BOOKKEEPER-236] - Benchmarking improvements from latest round of benchmarking
• [BOOKKEEPER-238] - Add log4j.properties in conf/ for bin packages
• [BOOKKEEPER-251] - Noise error message printed when scanning entry log files those have been garbage collected.
• [BOOKKEEPER-254] - Bump zookeeper version in poms
• [BOOKKEEPER-264] - Documentation for 4.1.0

New Feature

• [BOOKKEEPER-77] - Add a console client for hedwig
• [BOOKKEEPER-165] - Add versioning support for journal files
• [BOOKKEEPER-168] - Message bounding on subscriptions

Task

• [BOOKKEEPER-241] - Add documentation for bookie entry log compaction