Package org.apache.bookkeeper.client

Class BookKeeperAdmin

java.lang.Object

org.apache.bookkeeper.client.BookKeeperAdmin

All Implemented Interfaces:

AutoCloseable

public class BookKeeperAdmin
extends Object
implements AutoCloseable

Admin client for BookKeeper clusters.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
(package private) class	BookKeeperAdmin.LedgerEntriesIterable
(package private) class	BookKeeperAdmin.LedgerEntriesIterator
static class	BookKeeperAdmin.ResultCallBack	This is the class for getting the replication result.
(package private) static class	BookKeeperAdmin.SyncObject

Constructor Summary

Constructors

Constructor	Description
BookKeeperAdmin(String zkServers)	Constructor that takes in a ZooKeeper servers connect string so we know how to connect to ZooKeeper to retrieve information about the BookKeeper cluster.
BookKeeperAdmin(BookKeeper bkc)
BookKeeperAdmin(BookKeeper bkc, ClientConfiguration conf)
BookKeeperAdmin(BookKeeper bkc, StatsLogger statsLogger, ClientConfiguration conf)	Constructor that takes in a BookKeeper instance .
BookKeeperAdmin(ClientConfiguration conf)	Constructor that takes in a configuration object so we know how to connect to ZooKeeper to retrieve information about the BookKeeper cluster.

Method Summary

Modifier and Type	Method	Description
static boolean	areEntriesOfLedgerStoredInTheBookie(long ledgerId, BookieId bookieAddress, LedgerMetadata ledgerMetadata)
static boolean	areEntriesOfLedgerStoredInTheBookie(long ledgerId, BookieId bookieAddress, LedgerManager ledgerManager)
void	asyncGetLedgersContainBookies(Set<BookieId> bookies, BookkeeperInternalCallbacks.GenericCallback<SortedMap<Long,LedgerMetadata>> callback)
CompletableFuture<AvailabilityOfEntriesOfLedger>	asyncGetListOfEntriesOfLedger(BookieId address, long ledgerId)	Makes async request for getting list of entries of ledger from a bookie and returns Future for the result.
void	asyncOpenLedger(long lId, AsyncCallback.OpenCallback cb, Object ctx)	Open a ledger as an administrator.
void	asyncOpenLedgerNoRecovery(long lId, AsyncCallback.OpenCallback cb, Object ctx)	Open a ledger as an administrator without recovering the ledger.
void	asyncRecoverBookieData(long lid, Set<BookieId> bookieSrc, boolean dryrun, boolean skipOpenLedgers, AsyncCallback.RecoverCallback callback, Object context)	Recover a specific ledger.
void	asyncRecoverBookieData(Set<BookieId> bookieSrc, boolean dryrun, boolean skipOpenLedgers, boolean skipUnrecoverableLedgers, AsyncCallback.RecoverCallback cb, Object context)
void	asyncRecoverBookieData(Set<BookieId> bookieSrc, AsyncCallback.RecoverCallback cb, Object context)
void	asyncRecoverBookieData(BookieId bookieSrc, AsyncCallback.RecoverCallback cb, Object context)	Async method to rebuild and recover the ledger fragments data that was stored on the source bookie.
void	close()	Gracefully release resources that this client uses.
void	decommissionBookie(BookieId bookieAddress)	Triggers AuditTask by resetting lostBookieRecoveryDelay and then make sure the ledgers stored in the given decommissioning bookie are properly replicated and they are not underreplicated because of the given bookie.
static boolean	format(ServerConfiguration conf, boolean isInteractive, boolean force)	Format the BookKeeper metadata in zookeeper.
(package private) static String	formatEnsemble(List<BookieId> ensemble, Set<BookieId> bookiesSrc, char marker)
Collection<BookieId>	getAllBookies()	Get a list of all bookies including the not available ones.
Collection<BookieId>	getAvailableBookies()	Get a list of the available bookies.
BookieAddressResolver	getBookieAddressResolver()
BookieServiceInfo	getBookieServiceInfo(BookieId bookiedId)
ClientConfiguration	getConf()
BookieId	getCurrentAuditor()
LedgerMetadata	getLedgerMetadata(LedgerHandle lh)
SortedMap<Long,LedgerMetadata>	getLedgersContainBookies(Set<BookieId> bookies)
int	getLostBookieRecoveryDelay()	Returns the current LostBookieRecoveryDelay value (in seconds) in Zookeeper.
Collection<BookieId>	getReadOnlyBookies()	Get a list of readonly bookies synchronously.
static boolean	initBookie(ServerConfiguration conf)	Initializes bookie, by making sure that the journalDir, ledgerDirs and indexDirs are empty and there is no registered Bookie with this BookieId.
static boolean	initNewCluster(ServerConfiguration conf)	Initializes new cluster by creating required znodes for the cluster.
EnsemblePlacementPolicy.PlacementPolicyAdherence	isEnsembleAdheringToPlacementPolicy(List<BookieId> ensembleBookiesList, int writeQuorumSize, int ackQuorumSize)	returns boolean value specifying if the ensemble of the segment is adhering to the ensemble placement policy for the given writeQuorumSize and ackQuorumSize.
Iterable<Long>	listLedgers()	This method returns an iterable object for the list of ledger identifiers of the ledgers currently available.
static BookKeeperAdmin	newBookKeeperAdmin(ClientConfiguration conf)
static boolean	nukeExistingCluster(ServerConfiguration conf, String ledgersRootPath, String instanceId, boolean force)	Nukes existing cluster metadata.
LedgerHandle	openLedger(long lId)	Open a ledger as an administrator.
LedgerHandle	openLedgerNoRecovery(long lId)	Open a ledger as an administrator without recovering the ledger.
Iterable<LedgerEntry>	readEntries(long ledgerId, long firstEntry, long lastEntry)	Read entries from a ledger synchronously.
void	recoverBookieData(long lid, Set<BookieId> bookiesSrc, boolean dryrun, boolean skipOpenLedgers)
void	recoverBookieData(Set<BookieId> bookiesSrc)
void	recoverBookieData(Set<BookieId> bookiesSrc, boolean dryrun, boolean skipOpenLedgers)
void	recoverBookieData(Set<BookieId> bookiesSrc, boolean dryrun, boolean skipOpenLedgers, boolean skipUnrecoverableLedgers)
void	recoverBookieData(BookieId bookieSrc)	Synchronous method to rebuild and recover the ledger fragments data that was stored on the source bookie.
Map<Integer,BookieId>	replaceNotAdheringPlacementPolicyBookie(List<BookieId> ensembleBookiesList, int writeQuorumSize, int ackQuorumSize)
void	replicateLedgerFragment(LedgerHandle lh, LedgerFragment ledgerFragment, BiConsumer<Long,Long> onReadEntryFailureCallback)	Replicate the Ledger fragment to target Bookie passed.
void	setLostBookieRecoveryDelay(int lostBookieRecoveryDelay)	Setter for LostBookieRecoveryDelay value (in seconds) in Zookeeper.
void	triggerAudit()	Trigger AuditTask by resetting lostBookieRecoveryDelay to its current value.
void	watchReadOnlyBookiesChanged(RegistrationClient.RegistrationListener listener)	Notify when the available list of read only bookies changes.
void	watchWritableBookiesChanged(RegistrationClient.RegistrationListener listener)	Notify when the available list of bookies changes.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

BookKeeperAdmin

public BookKeeperAdmin(String zkServers)
                throws IOException,
InterruptedException,
BKException

Constructor that takes in a ZooKeeper servers connect string so we know how to connect to ZooKeeper to retrieve information about the BookKeeper cluster. We need this before we can do any type of admin operations on the BookKeeper cluster.

Parameters:

zkServers - Comma separated list of hostname:port pairs for the ZooKeeper servers cluster.

Throws:

IOException - throws this exception if there is an error instantiating the ZooKeeper client.

InterruptedException - Throws this exception if there is an error instantiating the BookKeeper client.

BKException - Throws this exception if there is an error instantiating the BookKeeper client.

BookKeeperAdmin

public BookKeeperAdmin(ClientConfiguration conf)
                throws IOException,
InterruptedException,
BKException

Constructor that takes in a configuration object so we know how to connect to ZooKeeper to retrieve information about the BookKeeper cluster. We need this before we can do any type of admin operations on the BookKeeper cluster.

Parameters:

conf - Client Configuration Object

Throws:

IOException - throws this exception if there is an error instantiating the ZooKeeper client.

InterruptedException - Throws this exception if there is an error instantiating the BookKeeper client.

BKException - Throws this exception if there is an error instantiating the BookKeeper client.

BookKeeperAdmin

public BookKeeperAdmin(BookKeeper bkc,
 StatsLogger statsLogger,
 ClientConfiguration conf)

Constructor that takes in a BookKeeper instance . This will be useful, when user already has bk instance ready.

Parameters:

bkc - - bookkeeper instance

statsLogger - - stats logger

BookKeeperAdmin

public BookKeeperAdmin(BookKeeper bkc,
 ClientConfiguration conf)

BookKeeperAdmin

public BookKeeperAdmin(BookKeeper bkc)

Method Details

newBookKeeperAdmin

public static BookKeeperAdmin newBookKeeperAdmin(ClientConfiguration conf)
                                          throws IOException,
InterruptedException,
BKException

Throws:

IOException

InterruptedException

BKException

getConf

public ClientConfiguration getConf()

close

public void close()
           throws InterruptedException,
BKException

Gracefully release resources that this client uses.

Specified by:

close in interface AutoCloseable

Throws:

InterruptedException - if there is an error shutting down the clients that this class uses.

BKException

getAvailableBookies

public Collection<BookieId> getAvailableBookies()
                                         throws BKException

Get a list of the available bookies.

Returns:

a collection of bookie addresses

Throws:

BKException

getAllBookies

public Collection<BookieId> getAllBookies()
                                   throws BKException

Get a list of all bookies including the not available ones.

Returns:

a collection of bookie addresses

Throws:

BKException

getBookieAddressResolver

public BookieAddressResolver getBookieAddressResolver()

getBookieServiceInfo

public BookieServiceInfo getBookieServiceInfo(BookieId bookiedId)
                                       throws BKException

Throws:

BKException

getReadOnlyBookies

public Collection<BookieId> getReadOnlyBookies()
                                        throws BKException

Get a list of readonly bookies synchronously.

Returns:

a collection of bookie addresses

Throws:

BKException - if there are issues trying to read the list.

watchWritableBookiesChanged

public void watchWritableBookiesChanged(RegistrationClient.RegistrationListener listener)
                                 throws BKException

Notify when the available list of bookies changes. Once registered, the listener will be notified when the list of available bookies changes.

Parameters:

listener - the listener to notify

Throws:

BKException

watchReadOnlyBookiesChanged

public void watchReadOnlyBookiesChanged(RegistrationClient.RegistrationListener listener)
                                 throws BKException

Notify when the available list of read only bookies changes. Once registered, the listener will be notified when the list of available bookies changes.

Parameters:

listener - the listener to notify

Throws:

BKException

asyncOpenLedger

public void asyncOpenLedger(long lId,
 AsyncCallback.OpenCallback cb,
 Object ctx)

Open a ledger as an administrator. This means that no digest password checks are done. Otherwise, the call is identical to BookKeeper#asyncOpenLedger

Parameters:

lId - ledger identifier

cb - Callback which will receive a LedgerHandle object

ctx - optional context object, to be passwd to the callback (can be null)

See Also:

• BookKeeper.asyncOpenLedger(long, org.apache.bookkeeper.client.BookKeeper.DigestType, byte[], org.apache.bookkeeper.client.AsyncCallback.OpenCallback, java.lang.Object)

openLedger

public LedgerHandle openLedger(long lId)
                        throws InterruptedException,
BKException

Open a ledger as an administrator. This means that no digest password checks are done. Otherwise, the call is identical to BookKeeper#openLedger

Parameters:

lId - - ledger identifier

Throws:

InterruptedException

BKException

See Also:

• BookKeeper.openLedger(long, org.apache.bookkeeper.client.BookKeeper.DigestType, byte[])

asyncOpenLedgerNoRecovery

public void asyncOpenLedgerNoRecovery(long lId,
 AsyncCallback.OpenCallback cb,
 Object ctx)

Open a ledger as an administrator without recovering the ledger. This means that no digest password checks are done. Otherwise, the call is identical to BookKeeper#asyncOpenLedgerNoRecovery

Parameters:

lId - ledger identifier

cb - Callback which will receive a LedgerHandle object

ctx - optional context object, to be passwd to the callback (can be null)

See Also:

• BookKeeper.asyncOpenLedgerNoRecovery(long, org.apache.bookkeeper.client.BookKeeper.DigestType, byte[], org.apache.bookkeeper.client.AsyncCallback.OpenCallback, java.lang.Object)

openLedgerNoRecovery

public LedgerHandle openLedgerNoRecovery(long lId)
                                  throws InterruptedException,
BKException

Open a ledger as an administrator without recovering the ledger. This means that no digest password checks are done. Otherwise, the call is identical to BookKeeper#openLedgerNoRecovery

Parameters:

lId - ledger identifier

Throws:

InterruptedException

BKException

See Also:

• BookKeeper.openLedgerNoRecovery(long, org.apache.bookkeeper.client.BookKeeper.DigestType, byte[])

readEntries

public Iterable<LedgerEntry> readEntries(long ledgerId,
 long firstEntry,
 long lastEntry)
                                  throws InterruptedException,
BKException

Read entries from a ledger synchronously. If the lastEntry is -1, it will read all the entries in the ledger from the firstEntry.

Parameters:

ledgerId -

firstEntry -

lastEntry -

Returns:

Throws:

InterruptedException

BKException

getLedgersContainBookies

public SortedMap<Long,LedgerMetadata> getLedgersContainBookies(Set<BookieId> bookies)
                                                        throws InterruptedException,
BKException

Throws:

InterruptedException

BKException

asyncGetLedgersContainBookies

public void asyncGetLedgersContainBookies(Set<BookieId> bookies,
 BookkeeperInternalCallbacks.GenericCallback<SortedMap<Long,LedgerMetadata>> callback)

recoverBookieData

public void recoverBookieData(BookieId bookieSrc)
                       throws InterruptedException,
BKException

Synchronous method to rebuild and recover the ledger fragments data that was stored on the source bookie. That bookie could have failed completely and now the ledger data that was stored on it is under replicated. An optional destination bookie server could be given if we want to copy all of the ledger fragments data on the failed source bookie to it. Otherwise, we will just randomly distribute the ledger fragments to the active set of bookies, perhaps based on load. All ZooKeeper ledger metadata will be updated to point to the new bookie(s) that contain the replicated ledger fragments.

Parameters:

bookieSrc - Source bookie that had a failure. We want to replicate the ledger fragments that were stored there.

Throws:

InterruptedException

BKException

recoverBookieData

public void recoverBookieData(Set<BookieId> bookiesSrc)
                       throws InterruptedException,
BKException

Throws:

InterruptedException

BKException

recoverBookieData

public void recoverBookieData(Set<BookieId> bookiesSrc,
 boolean dryrun,
 boolean skipOpenLedgers)
                       throws InterruptedException,
BKException

Throws:

InterruptedException

BKException

recoverBookieData

public void recoverBookieData(Set<BookieId> bookiesSrc,
 boolean dryrun,
 boolean skipOpenLedgers,
 boolean skipUnrecoverableLedgers)
                       throws InterruptedException,
BKException

Throws:

InterruptedException

BKException

recoverBookieData

public void recoverBookieData(long lid,
 Set<BookieId> bookiesSrc,
 boolean dryrun,
 boolean skipOpenLedgers)
                       throws InterruptedException,
BKException

Throws:

InterruptedException

BKException

asyncRecoverBookieData

public void asyncRecoverBookieData(BookieId bookieSrc,
 AsyncCallback.RecoverCallback cb,
 Object context)

Async method to rebuild and recover the ledger fragments data that was stored on the source bookie. That bookie could have failed completely and now the ledger data that was stored on it is under replicated. An optional destination bookie server could be given if we want to copy all of the ledger fragments data on the failed source bookie to it. Otherwise, we will just randomly distribute the ledger fragments to the active set of bookies, perhaps based on load. All ZooKeeper ledger metadata will be updated to point to the new bookie(s) that contain the replicated ledger fragments.

Parameters:

bookieSrc - Source bookie that had a failure. We want to replicate the ledger fragments that were stored there.

cb - RecoverCallback to invoke once all of the data on the dead bookie has been recovered and replicated.

context - Context for the RecoverCallback to call.

asyncRecoverBookieData

public void asyncRecoverBookieData(Set<BookieId> bookieSrc,
 AsyncCallback.RecoverCallback cb,
 Object context)

asyncRecoverBookieData

public void asyncRecoverBookieData(Set<BookieId> bookieSrc,
 boolean dryrun,
 boolean skipOpenLedgers,
 boolean skipUnrecoverableLedgers,
 AsyncCallback.RecoverCallback cb,
 Object context)

asyncRecoverBookieData

public void asyncRecoverBookieData(long lid,
 Set<BookieId> bookieSrc,
 boolean dryrun,
 boolean skipOpenLedgers,
 AsyncCallback.RecoverCallback callback,
 Object context)

Recover a specific ledger.

Parameters:

lid - ledger to recover

bookieSrc - Source bookies that had a failure. We want to replicate the ledger fragments that were stored there.

dryrun - dryrun the recover procedure.

skipOpenLedgers - Skip recovering open ledgers.

callback - RecoverCallback to invoke once all of the data on the dead bookie has been recovered and replicated.

context - Context for the RecoverCallback to call.

formatEnsemble

static String formatEnsemble(List<BookieId> ensemble,
 Set<BookieId> bookiesSrc,
 char marker)

replicateLedgerFragment

public void replicateLedgerFragment(LedgerHandle lh,
 LedgerFragment ledgerFragment,
 BiConsumer<Long,Long> onReadEntryFailureCallback)
                             throws InterruptedException,
BKException

Replicate the Ledger fragment to target Bookie passed.

Parameters:

lh - - ledgerHandle

ledgerFragment - - LedgerFragment to replicate

Throws:

InterruptedException

BKException

format

public static boolean format(ServerConfiguration conf,
 boolean isInteractive,
 boolean force)
                      throws Exception

Format the BookKeeper metadata in zookeeper.

Parameters:

isInteractive - Whether format should ask prompt for confirmation if old data exists or not.

force - If non interactive and force is true, then old data will be removed without prompt.

Returns:

Returns true if format succeeds else false.

Throws:

Exception

initNewCluster

public static boolean initNewCluster(ServerConfiguration conf)
                              throws Exception

Initializes new cluster by creating required znodes for the cluster. If ledgersrootpath is already existing then it will error out.

Parameters:

conf -

Returns:

Throws:

Exception

nukeExistingCluster

public static boolean nukeExistingCluster(ServerConfiguration conf,
 String ledgersRootPath,
 String instanceId,
 boolean force)
                                   throws Exception

Nukes existing cluster metadata. But it does only if the provided ledgersRootPath matches with configuration's zkLedgersRootPath and provided instanceid matches with the cluster metadata. If force is mentioned then instanceid will not be validated.

Parameters:

conf -

ledgersRootPath -

instanceId -

force -

Returns:

Throws:

Exception

initBookie

public static boolean initBookie(ServerConfiguration conf)
                          throws Exception

Initializes bookie, by making sure that the journalDir, ledgerDirs and indexDirs are empty and there is no registered Bookie with this BookieId.

Parameters:

conf -

Returns:

Throws:

Exception

listLedgers

public Iterable<Long> listLedgers()
                           throws IOException

This method returns an iterable object for the list of ledger identifiers of the ledgers currently available.

Returns:

an iterable object for the list of ledger identifiers

Throws:

IOException - if the list of ledger identifiers cannot be read from the metadata store

getLedgerMetadata

public LedgerMetadata getLedgerMetadata(LedgerHandle lh)

Returns:

the metadata for the passed ledger handle

setLostBookieRecoveryDelay

public void setLostBookieRecoveryDelay(int lostBookieRecoveryDelay)
                                throws ReplicationException.CompatibilityException,
org.apache.zookeeper.KeeperException,
InterruptedException,
ReplicationException.UnavailableException

Setter for LostBookieRecoveryDelay value (in seconds) in Zookeeper.

Parameters:

lostBookieRecoveryDelay - lostBookieRecoveryDelay value (in seconds) to set

Throws:

ReplicationException.CompatibilityException

org.apache.zookeeper.KeeperException

InterruptedException

ReplicationException.UnavailableException

getLostBookieRecoveryDelay

public int getLostBookieRecoveryDelay()
                               throws ReplicationException.CompatibilityException,
org.apache.zookeeper.KeeperException,
InterruptedException,
ReplicationException.UnavailableException

Returns the current LostBookieRecoveryDelay value (in seconds) in Zookeeper.

Returns:

current lostBookieRecoveryDelay value (in seconds)

Throws:

ReplicationException.CompatibilityException

org.apache.zookeeper.KeeperException

InterruptedException

ReplicationException.UnavailableException

triggerAudit

public void triggerAudit()
                  throws ReplicationException.CompatibilityException,
org.apache.zookeeper.KeeperException,
InterruptedException,
ReplicationException.UnavailableException,
IOException

Trigger AuditTask by resetting lostBookieRecoveryDelay to its current value. If Autorecovery is not enabled or if there is no Auditor then this method will throw UnavailableException.

Throws:

ReplicationException.CompatibilityException

org.apache.zookeeper.KeeperException

InterruptedException

ReplicationException.UnavailableException

IOException

decommissionBookie

public void decommissionBookie(BookieId bookieAddress)
                        throws ReplicationException.CompatibilityException,
ReplicationException.UnavailableException,
org.apache.zookeeper.KeeperException,
InterruptedException,
IOException,
ReplicationException.BKAuditException,
TimeoutException,
BKException

Triggers AuditTask by resetting lostBookieRecoveryDelay and then make sure the ledgers stored in the given decommissioning bookie are properly replicated and they are not underreplicated because of the given bookie. This method waits until there are no underreplicatedledgers because of this bookie. If the given Bookie is not shutdown yet, then it will throw BKIllegalOpException.

Parameters:

bookieAddress - address of the decommissioning bookie

Throws:

ReplicationException.CompatibilityException

ReplicationException.UnavailableException

org.apache.zookeeper.KeeperException

InterruptedException

IOException

ReplicationException.BKAuditException

TimeoutException

BKException

areEntriesOfLedgerStoredInTheBookie

public static boolean areEntriesOfLedgerStoredInTheBookie(long ledgerId,
 BookieId bookieAddress,
 LedgerManager ledgerManager)

areEntriesOfLedgerStoredInTheBookie

public static boolean areEntriesOfLedgerStoredInTheBookie(long ledgerId,
 BookieId bookieAddress,
 LedgerMetadata ledgerMetadata)

isEnsembleAdheringToPlacementPolicy

public EnsemblePlacementPolicy.PlacementPolicyAdherence isEnsembleAdheringToPlacementPolicy(List<BookieId> ensembleBookiesList,
 int writeQuorumSize,
 int ackQuorumSize)

returns boolean value specifying if the ensemble of the segment is adhering to the ensemble placement policy for the given writeQuorumSize and ackQuorumSize.

Parameters:

ensembleBookiesList - ensemble of the segment

writeQuorumSize - writeQuorumSize of the ledger

ackQuorumSize - ackQuorumSize of the ledger

Returns:

true if the ledger is adhering to EnsemblePlacementPolicy

replaceNotAdheringPlacementPolicyBookie

public Map<Integer,BookieId> replaceNotAdheringPlacementPolicyBookie(List<BookieId> ensembleBookiesList,
 int writeQuorumSize,
 int ackQuorumSize)

asyncGetListOfEntriesOfLedger

public CompletableFuture<AvailabilityOfEntriesOfLedger> asyncGetListOfEntriesOfLedger(BookieId address,
 long ledgerId)

Makes async request for getting list of entries of ledger from a bookie and returns Future for the result.

Parameters:

address - BookieId of the bookie

ledgerId - ledgerId

Returns:

returns Future

getCurrentAuditor

public BookieId getCurrentAuditor()
                           throws IOException,
InterruptedException

Throws:

IOException

InterruptedException