Package org.apache.bookkeeper.client

Class BookKeeper

  • All Implemented Interfaces:
    java.lang.AutoCloseable, BookKeeper
    public class BookKeeper
extends java.lang.Object
implements BookKeeper
    BookKeeper client.

    We assume there is one single writer to a ledger at any time.

    There are four possible operations: start a new ledger, write to a ledger, read from a ledger and delete a ledger.

    The exceptions resulting from synchronous calls and error code resulting from asynchronous calls can be found in the class BKException.

    • Constructor Detail

      • BookKeeper

        public BookKeeper​(java.lang.String servers)
           throws java.io.IOException,
                  java.lang.InterruptedException,
                  BKException
        Create a bookkeeper client. A zookeeper client and a client event loop group will be instantiated as part of this constructor.
        Parameters:
        servers - A list of one of more servers on which zookeeper is running. The client assumes that the running bookies have been registered with zookeeper under the path AbstractConfiguration.getZkAvailableBookiesPath()
        Throws:
        java.io.IOException
        java.lang.InterruptedException
        BKException

      • BookKeeper

        public BookKeeper​(ClientConfiguration conf)
           throws java.io.IOException,
                  java.lang.InterruptedException,
                  BKException
        Create a bookkeeper client using a configuration object. A zookeeper client and a client event loop group will be instantiated as part of this constructor.
        Parameters:
        conf - Client Configuration object
        Throws:
        java.io.IOException
        java.lang.InterruptedException
        BKException

      • BookKeeper

        public BookKeeper​(ClientConfiguration conf,
                  org.apache.zookeeper.ZooKeeper zk)
           throws java.io.IOException,
                  java.lang.InterruptedException,
                  BKException
        Create a bookkeeper client but use the passed in zookeeper client instead of instantiating one.
        Parameters:
        conf - Client Configuration object ClientConfiguration
        zk - Zookeeper client instance connected to the zookeeper with which the bookies have registered
        Throws:
        java.io.IOException
        java.lang.InterruptedException
        BKException

      • BookKeeper

        public BookKeeper​(ClientConfiguration conf,
                  org.apache.zookeeper.ZooKeeper zk,
                  io.netty.channel.EventLoopGroup eventLoopGroup)
           throws java.io.IOException,
                  java.lang.InterruptedException,
                  BKException
        Create a bookkeeper client but use the passed in zookeeper client and client event loop group instead of instantiating those.
        Parameters:
        conf - Client Configuration Object ClientConfiguration
        zk - Zookeeper client instance connected to the zookeeper with which the bookies have registered. The ZooKeeper client must be connected before it is passed to BookKeeper. Otherwise a KeeperException is thrown.
        eventLoopGroup - An event loop group that will be used to create connections to the bookies
        Throws:
        java.io.IOException
        java.lang.InterruptedException
        BKException - in the event of a bookkeeper connection error

      • BookKeeper

        BookKeeper​(ClientConfiguration conf,
           org.apache.zookeeper.ZooKeeper zkc,
           io.netty.channel.EventLoopGroup eventLoopGroup,
           io.netty.buffer.ByteBufAllocator byteBufAllocator,
           StatsLogger rootStatsLogger,
           DNSToSwitchMapping dnsResolver,
           io.netty.util.HashedWheelTimer requestTimer,
           FeatureProvider featureProvider)
    throws java.io.IOException,
           java.lang.InterruptedException,
           BKException
        Constructor for use with the builder. Other constructors also use it.
        Throws:
        java.io.IOException
        java.lang.InterruptedException
        BKException

      • BookKeeper

        BookKeeper()
        Allow to extend BookKeeper for mocking in unit tests.

    • Method Detail

      • getReturnRc

        int getReturnRc​(int rc)

      • getReturnRc

        static int getReturnRc​(BookieClient bookieClient,
                       int rc)

      • scheduleBookieHealthCheckIfEnabled

        void scheduleBookieHealthCheckIfEnabled​(ClientConfiguration conf)

      • checkForFaultyBookies

        void checkForFaultyBookies()

      • getLedgerManager

        public LedgerManager getLedgerManager()
        Returns ref to speculative read counter, needed in PendingReadOp.

      • getUnderlyingLedgerManager

        LedgerManager getUnderlyingLedgerManager()

      • getCloseLock

        java.util.concurrent.locks.ReentrantReadWriteLock getCloseLock()

      • isClosed

        boolean isClosed()

      • getBookieClient

        BookieClient getBookieClient()
        Get the BookieClient, currently used for doing bookie recovery.
        Returns:
        BookieClient for the BookKeeper instance.

      • getBookieInfo

        public java.util.Map<BookieId,​BookieInfoReader.BookieInfo> getBookieInfo()
                                                                        throws BKException,
                                                                               java.lang.InterruptedException
        Retrieves BookieInfo from all the bookies in the cluster. It sends requests to all the bookies in parallel and returns the info from the bookies that responded. If there was an error in reading from any bookie, nothing will be returned for that bookie in the map.
        Returns:
        map A map of bookieSocketAddress to its BookiInfo
        Throws:
        BKException
        java.lang.InterruptedException

      • asyncCreateLedger

        public void asyncCreateLedger​(int ensSize,
                              int writeQuorumSize,
                              BookKeeper.DigestType digestType,
                              byte[] passwd,
                              AsyncCallback.CreateCallback cb,
                              java.lang.Object ctx)
        Creates a new ledger asynchronously. To create a ledger, we need to specify the ensemble size, the quorum size, the digest type, a password, a callback implementation, and an optional control object. The ensemble size is how many bookies the entries should be striped among and the quorum size is the degree of replication of each entry. The digest type is either a MAC or a CRC. Note that the CRC option is not able to protect a client against a bookie that replaces an entry. The password is used not only to authenticate access to a ledger, but also to verify entries in ledgers.
        Parameters:
        ensSize - number of bookies over which to stripe entries
        writeQuorumSize - number of bookies each entry will be written to. each of these bookies must acknowledge the entry before the call is completed.
        digestType - digest type, either MAC or CRC32
        passwd - password
        cb - createCallback implementation
        ctx - optional control object

      • asyncCreateLedger

        public void asyncCreateLedger​(int ensSize,
                              int writeQuorumSize,
                              int ackQuorumSize,
                              BookKeeper.DigestType digestType,
                              byte[] passwd,
                              AsyncCallback.CreateCallback cb,
                              java.lang.Object ctx,
                              java.util.Map<java.lang.String,​byte[]> customMetadata)
        Creates a new ledger asynchronously. Ledgers created with this call have a separate write quorum and ack quorum size. The write quorum must be larger than the ack quorum.

        Separating the write and the ack quorum allows the BookKeeper client to continue writing when a bookie has failed but the failure has not yet been detected. Detecting a bookie has failed can take a number of seconds, as configured by the read timeout ClientConfiguration.getReadTimeout(). Once the bookie failure is detected, that bookie will be removed from the ensemble.

        The other parameters match those of asyncCreateLedger(int, int, DigestType, byte[], AsyncCallback.CreateCallback, Object)

        Parameters:
        ensSize - number of bookies over which to stripe entries
        writeQuorumSize - number of bookies each entry will be written to
        ackQuorumSize - number of bookies which must acknowledge an entry before the call is completed
        digestType - digest type, either MAC or CRC32
        passwd - password
        cb - createCallback implementation
        ctx - optional control object
        customMetadata - optional customMetadata that holds user specified metadata

      • createLedger

        public LedgerHandle createLedger​(BookKeeper.DigestType digestType,
                                 byte[] passwd)
                          throws BKException,
                                 java.lang.InterruptedException
        Creates a new ledger. Default of 3 servers, and quorum of 2 servers.
        Parameters:
        digestType - digest type, either MAC or CRC32
        passwd - password
        Returns:
        a handle to the newly created ledger
        Throws:
        java.lang.InterruptedException
        BKException

      • createLedger

        public LedgerHandle createLedger​(int ensSize,
                                 int writeQuorumSize,
                                 int ackQuorumSize,
                                 BookKeeper.DigestType digestType,
                                 byte[] passwd,
                                 java.util.Map<java.lang.String,​byte[]> customMetadata)
                          throws java.lang.InterruptedException,
                                 BKException
        Synchronous call to create ledger. Parameters match those of asyncCreateLedger
        Parameters:
        ensSize -
        writeQuorumSize -
        ackQuorumSize -
        digestType -
        passwd -
        customMetadata -
        Returns:
        a handle to the newly created ledger
        Throws:
        java.lang.InterruptedException
        BKException

      • createLedgerAdv

        public LedgerHandle createLedgerAdv​(int ensSize,
                                    int writeQuorumSize,
                                    int ackQuorumSize,
                                    BookKeeper.DigestType digestType,
                                    byte[] passwd)
                             throws java.lang.InterruptedException,
                                    BKException
        Synchronous call to create ledger. Creates a new ledger asynchronously and returns LedgerHandleAdv which can accept entryId. Parameters must match those of asyncCreateLedgerAdv
        Parameters:
        ensSize -
        writeQuorumSize -
        ackQuorumSize -
        digestType -
        passwd -
        Returns:
        a handle to the newly created ledger
        Throws:
        java.lang.InterruptedException
        BKException

      • createLedgerAdv

        public LedgerHandle createLedgerAdv​(int ensSize,
                                    int writeQuorumSize,
                                    int ackQuorumSize,
                                    BookKeeper.DigestType digestType,
                                    byte[] passwd,
                                    java.util.Map<java.lang.String,​byte[]> customMetadata)
                             throws java.lang.InterruptedException,
                                    BKException
        Synchronous call to create ledger. Creates a new ledger asynchronously and returns LedgerHandleAdv which can accept entryId. Parameters must match those of asyncCreateLedgerAdv
        Parameters:
        ensSize -
        writeQuorumSize -
        ackQuorumSize -
        digestType -
        passwd -
        customMetadata -
        Returns:
        a handle to the newly created ledger
        Throws:
        java.lang.InterruptedException
        BKException

      • asyncCreateLedgerAdv

        public void asyncCreateLedgerAdv​(int ensSize,
                                 int writeQuorumSize,
                                 int ackQuorumSize,
                                 BookKeeper.DigestType digestType,
                                 byte[] passwd,
                                 AsyncCallback.CreateCallback cb,
                                 java.lang.Object ctx,
                                 java.util.Map<java.lang.String,​byte[]> customMetadata)
        Creates a new ledger asynchronously and returns LedgerHandleAdv which can accept entryId. Ledgers created with this call have ability to accept a separate write quorum and ack quorum size. The write quorum must be larger than the ack quorum.

        Separating the write and the ack quorum allows the BookKeeper client to continue writing when a bookie has failed but the failure has not yet been detected. Detecting a bookie has failed can take a number of seconds, as configured by the read timeout ClientConfiguration.getReadTimeout(). Once the bookie failure is detected, that bookie will be removed from the ensemble.

        The other parameters match those of asyncCreateLedger(int, int, DigestType, byte[], AsyncCallback.CreateCallback, Object)

        Parameters:
        ensSize - number of bookies over which to stripe entries
        writeQuorumSize - number of bookies each entry will be written to
        ackQuorumSize - number of bookies which must acknowledge an entry before the call is completed
        digestType - digest type, either MAC or CRC32
        passwd - password
        cb - createCallback implementation
        ctx - optional control object
        customMetadata - optional customMetadata that holds user specified metadata

      • createLedgerAdv

        public LedgerHandle createLedgerAdv​(long ledgerId,
                                    int ensSize,
                                    int writeQuorumSize,
                                    int ackQuorumSize,
                                    BookKeeper.DigestType digestType,
                                    byte[] passwd,
                                    java.util.Map<java.lang.String,​byte[]> customMetadata)
                             throws java.lang.InterruptedException,
                                    BKException
        Synchronously creates a new ledger using the interface which accepts a ledgerId as input. This method returns LedgerHandleAdv which can accept entryId. Parameters must match those of asyncCreateLedgerAdvWithLedgerId
        Parameters:
        ledgerId -
        ensSize -
        writeQuorumSize -
        ackQuorumSize -
        digestType -
        passwd -
        customMetadata -
        Returns:
        a handle to the newly created ledger
        Throws:
        java.lang.InterruptedException
        BKException

      • asyncCreateLedgerAdv

        public void asyncCreateLedgerAdv​(long ledgerId,
                                 int ensSize,
                                 int writeQuorumSize,
                                 int ackQuorumSize,
                                 BookKeeper.DigestType digestType,
                                 byte[] passwd,
                                 AsyncCallback.CreateCallback cb,
                                 java.lang.Object ctx,
                                 java.util.Map<java.lang.String,​byte[]> customMetadata)
        Asynchronously creates a new ledger using the interface which accepts a ledgerId as input. This method returns LedgerHandleAdv which can accept entryId. Ledgers created with this call have ability to accept a separate write quorum and ack quorum size. The write quorum must be larger than the ack quorum.

        Separating the write and the ack quorum allows the BookKeeper client to continue writing when a bookie has failed but the failure has not yet been detected. Detecting a bookie has failed can take a number of seconds, as configured by the read timeout ClientConfiguration.getReadTimeout(). Once the bookie failure is detected, that bookie will be removed from the ensemble.

        The other parameters match those of asyncCreateLedger

        Parameters:
        ledgerId - ledger Id to use for the newly created ledger
        ensSize - number of bookies over which to stripe entries
        writeQuorumSize - number of bookies each entry will be written to
        ackQuorumSize - number of bookies which must acknowledge an entry before the call is completed
        digestType - digest type, either MAC or CRC32
        passwd - password
        cb - createCallback implementation
        ctx - optional control object
        customMetadata - optional customMetadata that holds user specified metadata

      • asyncOpenLedger

        public void asyncOpenLedger​(long lId,
                            BookKeeper.DigestType digestType,
                            byte[] passwd,
                            AsyncCallback.OpenCallback cb,
                            java.lang.Object ctx)
        Open existing ledger asynchronously for reading.

        Opening a ledger with this method invokes fencing and recovery on the ledger if the ledger has not been closed. Fencing will block all other clients from writing to the ledger. Recovery will make sure that the ledger is closed before reading from it.

        Recovery also makes sure that any entries which reached one bookie, but not a quorum, will be replicated to a quorum of bookies. This occurs in cases were the writer of a ledger crashes after sending a write request to one bookie but before being able to send it to the rest of the bookies in the quorum.

        If the ledger is already closed, neither fencing nor recovery will be applied.

        Parameters:
        lId - ledger identifier
        digestType - digest type, either MAC or CRC32
        passwd - password
        ctx - optional control object
        See Also:
        LedgerHandle.asyncClose(org.apache.bookkeeper.client.AsyncCallback.CloseCallback, java.lang.Object)

      • asyncOpenLedgerNoRecovery

        public void asyncOpenLedgerNoRecovery​(long lId,
                                      BookKeeper.DigestType digestType,
                                      byte[] passwd,
                                      AsyncCallback.OpenCallback cb,
                                      java.lang.Object ctx)
        Open existing ledger asynchronously for reading, but it does not try to recover the ledger if it is not yet closed. The application needs to use it carefully, since the writer might have crashed and ledger will remain unsealed forever if there is no external mechanism to detect the failure of the writer and the ledger is not open in a safe manner, invoking the recovery procedure.

        Opening a ledger without recovery does not fence the ledger. As such, other clients can continue to write to the ledger.

        This method returns a read only ledger handle. It will not be possible to add entries to the ledger. Any attempt to add entries will throw an exception.

        Reads from the returned ledger will be able to read entries up until the lastConfirmedEntry at the point in time at which the ledger was opened. If an attempt is made to read beyond the ledger handle's LAC, an attempt is made to get the latest LAC from bookies or metadata, and if the entry_id of the read request is less than or equal to the new LAC, read will be allowed to proceed.

        Parameters:
        lId - ledger identifier
        digestType - digest type, either MAC or CRC32
        passwd - password
        ctx - optional control object

      • asyncDeleteLedger

        public void asyncDeleteLedger​(long lId,
                              AsyncCallback.DeleteCallback cb,
                              java.lang.Object ctx)
        Deletes a ledger asynchronously.
        Parameters:
        lId - ledger Id
        cb - deleteCallback implementation
        ctx - optional control object

      • asyncIsClosed

        public void asyncIsClosed​(long lId,
                          AsyncCallback.IsClosedCallback cb,
                          java.lang.Object ctx)
        Check asynchronously whether the ledger with identifier lId has been closed.
        Parameters:
        lId - ledger identifier
        cb - callback method

      • isClosed

        public boolean isClosed​(long lId)
                 throws BKException,
                        java.lang.InterruptedException
        Check whether the ledger with identifier lId has been closed.
        Parameters:
        lId -
        Returns:
        boolean true if ledger has been closed
        Throws:
        BKException
        java.lang.InterruptedException

      • close

        public void close()
           throws BKException,
                  java.lang.InterruptedException
        Shuts down client.
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface BookKeeper
        Throws:
        BKException
        java.lang.InterruptedException

      • newOpenLedgerOp

        public OpenBuilder newOpenLedgerOp()
        Description copied from interface: BookKeeper
        Open an existing ledger.
        Specified by:
        newOpenLedgerOp in interface BookKeeper
        Returns:
        a builder useful to create a readable handler for an existing ledger

      • getLedgerMetadata

        public java.util.concurrent.CompletableFuture<LedgerMetadata> getLedgerMetadata​(long ledgerId)
        Description copied from interface: BookKeeper
        Get ledger metadata of a given ledger id.
        Specified by:
        getLedgerMetadata in interface BookKeeper
        Parameters:
        ledgerId - id of the ledger.
        Returns:
        a CompletableFuture instance containing ledger metadata.

      • isDriverMetadataServiceAvailable

        public java.util.concurrent.CompletableFuture<java.lang.Boolean> isDriverMetadataServiceAvailable()
        Description copied from interface: BookKeeper
        Return driver metadata service is available.
        Specified by:
        isDriverMetadataServiceAvailable in interface BookKeeper
        Returns:
        the metadata service is available.