Package org.apache.bookkeeper.bookie

Class DefaultEntryLogger

  • All Implemented Interfaces:
    java.lang.AutoCloseable, EntryLogger
    Direct Known Subclasses:
    ReadOnlyDefaultEntryLogger
    public class DefaultEntryLogger
extends java.lang.Object
implements EntryLogger
    This class manages the writing of the bookkeeper entries. All the new entries are written to a common log. The LedgerCache will have pointers into files created by this class with offsets into the files to find the actual ledger entry. The entry log files created by this class are identified by a long.

    • Field Detail

      • LOGFILE_HEADER_SIZE

        static final int LOGFILE_HEADER_SIZE
        The 1K block at the head of the entry logger file that contains the fingerprint and meta-data. 
         Header is composed of:
 Fingerprint: 4 bytes "BKLO"
 Log file HeaderVersion enum: 4 bytes
 Ledger map offset: 8 bytes
 Ledgers Count: 4 bytes
        See Also:
        Constant Field Values

      • LEDGERS_MAP_OFFSET_POSITION

        static final int LEDGERS_MAP_OFFSET_POSITION
        See Also:
        Constant Field Values

      • LEDGERS_MAP_HEADER_SIZE

        static final int LEDGERS_MAP_HEADER_SIZE
        Ledgers map is composed of multiple parts that can be split into separated entries. Each of them is composed of: 
         length: (4 bytes) [0-3]
 ledger id (-1): (8 bytes) [4 - 11]
 entry id: (8 bytes) [12-19]
 num ledgers stored in current metadata entry: (4 bytes) [20 - 23]
 ledger entries: sequence of (ledgerid, size) (8 + 8 bytes each) [24..]
        See Also:
        Constant Field Values

      • LEDGERS_MAP_MAX_BATCH_SIZE

        static final int LEDGERS_MAP_MAX_BATCH_SIZE
        See Also:
        Constant Field Values

    • Method Detail

      • putInReadChannels

        public BufferedReadChannel putInReadChannels​(long logId,
                                             BufferedReadChannel bc)
        Put the logId, bc pair in the map responsible for the current thread.
        Parameters:
        logId -
        bc -

      • removeFromChannelsAndClose

        public void removeFromChannelsAndClose​(long logId)
        Remove all entries for this log file in each thread's cache.
        Parameters:
        logId -

      • getLeastUnflushedLogId

        long getLeastUnflushedLogId()

      • getFlushedLogIds

        public java.util.Set<java.lang.Long> getFlushedLogIds()
        Description copied from interface: EntryLogger
        Get the log ids for the set of logs which have been completely flushed to disk. Only log ids in this set are considered for either compaction or garbage collection.
        Specified by:
        getFlushedLogIds in interface EntryLogger

      • getPreviousAllocatedEntryLogId

        long getPreviousAllocatedEntryLogId()

      • prepareSortedLedgerStorageCheckpoint

        void prepareSortedLedgerStorageCheckpoint​(long numBytesFlushed)
                                   throws java.io.IOException
        Throws:
        java.io.IOException

      • prepareEntryMemTableFlush

        void prepareEntryMemTableFlush()

      • commitEntryMemTableFlush

        boolean commitEntryMemTableFlush()
                          throws java.io.IOException
        Throws:
        java.io.IOException

      • getEntryLoggerAllocator

        EntryLoggerAllocator getEntryLoggerAllocator()
        get EntryLoggerAllocator, Just for tests.

      • removeEntryLog

        public boolean removeEntryLog​(long entryLogId)
        Remove entry log.
        Specified by:
        removeEntryLog in interface EntryLogger
        Parameters:
        entryLogId - Entry Log File Id
        Returns:
        false if the entrylog doesn't exist.

      • clearCompactingLogId

        void clearCompactingLogId()

      • checkpoint

        void checkpoint()
         throws java.io.IOException
        Flushes all rotated log channels. After log channels are flushed, move leastUnflushedLogId ptr to current logId.
        Throws:
        java.io.IOException

      • flush

        public void flush()
           throws java.io.IOException
        Description copied from interface: EntryLogger
        Flush any outstanding writes to disk.
        Specified by:
        flush in interface EntryLogger
        Throws:
        java.io.IOException

      • addEntry

        long addEntry​(long ledger,
              java.nio.ByteBuffer entry)
       throws java.io.IOException
        Throws:
        java.io.IOException

      • addEntry

        long addEntry​(long ledger,
              io.netty.buffer.ByteBuf entry,
              boolean rollLog)
       throws java.io.IOException
        Throws:
        java.io.IOException

      • addEntry

        public long addEntry​(long ledger,
                     io.netty.buffer.ByteBuf entry)
              throws java.io.IOException
        Description copied from interface: EntryLogger
        Add an entry for ledger ```ledgerId``` to the entrylog.
        Specified by:
        addEntry in interface EntryLogger
        Parameters:
        ledger - the ledger for which the entry is being added
        entry - the contents of the entry (this method does not take ownership of the refcount)
        Returns:
        the location in the entry log of the added entry
        Throws:
        java.io.IOException

      • logIdForOffset

        static long logIdForOffset​(long offset)

      • posForOffset

        static long posForOffset​(long location)

      • readEntry

        public io.netty.buffer.ByteBuf readEntry​(long ledgerId,
                                         long entryId,
                                         long entryLocation)
                                  throws java.io.IOException,
                                         Bookie.NoEntryException
        Description copied from interface: EntryLogger
        Read an entry from an entrylog location, and verify that is matches the expected ledger and entry ID.
        Specified by:
        readEntry in interface EntryLogger
        Parameters:
        ledgerId - the ledgerID to match
        entryId - the entryID to match
        entryLocation - the location from which to read the entry
        Returns:
        the entry
        Throws:
        java.io.IOException
        Bookie.NoEntryException

      • readEntry

        public io.netty.buffer.ByteBuf readEntry​(long location)
                                  throws java.io.IOException,
                                         Bookie.NoEntryException
        Description copied from interface: EntryLogger
        Read an entry from an entrylog location.
        Specified by:
        readEntry in interface EntryLogger
        Parameters:
        location - the location from which to read the entry
        Returns:
        the entry
        Throws:
        java.io.IOException
        Bookie.NoEntryException

      • getChannelForLogId

        BufferedReadChannel getChannelForLogId​(long entryLogId)
                                throws java.io.IOException
        Throws:
        java.io.IOException

      • logExists

        public boolean logExists​(long logId)
        Whether the log file exists or not.
        Specified by:
        logExists in interface EntryLogger

      • getEntryLogsSet

        public java.util.Set<java.lang.Long> getEntryLogsSet()
                                              throws java.io.IOException
        Returns a set with the ids of all the entry log files.
        Throws:
        java.io.IOException

      • scanEntryLog

        public void scanEntryLog​(long entryLogId,
                         EntryLogScanner scanner)
                  throws java.io.IOException
        Scan entry log.
        Specified by:
        scanEntryLog in interface EntryLogger
        Parameters:
        entryLogId - Entry Log Id
        scanner - Entry Log Scanner
        Throws:
        java.io.IOException

      • getEntryLogMetadata

        public EntryLogMetadata getEntryLogMetadata​(long entryLogId,
                                            AbstractLogCompactor.Throttler throttler)
                                     throws java.io.IOException
        Description copied from interface: EntryLogger
        Retrieve metadata for the given entrylog ID. The metadata contains the size of the log, the size of the data in the log which is still active, and a list of all the ledgers contained in the log and the size of the data stored for each ledger.
        Specified by:
        getEntryLogMetadata in interface EntryLogger
        Throws:
        java.io.IOException

      • extractEntryLogMetadataFromIndex

        EntryLogMetadata extractEntryLogMetadataFromIndex​(long entryLogId)
                                           throws java.io.IOException
        Throws:
        java.io.IOException

      • close

        public void close()
        Shutdown method to gracefully stop entry logger.
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface EntryLogger

      • fileName2LogId

        static long fileName2LogId​(java.lang.String fileName)
        Convert log filename (hex format with suffix) to logId in long.

      • logId2HexString

        static java.lang.String logId2HexString​(long logId)
        Convert log Id to hex string.

      • newCompactionLog

        public CompactionEntryLog newCompactionLog​(long logToCompact)
                                    throws java.io.IOException
        Description copied from interface: EntryLogger
        Create a new entrylog into which compacted entries can be added. There is a 1-1 mapping between logs that are being compacted and the log the compacted entries are written to.
        Specified by:
        newCompactionLog in interface EntryLogger
        Throws:
        java.io.IOException

      • incompleteCompactionLogs

        public java.util.Collection<CompactionEntryLog> incompleteCompactionLogs()
        Description copied from interface: EntryLogger
        Return a collection of all the compaction entry logs which have been compacted, but have not been cleaned up.
        Specified by:
        incompleteCompactionLogs in interface EntryLogger