Package org.apache.bookkeeper.bookie

Class SortedLedgerStorage

java.lang.Object

org.apache.bookkeeper.bookie.SortedLedgerStorage

All Implemented Interfaces:

CacheCallback, CompactableLedgerStorage, DefaultEntryLogger.EntryLogListener, LedgerStorage, SkipListFlusher

public class SortedLedgerStorage
extends java.lang.Object
implements LedgerStorage, CacheCallback, SkipListFlusher, CompactableLedgerStorage, DefaultEntryLogger.EntryLogListener

A SortedLedgerStorage is an extension of InterleavedLedgerStorage. It is comprised of two MemTables and a InterleavedLedgerStorage. All the entries will be first added into a MemTable, and then be flushed back to the InterleavedLedgerStorage when the MemTable becomes full.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.bookkeeper.bookie.LedgerStorage

LedgerStorage.DetectedInconsistency, LedgerStorage.LedgerDeletionListener, LedgerStorage.StorageState

Field Summary

Fields

Modifier and Type	Field	Description
(package private) EntryMemTable	memTable

Constructor Summary

Constructors

Modifier	Constructor	Description
	SortedLedgerStorage()
protected	SortedLedgerStorage(InterleavedLedgerStorage ils)

Method Summary

Modifier and Type	Method	Description
long	addEntry(io.netty.buffer.ByteBuf entry)	Add an entry to the storage.
void	cancelWaitForLastAddConfirmedUpdate(long ledgerId, Watcher<LastAddConfirmedUpdateNotification> watcher)	Cancel a previous wait for last add confirmed update.
void	checkpoint(CheckpointSource.Checkpoint checkpoint)	Ask the ledger storage to sync data until the given checkpoint.
void	clearLimboState(long ledgerId)	Clear the limbo state of a ledger.
void	clearStorageStateFlag(LedgerStorage.StorageState flags)	Clear a storage state flag for the storage instance.
void	deleteLedger(long ledgerId)
boolean	entryExists(long ledgerId, long entryId)	Whether an entry exists.
void	flush()	Flushes all data in the storage.
void	flushEntriesLocationsIndex()	Flush the entries locations index for the compacted entries.
void	forceGC()	Force trigger Garbage Collection.
void	forceGC(boolean forceMajor, boolean forceMinor)	Force trigger Garbage Collection with forceMajor or forceMinor parameter.
java.lang.Iterable<java.lang.Long>	getActiveLedgersInRange(long firstLedgerId, long lastLedgerId)	Get an iterator over a range of ledger ids stored in the bookie.
io.netty.buffer.ByteBuf	getEntry(long ledgerId, long entryId)	Read an entry from storage.
DefaultEntryLogger	getEntryLogger()
io.netty.buffer.ByteBuf	getExplicitLac(long ledgerId)
java.util.List<GarbageCollectionStatus>	getGarbageCollectionStatus()	Get Garbage Collection status.
long	getLastAddConfirmed(long ledgerId)	Get last add confirmed.
java.util.PrimitiveIterator.OfLong	getListOfEntriesOfLedger(long ledgerId)	Returns the primitive long iterator for entries of the ledger, stored in this LedgerStorage.
(package private) java.util.concurrent.ScheduledExecutorService	getScheduler()
(package private) BookieStateManager	getStateManager()
java.util.EnumSet<LedgerStorage.StorageState>	getStorageStateFlags()	Get the storage state flags currently set for the storage instance.
LedgerStorage	getUnderlyingLedgerStorage()
boolean	hasLimboState(long ledgerId)	Check whether a ledger is in limbo state.
void	initialize(ServerConfiguration conf, LedgerManager ledgerManager, LedgerDirsManager ledgerDirsManager, LedgerDirsManager indexDirsManager, StatsLogger statsLogger, io.netty.buffer.ByteBufAllocator allocator)	Initialize the LedgerStorage implementation.
boolean	isFenced(long ledgerId)	Check whether the ledger is fenced in ledger storage or not.
boolean	isInForceGC()	Whether force triggered Garbage Collection is running or not.
boolean	isMajorGcSuspended()
boolean	isMinorGcSuspended()
boolean	ledgerExists(long ledgerId)	Whether a ledger exists.
java.util.List<LedgerStorage.DetectedInconsistency>	localConsistencyCheck(java.util.Optional<com.google.common.util.concurrent.RateLimiter> rateLimiter)	Performs internal check of local storage logging any inconsistencies.
(package private) static java.util.concurrent.ScheduledExecutorService	newScheduledExecutorService()
void	onRotateEntryLog()	Rotate a new entry log to write.
void	onSizeLimitReached(CheckpointSource.Checkpoint cp)	Process notification that cache size limit reached.
void	process(long ledgerId, long entryId, io.netty.buffer.ByteBuf buffer)	Process an entry.
byte[]	readMasterKey(long ledgerId)	Get the master key for a ledger.
void	registerLedgerDeletionListener(LedgerStorage.LedgerDeletionListener listener)	Register a listener for ledgers deletion notifications.
void	resumeMajorGC()
void	resumeMinorGC()
void	setCheckpointer(Checkpointer checkpointer)
void	setCheckpointSource(CheckpointSource checkpointSource)
void	setExplicitLac(long ledgerId, io.netty.buffer.ByteBuf lac)
boolean	setFenced(long ledgerId)	Fenced the ledger id in ledger storage.
void	setLimboState(long ledgerId)	Set a ledger to limbo state.
void	setMasterKey(long ledgerId, byte[] masterKey)	Set the master key for a ledger.
void	setStateManager(StateManager stateManager)
void	setStorageStateFlag(LedgerStorage.StorageState flags)	Set a storage state flag for the storage instance.
void	shutdown()	Cleanup and free any resources being used by the storage system.
void	start()	Start any background threads belonging to the storage system.
void	suspendMajorGC()
void	suspendMinorGC()
void	updateEntriesLocations(java.lang.Iterable<EntryLocation> locations)	Update the location of several entries.
boolean	waitForLastAddConfirmedUpdate(long ledgerId, long previousLAC, Watcher<LastAddConfirmedUpdateNotification> watcher)	Wait for last add confirmed update.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.bookkeeper.bookie.LedgerStorage

entryLocationCompact, entryLocationCompact, getEntryLocationDBPath, isEntryLocationCompacting, isEntryLocationCompacting

Field Detail

memTable

EntryMemTable memTable

Constructor Detail

SortedLedgerStorage

public SortedLedgerStorage()

SortedLedgerStorage

protected SortedLedgerStorage(InterleavedLedgerStorage ils)

Method Detail

initialize

public void initialize(ServerConfiguration conf,
                       LedgerManager ledgerManager,
                       LedgerDirsManager ledgerDirsManager,
                       LedgerDirsManager indexDirsManager,
                       StatsLogger statsLogger,
                       io.netty.buffer.ByteBufAllocator allocator)
                throws java.io.IOException

Description copied from interface: LedgerStorage

Initialize the LedgerStorage implementation.

Specified by:

initialize in interface LedgerStorage

Throws:

java.io.IOException

newScheduledExecutorService

static java.util.concurrent.ScheduledExecutorService newScheduledExecutorService()

setStateManager

public void setStateManager(StateManager stateManager)

Specified by:

setStateManager in interface LedgerStorage

setCheckpointSource

public void setCheckpointSource(CheckpointSource checkpointSource)

Specified by:

setCheckpointSource in interface LedgerStorage

setCheckpointer

public void setCheckpointer(Checkpointer checkpointer)

Specified by:

setCheckpointer in interface LedgerStorage

getScheduler

java.util.concurrent.ScheduledExecutorService getScheduler()

start

public void start()

Description copied from interface: LedgerStorage

Start any background threads belonging to the storage system. For example, garbage collection.

Specified by:

start in interface LedgerStorage

shutdown

public void shutdown()
              throws java.lang.InterruptedException

Description copied from interface: LedgerStorage

Cleanup and free any resources being used by the storage system.

Specified by:

shutdown in interface LedgerStorage

Throws:

java.lang.InterruptedException

ledgerExists

public boolean ledgerExists(long ledgerId)
                     throws java.io.IOException

Description copied from interface: LedgerStorage

Whether a ledger exists.

Specified by:

ledgerExists in interface LedgerStorage

Throws:

java.io.IOException

entryExists

public boolean entryExists(long ledgerId,
                           long entryId)
                    throws java.io.IOException

Description copied from interface: LedgerStorage

Whether an entry exists.

Specified by:

entryExists in interface LedgerStorage

Throws:

java.io.IOException

setFenced

public boolean setFenced(long ledgerId)
                  throws java.io.IOException

Description copied from interface: LedgerStorage

Fenced the ledger id in ledger storage.

Specified by:

setFenced in interface LedgerStorage

Parameters:

ledgerId - Ledger Id.

Throws:

java.io.IOException - when failed to fence the ledger.

isFenced

public boolean isFenced(long ledgerId)
                 throws java.io.IOException

Description copied from interface: LedgerStorage

Check whether the ledger is fenced in ledger storage or not.

Specified by:

isFenced in interface LedgerStorage

Parameters:

ledgerId - Ledger ID.

Throws:

java.io.IOException

setMasterKey

public void setMasterKey(long ledgerId,
                         byte[] masterKey)
                  throws java.io.IOException

Description copied from interface: LedgerStorage

Set the master key for a ledger.

Specified by:

setMasterKey in interface LedgerStorage

Throws:

java.io.IOException

readMasterKey

public byte[] readMasterKey(long ledgerId)
                     throws java.io.IOException,
                            BookieException

Description copied from interface: LedgerStorage

Get the master key for a ledger.

Specified by:

readMasterKey in interface LedgerStorage

Throws:

java.io.IOException - if there is an error reading the from the ledger

BookieException - if no such ledger exists

addEntry

public long addEntry(io.netty.buffer.ByteBuf entry)
              throws java.io.IOException

Description copied from interface: LedgerStorage

Add an entry to the storage.

Specified by:

addEntry in interface LedgerStorage

Returns:

the entry id of the entry added

Throws:

java.io.IOException

getEntry

public io.netty.buffer.ByteBuf getEntry(long ledgerId,
                                        long entryId)
                                 throws java.io.IOException,
                                        BookieException

Description copied from interface: LedgerStorage

Read an entry from storage.

Specified by:

getEntry in interface LedgerStorage

Throws:

java.io.IOException

BookieException

getLastAddConfirmed

public long getLastAddConfirmed(long ledgerId)
                         throws java.io.IOException

Description copied from interface: LedgerStorage

Get last add confirmed.

Specified by:

getLastAddConfirmed in interface LedgerStorage

Parameters:

ledgerId - ledger id.

Returns:

last add confirmed.

Throws:

java.io.IOException

waitForLastAddConfirmedUpdate

public boolean waitForLastAddConfirmedUpdate(long ledgerId,
                                             long previousLAC,
                                             Watcher<LastAddConfirmedUpdateNotification> watcher)
                                      throws java.io.IOException

Description copied from interface: LedgerStorage

Wait for last add confirmed update.

Specified by:

waitForLastAddConfirmedUpdate in interface LedgerStorage

previousLAC - - The threshold beyond which we would wait for the update

watcher - - Watcher to notify on update

Returns:

Throws:

java.io.IOException

cancelWaitForLastAddConfirmedUpdate

public void cancelWaitForLastAddConfirmedUpdate(long ledgerId,
                                                Watcher<LastAddConfirmedUpdateNotification> watcher)
                                         throws java.io.IOException

Description copied from interface: LedgerStorage

Cancel a previous wait for last add confirmed update.

Specified by:

cancelWaitForLastAddConfirmedUpdate in interface LedgerStorage

Parameters:

ledgerId - The ledger being watched.

watcher - The watcher to cancel.

Throws:

java.io.IOException

checkpoint

public void checkpoint(CheckpointSource.Checkpoint checkpoint)
                throws java.io.IOException

Description copied from interface: LedgerStorage

Ask the ledger storage to sync data until the given checkpoint. The ledger storage implementation do checkpoint and return the real checkpoint that it finished. The returned the checkpoint indicates that all entries added before that point already persist.

Specified by:

checkpoint in interface LedgerStorage

Parameters:

checkpoint - Check Point that CheckpointSource.Checkpoint proposed.

Throws:

java.io.IOException

deleteLedger

public void deleteLedger(long ledgerId)
                  throws java.io.IOException

Specified by:

deleteLedger in interface LedgerStorage

Throws:

java.io.IOException

registerLedgerDeletionListener

public void registerLedgerDeletionListener(LedgerStorage.LedgerDeletionListener listener)

Description copied from interface: LedgerStorage

Register a listener for ledgers deletion notifications.

Specified by:

registerLedgerDeletionListener in interface LedgerStorage

Parameters:

listener - object that will be notified every time a ledger is deleted

setExplicitLac

public void setExplicitLac(long ledgerId,
                           io.netty.buffer.ByteBuf lac)
                    throws java.io.IOException

Specified by:

setExplicitLac in interface LedgerStorage

Throws:

java.io.IOException

getExplicitLac

public io.netty.buffer.ByteBuf getExplicitLac(long ledgerId)

Specified by:

getExplicitLac in interface LedgerStorage

process

public void process(long ledgerId,
                    long entryId,
                    io.netty.buffer.ByteBuf buffer)
             throws java.io.IOException

Description copied from interface: SkipListFlusher

Process an entry.

Specified by:

process in interface SkipListFlusher

Parameters:

ledgerId - Ledger ID.

entryId - The entry id this entry.

buffer - Entry ByteBuffer

Throws:

java.io.IOException

flush

public void flush()
           throws java.io.IOException

Description copied from interface: LedgerStorage

Flushes all data in the storage. Once this is called, add data written to the LedgerStorage up until this point has been persisted to perminant storage

Specified by:

flush in interface LedgerStorage

Throws:

java.io.IOException

onSizeLimitReached

public void onSizeLimitReached(CheckpointSource.Checkpoint cp)
                        throws java.io.IOException

Description copied from interface: CacheCallback

Process notification that cache size limit reached.

Specified by:

onSizeLimitReached in interface CacheCallback

Throws:

java.io.IOException

onRotateEntryLog

public void onRotateEntryLog()

Description copied from interface: DefaultEntryLogger.EntryLogListener

Rotate a new entry log to write.

Specified by:

onRotateEntryLog in interface DefaultEntryLogger.EntryLogListener

getStateManager

BookieStateManager getStateManager()

getEntryLogger

public DefaultEntryLogger getEntryLogger()

getActiveLedgersInRange

public java.lang.Iterable<java.lang.Long> getActiveLedgersInRange(long firstLedgerId,
                                                                  long lastLedgerId)
                                                           throws java.io.IOException

Description copied from interface: CompactableLedgerStorage

Get an iterator over a range of ledger ids stored in the bookie.

Specified by:

getActiveLedgersInRange in interface CompactableLedgerStorage

Parameters:

firstLedgerId - first ledger id in the sequence (included)

lastLedgerId - last ledger id in the sequence (not included)

Returns:

Throws:

java.io.IOException

updateEntriesLocations

public void updateEntriesLocations(java.lang.Iterable<EntryLocation> locations)
                            throws java.io.IOException

Description copied from interface: CompactableLedgerStorage

Update the location of several entries.

Specified by:

updateEntriesLocations in interface CompactableLedgerStorage

Parameters:

locations - the list of locations to update

Throws:

java.io.IOException

flushEntriesLocationsIndex

public void flushEntriesLocationsIndex()
                                throws java.io.IOException

Description copied from interface: CompactableLedgerStorage

Flush the entries locations index for the compacted entries.

Specified by:

flushEntriesLocationsIndex in interface CompactableLedgerStorage

Throws:

java.io.IOException

getUnderlyingLedgerStorage

public LedgerStorage getUnderlyingLedgerStorage()

Specified by:

getUnderlyingLedgerStorage in interface LedgerStorage

forceGC

public void forceGC()

Description copied from interface: LedgerStorage

Force trigger Garbage Collection.

Specified by:

forceGC in interface LedgerStorage

forceGC

public void forceGC(boolean forceMajor,
                    boolean forceMinor)

Description copied from interface: LedgerStorage

Force trigger Garbage Collection with forceMajor or forceMinor parameter.

Specified by:

forceGC in interface LedgerStorage

suspendMinorGC

public void suspendMinorGC()

Specified by:

suspendMinorGC in interface LedgerStorage

suspendMajorGC

public void suspendMajorGC()

Specified by:

suspendMajorGC in interface LedgerStorage

resumeMinorGC

public void resumeMinorGC()

Specified by:

resumeMinorGC in interface LedgerStorage

resumeMajorGC

public void resumeMajorGC()

Specified by:

resumeMajorGC in interface LedgerStorage

isMajorGcSuspended

public boolean isMajorGcSuspended()

Specified by:

isMajorGcSuspended in interface LedgerStorage

isMinorGcSuspended

public boolean isMinorGcSuspended()

Specified by:

isMinorGcSuspended in interface LedgerStorage

localConsistencyCheck

public java.util.List<LedgerStorage.DetectedInconsistency> localConsistencyCheck(java.util.Optional<com.google.common.util.concurrent.RateLimiter> rateLimiter)
                                                                          throws java.io.IOException

Description copied from interface: LedgerStorage

Performs internal check of local storage logging any inconsistencies.

Specified by:

localConsistencyCheck in interface LedgerStorage

Parameters:

rateLimiter - Provide to rate of entry checking. null for unlimited.

Returns:

List of inconsistencies detected

Throws:

java.io.IOException

isInForceGC

public boolean isInForceGC()

Description copied from interface: LedgerStorage

Whether force triggered Garbage Collection is running or not.

Specified by:

isInForceGC in interface LedgerStorage

Returns:

true -- force triggered Garbage Collection is running, false -- force triggered Garbage Collection is not running

getGarbageCollectionStatus

public java.util.List<GarbageCollectionStatus> getGarbageCollectionStatus()

Description copied from interface: LedgerStorage

Get Garbage Collection status. Since DbLedgerStorage is a list of storage instances, we should return a list.

Specified by:

getGarbageCollectionStatus in interface LedgerStorage

getListOfEntriesOfLedger

public java.util.PrimitiveIterator.OfLong getListOfEntriesOfLedger(long ledgerId)
                                                            throws java.io.IOException

Description copied from interface: LedgerStorage

Returns the primitive long iterator for entries of the ledger, stored in this LedgerStorage. The returned iterator provide weakly consistent state of the ledger. It is guaranteed that entries of the ledger added to this LedgerStorage by the time this method is called will be available but modifications made after method invocation may not be available.

Specified by:

getListOfEntriesOfLedger in interface LedgerStorage

Parameters:

ledgerId - - id of the ledger

Returns:

the list of entries of the ledger available in this ledgerstorage.

Throws:

java.io.IOException

setLimboState

public void setLimboState(long ledgerId)
                   throws java.io.IOException

Description copied from interface: LedgerStorage

Set a ledger to limbo state. When a ledger is in limbo state, we cannot answer any requests about it. For example, if a client asks for an entry, we cannot say we don't have it because it may have been written to us in the past, but we are waiting for data integrity checks to copy it over.

Specified by:

setLimboState in interface LedgerStorage

Throws:

java.io.IOException

hasLimboState

public boolean hasLimboState(long ledgerId)
                      throws java.io.IOException

Description copied from interface: LedgerStorage

Check whether a ledger is in limbo state.

Specified by:

hasLimboState in interface LedgerStorage

Throws:

java.io.IOException

See Also:

LedgerStorage.setLimboState(long)

clearLimboState

public void clearLimboState(long ledgerId)
                     throws java.io.IOException

Description copied from interface: LedgerStorage

Clear the limbo state of a ledger.

Specified by:

clearLimboState in interface LedgerStorage

Throws:

java.io.IOException

See Also:

LedgerStorage.setLimboState(long)

getStorageStateFlags

public java.util.EnumSet<LedgerStorage.StorageState> getStorageStateFlags()
                                                                   throws java.io.IOException

Description copied from interface: LedgerStorage

Get the storage state flags currently set for the storage instance.

Specified by:

getStorageStateFlags in interface LedgerStorage

Throws:

java.io.IOException

setStorageStateFlag

public void setStorageStateFlag(LedgerStorage.StorageState flags)
                         throws java.io.IOException

Description copied from interface: LedgerStorage

Set a storage state flag for the storage instance. Implementations must ensure this method is atomic, and the flag is persisted to storage when the method returns.

Specified by:

setStorageStateFlag in interface LedgerStorage

Throws:

java.io.IOException

clearStorageStateFlag

public void clearStorageStateFlag(LedgerStorage.StorageState flags)
                           throws java.io.IOException

Description copied from interface: LedgerStorage

Clear a storage state flag for the storage instance. Implementations must ensure this method is atomic, and the flag is persisted to storage when the method returns.

Specified by:

clearStorageStateFlag in interface LedgerStorage

Throws:

java.io.IOException