Package org.apache.bookkeeper.bookie.storage.ldb

Class SingleDirectoryDbLedgerStorage

java.lang.Object

org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage

All Implemented Interfaces:

CompactableLedgerStorage, LedgerStorage

public class SingleDirectoryDbLedgerStorage
extends Object
implements CompactableLedgerStorage

Single directory implementation of LedgerStorage that uses RocksDB to keep the indexes for entries stored in EntryLogs.

This is meant only to be used from DbLedgerStorage.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	SingleDirectoryDbLedgerStorage.LedgerLoggerProcessor	Interface which process ledger logger.

Nested classes/interfaces inherited from interface org.apache.bookkeeper.bookie.LedgerStorage

LedgerStorage.DetectedInconsistency, LedgerStorage.LedgerDeletionListener, LedgerStorage.StorageState

Field Summary

Fields

Modifier and Type	Field	Description
protected final ReentrantLock	flushMutex
protected final AtomicBoolean	hasFlushBeenTriggered
protected WriteCache	writeCache
protected WriteCache	writeCacheBeingFlushed

Constructor Summary

Constructors

Constructor	Description
SingleDirectoryDbLedgerStorage(ServerConfiguration conf, LedgerManager ledgerManager, LedgerDirsManager ledgerDirsManager, LedgerDirsManager indexDirsManager, EntryLogger entryLogger, StatsLogger statsLogger, io.netty.buffer.ByteBufAllocator allocator, long writeCacheSize, long readCacheSize, int readAheadCacheBatchSize, long readAheadCacheBatchBytesSize)

Method Summary

Modifier and Type	Method	Description
long	addEntry(io.netty.buffer.ByteBuf entry)	Add an entry to the storage.
long	addLedgerToIndex(long ledgerId, boolean isFenced, byte[] masterKey, LedgerCache.PageEntriesIterable pages)	Add an already existing ledger to the index.
void	cancelWaitForLastAddConfirmedUpdate(long ledgerId, Watcher<LastAddConfirmedUpdateNotification> watcher)	Cancel a previous wait for last add confirmed update.
protected boolean	chargeReadAheadCache(int currentReadAheadCount, long currentReadAheadBytes)
void	checkpoint(CheckpointSource.Checkpoint checkpoint)	Ask the ledger storage to sync data until the given checkpoint.
void	clearLimboState(long ledgerId)	Clear the limbo state of a ledger.
void	clearStorageStateFlag(LedgerStorage.StorageState flag)	Clear a storage state flag for the storage instance.
void	deleteLedger(long ledgerId)
boolean	entryExists(long ledgerId, long entryId)	Whether an entry exists.
void	entryLocationCompact()
void	flush()	Flushes all data in the storage.
void	flushEntriesLocationsIndex()	Flush the entries locations index for the compacted entries.
void	forceGC()	Force trigger Garbage Collection.
void	forceGC(boolean forceMajor, boolean forceMinor)	Force trigger Garbage Collection with forceMajor or forceMinor parameter.
Iterable<Long>	getActiveLedgersInRange(long firstLedgerId, long lastLedgerId)	Get an iterator over a range of ledger ids stored in the bookie.
(package private) DbLedgerStorageStats	getDbLedgerStorageStats()
io.netty.buffer.ByteBuf	getEntry(long ledgerId, long entryId)	Read an entry from storage.
List<String>	getEntryLocationDBPath()
EntryLocationIndex	getEntryLocationIndex()
(package private) EntryLogger	getEntryLogger()
io.netty.buffer.ByteBuf	getExplicitLac(long ledgerId)
List<GarbageCollectionStatus>	getGarbageCollectionStatus()	Get Garbage Collection status.
String	getIndexBaseDir()
long	getLastAddConfirmed(long ledgerId)	Get last add confirmed.
io.netty.buffer.ByteBuf	getLastEntry(long ledgerId)
String	getLedgerBaseDir()
PrimitiveIterator.OfLong	getListOfEntriesOfLedger(long ledgerId)	Returns the primitive long iterator for entries of the ledger, stored in this LedgerStorage.
EnumSet<LedgerStorage.StorageState>	getStorageStateFlags()	Get the storage state flags currently set for the storage instance.
boolean	hasLimboState(long ledgerId)	Check whether a ledger is in limbo state.
void	initialize(ServerConfiguration conf, LedgerManager ledgerManager, LedgerDirsManager ledgerDirsManager, LedgerDirsManager indexDirsManager, StatsLogger statsLogger, io.netty.buffer.ByteBufAllocator allocator)	Initialize the LedgerStorage implementation.
boolean	isEntryLocationCompacting()
boolean	isFenced(long ledgerId)	Check whether the ledger is fenced in ledger storage or not.
(package private) boolean	isFlushRequired()
boolean	isInForceGC()	Whether force triggered Garbage Collection is running or not.
boolean	isMajorGcSuspended()
boolean	isMinorGcSuspended()
boolean	ledgerExists(long ledgerId)	Whether a ledger exists.
byte[]	readMasterKey(long ledgerId)	Get the master key for a ledger.
void	registerLedgerDeletionListener(LedgerStorage.LedgerDeletionListener listener)	Register a listener for ledgers deletion notifications.
void	resumeMajorGC()
void	resumeMinorGC()
void	setCheckpointer(Checkpointer checkpointer)
void	setCheckpointSource(CheckpointSource checkpointSource)
void	setExplicitLac(long ledgerId, io.netty.buffer.ByteBuf lac)
boolean	setFenced(long ledgerId)	Fenced the ledger id in ledger storage.
void	setLimboState(long ledgerId)	Set a ledger to limbo state.
void	setMasterKey(long ledgerId, byte[] masterKey)	Set the master key for a ledger.
void	setStateManager(StateManager stateManager)
void	setStorageStateFlag(LedgerStorage.StorageState flag)	Set a storage state flag for the storage instance.
void	shutdown()	Cleanup and free any resources being used by the storage system.
void	start()	Start any background threads belonging to the storage system.
void	suspendMajorGC()
void	suspendMinorGC()
void	updateEntriesLocations(Iterable<EntryLocation> locations)	Update the location of several entries.
boolean	waitForLastAddConfirmedUpdate(long ledgerId, long previousLAC, Watcher<LastAddConfirmedUpdateNotification> watcher)	Wait for last add confirmed update.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.bookkeeper.bookie.LedgerStorage

entryLocationCompact, getUnderlyingLedgerStorage, isEntryLocationCompacting, localConsistencyCheck

Field Details

writeCache

protected volatile WriteCache writeCache

writeCacheBeingFlushed

protected volatile WriteCache writeCacheBeingFlushed

flushMutex

protected final ReentrantLock flushMutex

hasFlushBeenTriggered

protected final AtomicBoolean hasFlushBeenTriggered

Constructor Details

SingleDirectoryDbLedgerStorage

public SingleDirectoryDbLedgerStorage(ServerConfiguration conf,
 LedgerManager ledgerManager,
 LedgerDirsManager ledgerDirsManager,
 LedgerDirsManager indexDirsManager,
 EntryLogger entryLogger,
 StatsLogger statsLogger,
 io.netty.buffer.ByteBufAllocator allocator,
 long writeCacheSize,
 long readCacheSize,
 int readAheadCacheBatchSize,
 long readAheadCacheBatchBytesSize)
                               throws IOException

Throws:

IOException

Method Details

initialize

public void initialize(ServerConfiguration conf,
 LedgerManager ledgerManager,
 LedgerDirsManager ledgerDirsManager,
 LedgerDirsManager indexDirsManager,
 StatsLogger statsLogger,
 io.netty.buffer.ByteBufAllocator allocator)
                throws IOException

Description copied from interface: LedgerStorage

Initialize the LedgerStorage implementation.

Specified by:

initialize in interface LedgerStorage

Throws:

IOException

setStateManager

public void setStateManager(StateManager stateManager)

Specified by:

setStateManager in interface LedgerStorage

setCheckpointSource

public void setCheckpointSource(CheckpointSource checkpointSource)

Specified by:

setCheckpointSource in interface LedgerStorage

setCheckpointer

public void setCheckpointer(Checkpointer checkpointer)

Specified by:

setCheckpointer in interface LedgerStorage

start

public void start()

Description copied from interface: LedgerStorage

Start any background threads belonging to the storage system. For example, garbage collection.

Specified by:

start in interface LedgerStorage

forceGC

public void forceGC()

Description copied from interface: LedgerStorage

Force trigger Garbage Collection.

Specified by:

forceGC in interface LedgerStorage

forceGC

public void forceGC(boolean forceMajor,
 boolean forceMinor)

Description copied from interface: LedgerStorage

Force trigger Garbage Collection with forceMajor or forceMinor parameter.

Specified by:

forceGC in interface LedgerStorage

isInForceGC

public boolean isInForceGC()

Description copied from interface: LedgerStorage

Whether force triggered Garbage Collection is running or not.

Specified by:

isInForceGC in interface LedgerStorage

Returns:

true -- force triggered Garbage Collection is running, false -- force triggered Garbage Collection is not running

suspendMinorGC

public void suspendMinorGC()

Specified by:

suspendMinorGC in interface LedgerStorage

suspendMajorGC

public void suspendMajorGC()

Specified by:

suspendMajorGC in interface LedgerStorage

resumeMinorGC

public void resumeMinorGC()

Specified by:

resumeMinorGC in interface LedgerStorage

resumeMajorGC

public void resumeMajorGC()

Specified by:

resumeMajorGC in interface LedgerStorage

isMajorGcSuspended

public boolean isMajorGcSuspended()

Specified by:

isMajorGcSuspended in interface LedgerStorage

isMinorGcSuspended

public boolean isMinorGcSuspended()

Specified by:

isMinorGcSuspended in interface LedgerStorage

entryLocationCompact

public void entryLocationCompact()

Specified by:

entryLocationCompact in interface LedgerStorage

isEntryLocationCompacting

public boolean isEntryLocationCompacting()

Specified by:

isEntryLocationCompacting in interface LedgerStorage

getEntryLocationDBPath

public List<String> getEntryLocationDBPath()

Specified by:

getEntryLocationDBPath in interface LedgerStorage

shutdown

public void shutdown()
              throws InterruptedException

Description copied from interface: LedgerStorage

Cleanup and free any resources being used by the storage system.

Specified by:

shutdown in interface LedgerStorage

Throws:

InterruptedException

ledgerExists

public boolean ledgerExists(long ledgerId)
                     throws IOException

Description copied from interface: LedgerStorage

Whether a ledger exists.

Specified by:

ledgerExists in interface LedgerStorage

Throws:

IOException

entryExists

public boolean entryExists(long ledgerId,
 long entryId)
                    throws IOException,
BookieException

Description copied from interface: LedgerStorage

Whether an entry exists.

Specified by:

entryExists in interface LedgerStorage

Throws:

IOException

BookieException

isFenced

public boolean isFenced(long ledgerId)
                 throws IOException,
BookieException

Description copied from interface: LedgerStorage

Check whether the ledger is fenced in ledger storage or not.

Specified by:

isFenced in interface LedgerStorage

Parameters:

ledgerId - Ledger ID.

Throws:

IOException

BookieException

setFenced

public boolean setFenced(long ledgerId)
                  throws IOException

Description copied from interface: LedgerStorage

Fenced the ledger id in ledger storage.

Specified by:

setFenced in interface LedgerStorage

Parameters:

ledgerId - Ledger Id.

Throws:

IOException - when failed to fence the ledger.

setMasterKey

public void setMasterKey(long ledgerId,
 byte[] masterKey)
                  throws IOException

Description copied from interface: LedgerStorage

Set the master key for a ledger.

Specified by:

setMasterKey in interface LedgerStorage

Throws:

IOException

readMasterKey

public byte[] readMasterKey(long ledgerId)
                     throws IOException,
BookieException

Description copied from interface: LedgerStorage

Get the master key for a ledger.

Specified by:

readMasterKey in interface LedgerStorage

Throws:

IOException - if there is an error reading the from the ledger

BookieException - if no such ledger exists

addEntry

public long addEntry(io.netty.buffer.ByteBuf entry)
              throws IOException,
BookieException

Description copied from interface: LedgerStorage

Add an entry to the storage.

Specified by:

addEntry in interface LedgerStorage

Returns:

the entry id of the entry added

Throws:

IOException

BookieException

getEntry

public io.netty.buffer.ByteBuf getEntry(long ledgerId,
 long entryId)
                                 throws IOException,
BookieException

Description copied from interface: LedgerStorage

Read an entry from storage.

Specified by:

getEntry in interface LedgerStorage

Throws:

IOException

BookieException

chargeReadAheadCache

protected boolean chargeReadAheadCache(int currentReadAheadCount,
 long currentReadAheadBytes)

getLastEntry

public io.netty.buffer.ByteBuf getLastEntry(long ledgerId)
                                     throws IOException,
BookieException

Throws:

IOException

BookieException

isFlushRequired

boolean isFlushRequired()

checkpoint

public void checkpoint(CheckpointSource.Checkpoint checkpoint)
                throws IOException

Description copied from interface: LedgerStorage

Ask the ledger storage to sync data until the given checkpoint. The ledger storage implementation do checkpoint and return the real checkpoint that it finished. The returned the checkpoint indicates that all entries added before that point already persist.

Specified by:

checkpoint in interface LedgerStorage

Parameters:

checkpoint - Check Point that CheckpointSource.Checkpoint proposed.

Throws:

IOException

flush

public void flush()
           throws IOException

Description copied from interface: LedgerStorage

Flushes all data in the storage. Once this is called, add data written to the LedgerStorage up until this point has been persisted to perminant storage

Specified by:

flush in interface LedgerStorage

Throws:

IOException

deleteLedger

public void deleteLedger(long ledgerId)
                  throws IOException

Specified by:

deleteLedger in interface LedgerStorage

Throws:

IOException

getActiveLedgersInRange

public Iterable<Long> getActiveLedgersInRange(long firstLedgerId,
 long lastLedgerId)
                                       throws IOException

Description copied from interface: CompactableLedgerStorage

Get an iterator over a range of ledger ids stored in the bookie.

Specified by:

getActiveLedgersInRange in interface CompactableLedgerStorage

Parameters:

firstLedgerId - first ledger id in the sequence (included)

lastLedgerId - last ledger id in the sequence (not included)

Returns:

Throws:

IOException

updateEntriesLocations

public void updateEntriesLocations(Iterable<EntryLocation> locations)
                            throws IOException

Description copied from interface: CompactableLedgerStorage

Update the location of several entries.

Specified by:

updateEntriesLocations in interface CompactableLedgerStorage

Parameters:

locations - the list of locations to update

Throws:

IOException

getEntryLogger

EntryLogger getEntryLogger()

getLastAddConfirmed

public long getLastAddConfirmed(long ledgerId)
                         throws IOException,
BookieException

Description copied from interface: LedgerStorage

Get last add confirmed.

Specified by:

getLastAddConfirmed in interface LedgerStorage

Parameters:

ledgerId - ledger id.

Returns:

last add confirmed.

Throws:

IOException

BookieException

waitForLastAddConfirmedUpdate

public boolean waitForLastAddConfirmedUpdate(long ledgerId,
 long previousLAC,
 Watcher<LastAddConfirmedUpdateNotification> watcher)
                                      throws IOException

Description copied from interface: LedgerStorage

Wait for last add confirmed update.

Specified by:

waitForLastAddConfirmedUpdate in interface LedgerStorage

previousLAC - - The threshold beyond which we would wait for the update

watcher - - Watcher to notify on update

Returns:

Throws:

IOException

cancelWaitForLastAddConfirmedUpdate

public void cancelWaitForLastAddConfirmedUpdate(long ledgerId,
 Watcher<LastAddConfirmedUpdateNotification> watcher)
                                         throws IOException

Description copied from interface: LedgerStorage

Cancel a previous wait for last add confirmed update.

Specified by:

cancelWaitForLastAddConfirmedUpdate in interface LedgerStorage

Parameters:

ledgerId - The ledger being watched.

watcher - The watcher to cancel.

Throws:

IOException

setExplicitLac

public void setExplicitLac(long ledgerId,
 io.netty.buffer.ByteBuf lac)
                    throws IOException

Specified by:

setExplicitLac in interface LedgerStorage

Throws:

IOException

getExplicitLac

public io.netty.buffer.ByteBuf getExplicitLac(long ledgerId)
                                       throws IOException,
BookieException

Specified by:

getExplicitLac in interface LedgerStorage

Throws:

IOException

BookieException

flushEntriesLocationsIndex

public void flushEntriesLocationsIndex()
                                throws IOException

Description copied from interface: CompactableLedgerStorage

Flush the entries locations index for the compacted entries.

Specified by:

flushEntriesLocationsIndex in interface CompactableLedgerStorage

Throws:

IOException

addLedgerToIndex

public long addLedgerToIndex(long ledgerId,
 boolean isFenced,
 byte[] masterKey,
 LedgerCache.PageEntriesIterable pages)
                      throws Exception

Add an already existing ledger to the index.

This method is only used as a tool to help the migration from InterleaveLedgerStorage to DbLedgerStorage

Parameters:

ledgerId - the ledger id

pages - Iterator over index pages from Indexed

Returns:

the number of

Throws:

Exception

registerLedgerDeletionListener

public void registerLedgerDeletionListener(LedgerStorage.LedgerDeletionListener listener)

Description copied from interface: LedgerStorage

Register a listener for ledgers deletion notifications.

Specified by:

registerLedgerDeletionListener in interface LedgerStorage

Parameters:

listener - object that will be notified every time a ledger is deleted

getEntryLocationIndex

public EntryLocationIndex getEntryLocationIndex()

getGarbageCollectionStatus

public List<GarbageCollectionStatus> getGarbageCollectionStatus()

Description copied from interface: LedgerStorage

Get Garbage Collection status. Since DbLedgerStorage is a list of storage instances, we should return a list.

Specified by:

getGarbageCollectionStatus in interface LedgerStorage

getListOfEntriesOfLedger

public PrimitiveIterator.OfLong getListOfEntriesOfLedger(long ledgerId)
                                                  throws IOException

Description copied from interface: LedgerStorage

Returns the primitive long iterator for entries of the ledger, stored in this LedgerStorage. The returned iterator provide weakly consistent state of the ledger. It is guaranteed that entries of the ledger added to this LedgerStorage by the time this method is called will be available but modifications made after method invocation may not be available.

Specified by:

getListOfEntriesOfLedger in interface LedgerStorage

Parameters:

ledgerId - - id of the ledger

Returns:

the list of entries of the ledger available in this ledgerstorage.

Throws:

IOException

setLimboState

public void setLimboState(long ledgerId)
                   throws IOException

Description copied from interface: LedgerStorage

Set a ledger to limbo state. When a ledger is in limbo state, we cannot answer any requests about it. For example, if a client asks for an entry, we cannot say we don't have it because it may have been written to us in the past, but we are waiting for data integrity checks to copy it over.

Specified by:

setLimboState in interface LedgerStorage

Throws:

IOException

hasLimboState

public boolean hasLimboState(long ledgerId)
                      throws IOException

Description copied from interface: LedgerStorage

Check whether a ledger is in limbo state.

Specified by:

hasLimboState in interface LedgerStorage

Throws:

IOException

See Also:

• LedgerStorage.setLimboState(long)

clearLimboState

public void clearLimboState(long ledgerId)
                     throws IOException

Description copied from interface: LedgerStorage

Clear the limbo state of a ledger.

Specified by:

clearLimboState in interface LedgerStorage

Throws:

IOException

See Also:

• LedgerStorage.setLimboState(long)

getStorageStateFlags

public EnumSet<LedgerStorage.StorageState> getStorageStateFlags()
                                                         throws IOException

Description copied from interface: LedgerStorage

Get the storage state flags currently set for the storage instance.

Specified by:

getStorageStateFlags in interface LedgerStorage

Throws:

IOException

setStorageStateFlag

public void setStorageStateFlag(LedgerStorage.StorageState flag)
                         throws IOException

Description copied from interface: LedgerStorage

Set a storage state flag for the storage instance. Implementations must ensure this method is atomic, and the flag is persisted to storage when the method returns.

Specified by:

setStorageStateFlag in interface LedgerStorage

Throws:

IOException

clearStorageStateFlag

public void clearStorageStateFlag(LedgerStorage.StorageState flag)
                           throws IOException

Description copied from interface: LedgerStorage

Clear a storage state flag for the storage instance. Implementations must ensure this method is atomic, and the flag is persisted to storage when the method returns.

Specified by:

clearStorageStateFlag in interface LedgerStorage

Throws:

IOException

getDbLedgerStorageStats

DbLedgerStorageStats getDbLedgerStorageStats()

getLedgerBaseDir

public String getLedgerBaseDir()

getIndexBaseDir

public String getIndexBaseDir()