Package org.apache.bookkeeper.bookie

Class InterleavedLedgerStorage

java.lang.Object

org.apache.bookkeeper.bookie.InterleavedLedgerStorage

All Implemented Interfaces:

CompactableLedgerStorage, DefaultEntryLogger.EntryLogListener, LedgerStorage

@StatsDoc(name="bookie",
          category="server",
          help="Bookie related stats")
public class InterleavedLedgerStorage
extends Object
implements CompactableLedgerStorage, DefaultEntryLogger.EntryLogListener

Interleave ledger storage.

This ledger storage implementation stores all entries in a single file and maintains an index file for each ledger.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.bookkeeper.bookie.LedgerStorage

LedgerStorage.DetectedInconsistency, LedgerStorage.LedgerDeletionListener, LedgerStorage.StorageState

Field Summary

Fields

Modifier and Type	Field	Description
protected final SnapshotMap<Long,Boolean>	activeLedgers
protected Checkpointer	checkpointer
protected CheckpointSource	checkpointSource
(package private) DefaultEntryLogger	entryLogger
(package private) GarbageCollectorThread	gcThread
(package private) LedgerCache	ledgerCache

Constructor Summary

Constructors

Constructor	Description
InterleavedLedgerStorage()

Method Summary

Modifier and Type	Method	Description
long	addEntry(io.netty.buffer.ByteBuf entry)	Add an entry to the storage.
void	cancelWaitForLastAddConfirmedUpdate(long ledgerId, Watcher<LastAddConfirmedUpdateNotification> watcher)	Cancel a previous wait for last add confirmed update.
void	checkpoint(CheckpointSource.Checkpoint checkpoint)	Ask the ledger storage to sync data until the given checkpoint.
void	clearLimboState(long ledgerId)	Clear the limbo state of a ledger.
void	clearStorageStateFlag(LedgerStorage.StorageState flags)	Clear a storage state flag for the storage instance.
void	deleteLedger(long ledgerId)
boolean	entryExists(long ledgerId, long entryId)	Whether an entry exists.
void	flush()	Flushes all data in the storage.
void	flushEntriesLocationsIndex()	Flush the entries locations index for the compacted entries.
void	forceGC()	Force trigger Garbage Collection.
void	forceGC(boolean forceMajor, boolean forceMinor)	Force trigger Garbage Collection with forceMajor or forceMinor parameter.
Iterable<Long>	getActiveLedgersInRange(long firstLedgerId, long lastLedgerId)	Get an iterator over a range of ledger ids stored in the bookie.
io.netty.buffer.ByteBuf	getEntry(long ledgerId, long entryId)	Read an entry from storage.
DefaultEntryLogger	getEntryLogger()
io.netty.buffer.ByteBuf	getExplicitLac(long ledgerId)
List<GarbageCollectionStatus>	getGarbageCollectionStatus()	Get Garbage Collection status.
LedgerCache.PageEntriesIterable	getIndexEntries(long ledgerId)	Return iterable for index entries for ledgerId.
long	getLastAddConfirmed(long ledgerId)	Get last add confirmed.
LedgerCache	getLedgerCache()
PrimitiveIterator.OfLong	getListOfEntriesOfLedger(long ledgerId)	Returns the primitive long iterator for entries of the ledger, stored in this LedgerStorage.
EnumSet<LedgerStorage.StorageState>	getStorageStateFlags()	Get the storage state flags currently set for the storage instance.
boolean	hasLimboState(long ledgerId)	Check whether a ledger is in limbo state.
void	initialize(ServerConfiguration conf, LedgerManager ledgerManager, LedgerDirsManager ledgerDirsManager, LedgerDirsManager indexDirsManager, StatsLogger statsLogger, io.netty.buffer.ByteBufAllocator allocator)	Initialize the LedgerStorage implementation.
void	initializeWithEntryLogger(ServerConfiguration conf, LedgerManager ledgerManager, LedgerDirsManager ledgerDirsManager, LedgerDirsManager indexDirsManager, EntryLogger entryLogger, StatsLogger statsLogger)
(package private) void	initializeWithEntryLogListener(ServerConfiguration conf, LedgerManager ledgerManager, LedgerDirsManager ledgerDirsManager, LedgerDirsManager indexDirsManager, DefaultEntryLogger.EntryLogListener entryLogListener, StatsLogger statsLogger, io.netty.buffer.ByteBufAllocator allocator)
boolean	isFenced(long ledgerId)	Check whether the ledger is fenced in ledger storage or not.
boolean	isInForceGC()	Whether force triggered Garbage Collection is running or not.
boolean	isMajorGcSuspended()
boolean	isMinorGcSuspended()
boolean	ledgerExists(long ledgerId)	Whether a ledger exists.
List<LedgerStorage.DetectedInconsistency>	localConsistencyCheck(Optional<com.google.common.util.concurrent.RateLimiter> rateLimiter)	Performs internal check of local storage logging any inconsistencies.
void	onRotateEntryLog()	Rotate a new entry log to write.
protected void	processEntry(long ledgerId, long entryId, io.netty.buffer.ByteBuf entry)
protected void	processEntry(long ledgerId, long entryId, io.netty.buffer.ByteBuf entry, boolean rollLog)
LedgerCache.LedgerIndexMetadata	readLedgerIndexMetadata(long ledgerId)	Read implementation metadata for index file.
byte[]	readMasterKey(long ledgerId)	Get the master key for a ledger.
void	registerLedgerDeletionListener(LedgerStorage.LedgerDeletionListener listener)	Register a listener for ledgers deletion notifications.
void	resumeMajorGC()
void	resumeMinorGC()
void	setCheckpointer(Checkpointer checkpointer)
void	setCheckpointSource(CheckpointSource checkpointSource)
void	setExplicitLac(long ledgerId, io.netty.buffer.ByteBuf lac)
boolean	setFenced(long ledgerId)	Fenced the ledger id in ledger storage.
void	setLimboState(long ledgerId)	Set a ledger to limbo state.
void	setMasterKey(long ledgerId, byte[] masterKey)	Set the master key for a ledger.
void	setStateManager(StateManager stateManager)
void	setStorageStateFlag(LedgerStorage.StorageState flags)	Set a storage state flag for the storage instance.
void	shutdown()	Cleanup and free any resources being used by the storage system.
void	start()	Start any background threads belonging to the storage system.
void	suspendMajorGC()
void	suspendMinorGC()
void	updateEntriesLocations(Iterable<EntryLocation> locations)	Update the location of several entries.
boolean	waitForLastAddConfirmedUpdate(long ledgerId, long previousLAC, Watcher<LastAddConfirmedUpdateNotification> watcher)	Wait for last add confirmed update.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.bookkeeper.bookie.LedgerStorage

entryLocationCompact, entryLocationCompact, getEntryLocationDBPath, getUnderlyingLedgerStorage, isEntryLocationCompacting, isEntryLocationCompacting

Field Details

entryLogger

DefaultEntryLogger entryLogger

ledgerCache

LedgerCache ledgerCache

checkpointSource

protected CheckpointSource checkpointSource

checkpointer

protected Checkpointer checkpointer

activeLedgers

protected final SnapshotMap<Long,Boolean> activeLedgers

gcThread

GarbageCollectorThread gcThread

Constructor Details

InterleavedLedgerStorage

public InterleavedLedgerStorage()

Method Details

initialize

public void initialize(ServerConfiguration conf,
 LedgerManager ledgerManager,
 LedgerDirsManager ledgerDirsManager,
 LedgerDirsManager indexDirsManager,
 StatsLogger statsLogger,
 io.netty.buffer.ByteBufAllocator allocator)
                throws IOException

Description copied from interface: LedgerStorage

Initialize the LedgerStorage implementation.

Specified by:

initialize in interface LedgerStorage

Throws:

IOException

initializeWithEntryLogListener

void initializeWithEntryLogListener(ServerConfiguration conf,
 LedgerManager ledgerManager,
 LedgerDirsManager ledgerDirsManager,
 LedgerDirsManager indexDirsManager,
 DefaultEntryLogger.EntryLogListener entryLogListener,
 StatsLogger statsLogger,
 io.netty.buffer.ByteBufAllocator allocator)
                             throws IOException

Throws:

IOException

setStateManager

public void setStateManager(StateManager stateManager)

Specified by:

setStateManager in interface LedgerStorage

setCheckpointSource

public void setCheckpointSource(CheckpointSource checkpointSource)

Specified by:

setCheckpointSource in interface LedgerStorage

setCheckpointer

public void setCheckpointer(Checkpointer checkpointer)

Specified by:

setCheckpointer in interface LedgerStorage

initializeWithEntryLogger

public void initializeWithEntryLogger(ServerConfiguration conf,
 LedgerManager ledgerManager,
 LedgerDirsManager ledgerDirsManager,
 LedgerDirsManager indexDirsManager,
 EntryLogger entryLogger,
 StatsLogger statsLogger)
                               throws IOException

Throws:

IOException

forceGC

public void forceGC()

Description copied from interface: LedgerStorage

Force trigger Garbage Collection.

Specified by:

forceGC in interface LedgerStorage

forceGC

public void forceGC(boolean forceMajor,
 boolean forceMinor)

Description copied from interface: LedgerStorage

Force trigger Garbage Collection with forceMajor or forceMinor parameter.

Specified by:

forceGC in interface LedgerStorage

isInForceGC

public boolean isInForceGC()

Description copied from interface: LedgerStorage

Whether force triggered Garbage Collection is running or not.

Specified by:

isInForceGC in interface LedgerStorage

Returns:

true -- force triggered Garbage Collection is running, false -- force triggered Garbage Collection is not running

suspendMinorGC

public void suspendMinorGC()

Specified by:

suspendMinorGC in interface LedgerStorage

suspendMajorGC

public void suspendMajorGC()

Specified by:

suspendMajorGC in interface LedgerStorage

resumeMinorGC

public void resumeMinorGC()

Specified by:

resumeMinorGC in interface LedgerStorage

resumeMajorGC

public void resumeMajorGC()

Specified by:

resumeMajorGC in interface LedgerStorage

isMajorGcSuspended

public boolean isMajorGcSuspended()

Specified by:

isMajorGcSuspended in interface LedgerStorage

isMinorGcSuspended

public boolean isMinorGcSuspended()

Specified by:

isMinorGcSuspended in interface LedgerStorage

start

public void start()

Description copied from interface: LedgerStorage

Start any background threads belonging to the storage system. For example, garbage collection.

Specified by:

start in interface LedgerStorage

shutdown

public void shutdown()
              throws InterruptedException

Description copied from interface: LedgerStorage

Cleanup and free any resources being used by the storage system.

Specified by:

shutdown in interface LedgerStorage

Throws:

InterruptedException

setFenced

public boolean setFenced(long ledgerId)
                  throws IOException

Description copied from interface: LedgerStorage

Fenced the ledger id in ledger storage.

Specified by:

setFenced in interface LedgerStorage

Parameters:

ledgerId - Ledger Id.

Throws:

IOException - when failed to fence the ledger.

isFenced

public boolean isFenced(long ledgerId)
                 throws IOException

Description copied from interface: LedgerStorage

Check whether the ledger is fenced in ledger storage or not.

Specified by:

isFenced in interface LedgerStorage

Parameters:

ledgerId - Ledger ID.

Throws:

IOException

setExplicitLac

public void setExplicitLac(long ledgerId,
 io.netty.buffer.ByteBuf lac)
                    throws IOException

Specified by:

setExplicitLac in interface LedgerStorage

Throws:

IOException

getExplicitLac

public io.netty.buffer.ByteBuf getExplicitLac(long ledgerId)

Specified by:

getExplicitLac in interface LedgerStorage

setMasterKey

public void setMasterKey(long ledgerId,
 byte[] masterKey)
                  throws IOException

Description copied from interface: LedgerStorage

Set the master key for a ledger.

Specified by:

setMasterKey in interface LedgerStorage

Throws:

IOException

readMasterKey

public byte[] readMasterKey(long ledgerId)
                     throws IOException,
BookieException

Description copied from interface: LedgerStorage

Get the master key for a ledger.

Specified by:

readMasterKey in interface LedgerStorage

Throws:

IOException - if there is an error reading the from the ledger

BookieException - if no such ledger exists

ledgerExists

public boolean ledgerExists(long ledgerId)
                     throws IOException

Description copied from interface: LedgerStorage

Whether a ledger exists.

Specified by:

ledgerExists in interface LedgerStorage

Throws:

IOException

entryExists

public boolean entryExists(long ledgerId,
 long entryId)
                    throws IOException

Description copied from interface: LedgerStorage

Whether an entry exists.

Specified by:

entryExists in interface LedgerStorage

Throws:

IOException

getLastAddConfirmed

public long getLastAddConfirmed(long ledgerId)
                         throws IOException

Description copied from interface: LedgerStorage

Get last add confirmed.

Specified by:

getLastAddConfirmed in interface LedgerStorage

Parameters:

ledgerId - ledger id.

Returns:

last add confirmed.

Throws:

IOException

waitForLastAddConfirmedUpdate

public boolean waitForLastAddConfirmedUpdate(long ledgerId,
 long previousLAC,
 Watcher<LastAddConfirmedUpdateNotification> watcher)
                                      throws IOException

Description copied from interface: LedgerStorage

Wait for last add confirmed update.

Specified by:

waitForLastAddConfirmedUpdate in interface LedgerStorage

previousLAC - - The threshold beyond which we would wait for the update

watcher - - Watcher to notify on update

Returns:

Throws:

IOException

cancelWaitForLastAddConfirmedUpdate

public void cancelWaitForLastAddConfirmedUpdate(long ledgerId,
 Watcher<LastAddConfirmedUpdateNotification> watcher)
                                         throws IOException

Description copied from interface: LedgerStorage

Cancel a previous wait for last add confirmed update.

Specified by:

cancelWaitForLastAddConfirmedUpdate in interface LedgerStorage

Parameters:

ledgerId - The ledger being watched.

watcher - The watcher to cancel.

Throws:

IOException

addEntry

public long addEntry(io.netty.buffer.ByteBuf entry)
              throws IOException

Description copied from interface: LedgerStorage

Add an entry to the storage.

Specified by:

addEntry in interface LedgerStorage

Returns:

the entry id of the entry added

Throws:

IOException

getEntry

public io.netty.buffer.ByteBuf getEntry(long ledgerId,
 long entryId)
                                 throws IOException

Description copied from interface: LedgerStorage

Read an entry from storage.

Specified by:

getEntry in interface LedgerStorage

Throws:

IOException

checkpoint

public void checkpoint(CheckpointSource.Checkpoint checkpoint)
                throws IOException

Description copied from interface: LedgerStorage

Ask the ledger storage to sync data until the given checkpoint. The ledger storage implementation do checkpoint and return the real checkpoint that it finished. The returned the checkpoint indicates that all entries added before that point already persist.

Specified by:

checkpoint in interface LedgerStorage

Parameters:

checkpoint - Check Point that CheckpointSource.Checkpoint proposed.

Throws:

IOException

flush

public void flush()
           throws IOException

Description copied from interface: LedgerStorage

Flushes all data in the storage. Once this is called, add data written to the LedgerStorage up until this point has been persisted to perminant storage

Specified by:

flush in interface LedgerStorage

Throws:

IOException

deleteLedger

public void deleteLedger(long ledgerId)
                  throws IOException

Specified by:

deleteLedger in interface LedgerStorage

Throws:

IOException

getActiveLedgersInRange

public Iterable<Long> getActiveLedgersInRange(long firstLedgerId,
 long lastLedgerId)

Description copied from interface: CompactableLedgerStorage

Get an iterator over a range of ledger ids stored in the bookie.

Specified by:

getActiveLedgersInRange in interface CompactableLedgerStorage

Parameters:

firstLedgerId - first ledger id in the sequence (included)

lastLedgerId - last ledger id in the sequence (not included)

Returns:

updateEntriesLocations

public void updateEntriesLocations(Iterable<EntryLocation> locations)
                            throws IOException

Description copied from interface: CompactableLedgerStorage

Update the location of several entries.

Specified by:

updateEntriesLocations in interface CompactableLedgerStorage

Parameters:

locations - the list of locations to update

Throws:

IOException

flushEntriesLocationsIndex

public void flushEntriesLocationsIndex()
                                throws IOException

Description copied from interface: CompactableLedgerStorage

Flush the entries locations index for the compacted entries.

Specified by:

flushEntriesLocationsIndex in interface CompactableLedgerStorage

Throws:

IOException

getEntryLogger

public DefaultEntryLogger getEntryLogger()

registerLedgerDeletionListener

public void registerLedgerDeletionListener(LedgerStorage.LedgerDeletionListener listener)

Description copied from interface: LedgerStorage

Register a listener for ledgers deletion notifications.

Specified by:

registerLedgerDeletionListener in interface LedgerStorage

Parameters:

listener - object that will be notified every time a ledger is deleted

processEntry

protected void processEntry(long ledgerId,
 long entryId,
 io.netty.buffer.ByteBuf entry)
                     throws IOException

Throws:

IOException

processEntry

protected void processEntry(long ledgerId,
 long entryId,
 io.netty.buffer.ByteBuf entry,
 boolean rollLog)
                     throws IOException

Throws:

IOException

onRotateEntryLog

public void onRotateEntryLog()

Description copied from interface: DefaultEntryLogger.EntryLogListener

Rotate a new entry log to write.

Specified by:

onRotateEntryLog in interface DefaultEntryLogger.EntryLogListener

getIndexEntries

public LedgerCache.PageEntriesIterable getIndexEntries(long ledgerId)
                                                throws IOException

Return iterable for index entries for ledgerId.

Parameters:

ledgerId - ledger to scan

Returns:

Iterator

Throws:

IOException

readLedgerIndexMetadata

public LedgerCache.LedgerIndexMetadata readLedgerIndexMetadata(long ledgerId)
                                                        throws IOException

Read implementation metadata for index file.

Parameters:

ledgerId -

Returns:

Implementation metadata

Throws:

IOException

localConsistencyCheck

public List<LedgerStorage.DetectedInconsistency> localConsistencyCheck(Optional<com.google.common.util.concurrent.RateLimiter> rateLimiter)
                                                                throws IOException

Description copied from interface: LedgerStorage

Performs internal check of local storage logging any inconsistencies.

Specified by:

localConsistencyCheck in interface LedgerStorage

Parameters:

rateLimiter - Provide to rate of entry checking. null for unlimited.

Returns:

List of inconsistencies detected

Throws:

IOException

getGarbageCollectionStatus

public List<GarbageCollectionStatus> getGarbageCollectionStatus()

Description copied from interface: LedgerStorage

Get Garbage Collection status. Since DbLedgerStorage is a list of storage instances, we should return a list.

Specified by:

getGarbageCollectionStatus in interface LedgerStorage

getListOfEntriesOfLedger

public PrimitiveIterator.OfLong getListOfEntriesOfLedger(long ledgerId)
                                                  throws IOException

Description copied from interface: LedgerStorage

Returns the primitive long iterator for entries of the ledger, stored in this LedgerStorage. The returned iterator provide weakly consistent state of the ledger. It is guaranteed that entries of the ledger added to this LedgerStorage by the time this method is called will be available but modifications made after method invocation may not be available.

Specified by:

getListOfEntriesOfLedger in interface LedgerStorage

Parameters:

ledgerId - - id of the ledger

Returns:

the list of entries of the ledger available in this ledgerstorage.

Throws:

IOException

setLimboState

public void setLimboState(long ledgerId)
                   throws IOException

Description copied from interface: LedgerStorage

Set a ledger to limbo state. When a ledger is in limbo state, we cannot answer any requests about it. For example, if a client asks for an entry, we cannot say we don't have it because it may have been written to us in the past, but we are waiting for data integrity checks to copy it over.

Specified by:

setLimboState in interface LedgerStorage

Throws:

IOException

hasLimboState

public boolean hasLimboState(long ledgerId)
                      throws IOException

Description copied from interface: LedgerStorage

Check whether a ledger is in limbo state.

Specified by:

hasLimboState in interface LedgerStorage

Throws:

IOException

See Also:

• LedgerStorage.setLimboState(long)

clearLimboState

public void clearLimboState(long ledgerId)
                     throws IOException

Description copied from interface: LedgerStorage

Clear the limbo state of a ledger.

Specified by:

clearLimboState in interface LedgerStorage

Throws:

IOException

See Also:

• LedgerStorage.setLimboState(long)

getStorageStateFlags

public EnumSet<LedgerStorage.StorageState> getStorageStateFlags()
                                                         throws IOException

Description copied from interface: LedgerStorage

Get the storage state flags currently set for the storage instance.

Specified by:

getStorageStateFlags in interface LedgerStorage

Throws:

IOException

setStorageStateFlag

public void setStorageStateFlag(LedgerStorage.StorageState flags)
                         throws IOException

Description copied from interface: LedgerStorage

Set a storage state flag for the storage instance. Implementations must ensure this method is atomic, and the flag is persisted to storage when the method returns.

Specified by:

setStorageStateFlag in interface LedgerStorage

Throws:

IOException

clearStorageStateFlag

public void clearStorageStateFlag(LedgerStorage.StorageState flags)
                           throws IOException

Description copied from interface: LedgerStorage

Clear a storage state flag for the storage instance. Implementations must ensure this method is atomic, and the flag is persisted to storage when the method returns.

Specified by:

clearStorageStateFlag in interface LedgerStorage

Throws:

IOException

getLedgerCache

public LedgerCache getLedgerCache()