Package org.apache.bookkeeper.bookie
Class SortedLedgerStorage
- java.lang.Object
-
- org.apache.bookkeeper.bookie.SortedLedgerStorage
-
- All Implemented Interfaces:
CacheCallback
,CompactableLedgerStorage
,DefaultEntryLogger.EntryLogListener
,LedgerStorage
,SkipListFlusher
public class SortedLedgerStorage extends java.lang.Object implements LedgerStorage, CacheCallback, SkipListFlusher, CompactableLedgerStorage, DefaultEntryLogger.EntryLogListener
ASortedLedgerStorage
is an extension ofInterleavedLedgerStorage
. It is comprised of twoMemTable
s and aInterleavedLedgerStorage
. All the entries will be first added into aMemTable
, and then be flushed back to theInterleavedLedgerStorage
when theMemTable
becomes full.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from interface org.apache.bookkeeper.bookie.LedgerStorage
LedgerStorage.DetectedInconsistency, LedgerStorage.LedgerDeletionListener, LedgerStorage.StorageState
-
-
Field Summary
Fields Modifier and Type Field Description (package private) EntryMemTable
memTable
-
Constructor Summary
Constructors Modifier Constructor Description SortedLedgerStorage()
protected
SortedLedgerStorage(InterleavedLedgerStorage ils)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description long
addEntry(io.netty.buffer.ByteBuf entry)
Add an entry to the storage.void
cancelWaitForLastAddConfirmedUpdate(long ledgerId, Watcher<LastAddConfirmedUpdateNotification> watcher)
Cancel a previous wait for last add confirmed update.void
checkpoint(CheckpointSource.Checkpoint checkpoint)
Ask the ledger storage to sync data until the given checkpoint.void
clearLimboState(long ledgerId)
Clear the limbo state of a ledger.void
clearStorageStateFlag(LedgerStorage.StorageState flags)
Clear a storage state flag for the storage instance.void
deleteLedger(long ledgerId)
boolean
entryExists(long ledgerId, long entryId)
Whether an entry exists.void
flush()
Flushes all data in the storage.void
flushEntriesLocationsIndex()
Flush the entries locations index for the compacted entries.void
forceGC()
Force trigger Garbage Collection.void
forceGC(boolean forceMajor, boolean forceMinor)
Force trigger Garbage Collection with forceMajor or forceMinor parameter.java.lang.Iterable<java.lang.Long>
getActiveLedgersInRange(long firstLedgerId, long lastLedgerId)
Get an iterator over a range of ledger ids stored in the bookie.io.netty.buffer.ByteBuf
getEntry(long ledgerId, long entryId)
Read an entry from storage.DefaultEntryLogger
getEntryLogger()
io.netty.buffer.ByteBuf
getExplicitLac(long ledgerId)
java.util.List<GarbageCollectionStatus>
getGarbageCollectionStatus()
Get Garbage Collection status.long
getLastAddConfirmed(long ledgerId)
Get last add confirmed.java.util.PrimitiveIterator.OfLong
getListOfEntriesOfLedger(long ledgerId)
Returns the primitive long iterator for entries of the ledger, stored in this LedgerStorage.(package private) java.util.concurrent.ScheduledExecutorService
getScheduler()
(package private) BookieStateManager
getStateManager()
java.util.EnumSet<LedgerStorage.StorageState>
getStorageStateFlags()
Get the storage state flags currently set for the storage instance.LedgerStorage
getUnderlyingLedgerStorage()
boolean
hasLimboState(long ledgerId)
Check whether a ledger is in limbo state.void
initialize(ServerConfiguration conf, LedgerManager ledgerManager, LedgerDirsManager ledgerDirsManager, LedgerDirsManager indexDirsManager, StatsLogger statsLogger, io.netty.buffer.ByteBufAllocator allocator)
Initialize the LedgerStorage implementation.boolean
isFenced(long ledgerId)
Check whether the ledger is fenced in ledger storage or not.boolean
isInForceGC()
Whether force triggered Garbage Collection is running or not.boolean
isMajorGcSuspended()
boolean
isMinorGcSuspended()
boolean
ledgerExists(long ledgerId)
Whether a ledger exists.java.util.List<LedgerStorage.DetectedInconsistency>
localConsistencyCheck(java.util.Optional<com.google.common.util.concurrent.RateLimiter> rateLimiter)
Performs internal check of local storage logging any inconsistencies.(package private) static java.util.concurrent.ScheduledExecutorService
newScheduledExecutorService()
void
onRotateEntryLog()
Rotate a new entry log to write.void
onSizeLimitReached(CheckpointSource.Checkpoint cp)
Process notification that cache size limit reached.void
process(long ledgerId, long entryId, io.netty.buffer.ByteBuf buffer)
Process an entry.byte[]
readMasterKey(long ledgerId)
Get the master key for a ledger.void
registerLedgerDeletionListener(LedgerStorage.LedgerDeletionListener listener)
Register a listener for ledgers deletion notifications.void
resumeMajorGC()
void
resumeMinorGC()
void
setCheckpointer(Checkpointer checkpointer)
void
setCheckpointSource(CheckpointSource checkpointSource)
void
setExplicitLac(long ledgerId, io.netty.buffer.ByteBuf lac)
boolean
setFenced(long ledgerId)
Fenced the ledger id in ledger storage.void
setLimboState(long ledgerId)
Set a ledger to limbo state.void
setMasterKey(long ledgerId, byte[] masterKey)
Set the master key for a ledger.void
setStateManager(StateManager stateManager)
void
setStorageStateFlag(LedgerStorage.StorageState flags)
Set a storage state flag for the storage instance.void
shutdown()
Cleanup and free any resources being used by the storage system.void
start()
Start any background threads belonging to the storage system.void
suspendMajorGC()
void
suspendMinorGC()
void
updateEntriesLocations(java.lang.Iterable<EntryLocation> locations)
Update the location of several entries.boolean
waitForLastAddConfirmedUpdate(long ledgerId, long previousLAC, Watcher<LastAddConfirmedUpdateNotification> watcher)
Wait for last add confirmed update.-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface org.apache.bookkeeper.bookie.LedgerStorage
entryLocationCompact, entryLocationCompact, getEntryLocationDBPath, isEntryLocationCompacting, isEntryLocationCompacting
-
-
-
-
Field Detail
-
memTable
EntryMemTable memTable
-
-
Constructor Detail
-
SortedLedgerStorage
public SortedLedgerStorage()
-
SortedLedgerStorage
protected SortedLedgerStorage(InterleavedLedgerStorage ils)
-
-
Method Detail
-
initialize
public void initialize(ServerConfiguration conf, LedgerManager ledgerManager, LedgerDirsManager ledgerDirsManager, LedgerDirsManager indexDirsManager, StatsLogger statsLogger, io.netty.buffer.ByteBufAllocator allocator) throws java.io.IOException
Description copied from interface:LedgerStorage
Initialize the LedgerStorage implementation.- Specified by:
initialize
in interfaceLedgerStorage
- Throws:
java.io.IOException
-
newScheduledExecutorService
static java.util.concurrent.ScheduledExecutorService newScheduledExecutorService()
-
setStateManager
public void setStateManager(StateManager stateManager)
- Specified by:
setStateManager
in interfaceLedgerStorage
-
setCheckpointSource
public void setCheckpointSource(CheckpointSource checkpointSource)
- Specified by:
setCheckpointSource
in interfaceLedgerStorage
-
setCheckpointer
public void setCheckpointer(Checkpointer checkpointer)
- Specified by:
setCheckpointer
in interfaceLedgerStorage
-
getScheduler
java.util.concurrent.ScheduledExecutorService getScheduler()
-
start
public void start()
Description copied from interface:LedgerStorage
Start any background threads belonging to the storage system. For example, garbage collection.- Specified by:
start
in interfaceLedgerStorage
-
shutdown
public void shutdown() throws java.lang.InterruptedException
Description copied from interface:LedgerStorage
Cleanup and free any resources being used by the storage system.- Specified by:
shutdown
in interfaceLedgerStorage
- Throws:
java.lang.InterruptedException
-
ledgerExists
public boolean ledgerExists(long ledgerId) throws java.io.IOException
Description copied from interface:LedgerStorage
Whether a ledger exists.- Specified by:
ledgerExists
in interfaceLedgerStorage
- Throws:
java.io.IOException
-
entryExists
public boolean entryExists(long ledgerId, long entryId) throws java.io.IOException
Description copied from interface:LedgerStorage
Whether an entry exists.- Specified by:
entryExists
in interfaceLedgerStorage
- Throws:
java.io.IOException
-
setFenced
public boolean setFenced(long ledgerId) throws java.io.IOException
Description copied from interface:LedgerStorage
Fenced the ledger id in ledger storage.- Specified by:
setFenced
in interfaceLedgerStorage
- Parameters:
ledgerId
- Ledger Id.- Throws:
java.io.IOException
- when failed to fence the ledger.
-
isFenced
public boolean isFenced(long ledgerId) throws java.io.IOException
Description copied from interface:LedgerStorage
Check whether the ledger is fenced in ledger storage or not.- Specified by:
isFenced
in interfaceLedgerStorage
- Parameters:
ledgerId
- Ledger ID.- Throws:
java.io.IOException
-
setMasterKey
public void setMasterKey(long ledgerId, byte[] masterKey) throws java.io.IOException
Description copied from interface:LedgerStorage
Set the master key for a ledger.- Specified by:
setMasterKey
in interfaceLedgerStorage
- Throws:
java.io.IOException
-
readMasterKey
public byte[] readMasterKey(long ledgerId) throws java.io.IOException, BookieException
Description copied from interface:LedgerStorage
Get the master key for a ledger.- Specified by:
readMasterKey
in interfaceLedgerStorage
- Throws:
java.io.IOException
- if there is an error reading the from the ledgerBookieException
- if no such ledger exists
-
addEntry
public long addEntry(io.netty.buffer.ByteBuf entry) throws java.io.IOException
Description copied from interface:LedgerStorage
Add an entry to the storage.- Specified by:
addEntry
in interfaceLedgerStorage
- Returns:
- the entry id of the entry added
- Throws:
java.io.IOException
-
getEntry
public io.netty.buffer.ByteBuf getEntry(long ledgerId, long entryId) throws java.io.IOException, BookieException
Description copied from interface:LedgerStorage
Read an entry from storage.- Specified by:
getEntry
in interfaceLedgerStorage
- Throws:
java.io.IOException
BookieException
-
getLastAddConfirmed
public long getLastAddConfirmed(long ledgerId) throws java.io.IOException
Description copied from interface:LedgerStorage
Get last add confirmed.- Specified by:
getLastAddConfirmed
in interfaceLedgerStorage
- Parameters:
ledgerId
- ledger id.- Returns:
- last add confirmed.
- Throws:
java.io.IOException
-
waitForLastAddConfirmedUpdate
public boolean waitForLastAddConfirmedUpdate(long ledgerId, long previousLAC, Watcher<LastAddConfirmedUpdateNotification> watcher) throws java.io.IOException
Description copied from interface:LedgerStorage
Wait for last add confirmed update.- Specified by:
waitForLastAddConfirmedUpdate
in interfaceLedgerStorage
previousLAC
- - The threshold beyond which we would wait for the updatewatcher
- - Watcher to notify on update- Returns:
- Throws:
java.io.IOException
-
cancelWaitForLastAddConfirmedUpdate
public void cancelWaitForLastAddConfirmedUpdate(long ledgerId, Watcher<LastAddConfirmedUpdateNotification> watcher) throws java.io.IOException
Description copied from interface:LedgerStorage
Cancel a previous wait for last add confirmed update.- Specified by:
cancelWaitForLastAddConfirmedUpdate
in interfaceLedgerStorage
- Parameters:
ledgerId
- The ledger being watched.watcher
- The watcher to cancel.- Throws:
java.io.IOException
-
checkpoint
public void checkpoint(CheckpointSource.Checkpoint checkpoint) throws java.io.IOException
Description copied from interface:LedgerStorage
Ask the ledger storage to sync data until the given checkpoint. The ledger storage implementation do checkpoint and return the real checkpoint that it finished. The returned the checkpoint indicates that all entries added before that point already persist.- Specified by:
checkpoint
in interfaceLedgerStorage
- Parameters:
checkpoint
- Check Point thatCheckpointSource.Checkpoint
proposed.- Throws:
java.io.IOException
-
deleteLedger
public void deleteLedger(long ledgerId) throws java.io.IOException
- Specified by:
deleteLedger
in interfaceLedgerStorage
- Throws:
java.io.IOException
-
registerLedgerDeletionListener
public void registerLedgerDeletionListener(LedgerStorage.LedgerDeletionListener listener)
Description copied from interface:LedgerStorage
Register a listener for ledgers deletion notifications.- Specified by:
registerLedgerDeletionListener
in interfaceLedgerStorage
- Parameters:
listener
- object that will be notified every time a ledger is deleted
-
setExplicitLac
public void setExplicitLac(long ledgerId, io.netty.buffer.ByteBuf lac) throws java.io.IOException
- Specified by:
setExplicitLac
in interfaceLedgerStorage
- Throws:
java.io.IOException
-
getExplicitLac
public io.netty.buffer.ByteBuf getExplicitLac(long ledgerId)
- Specified by:
getExplicitLac
in interfaceLedgerStorage
-
process
public void process(long ledgerId, long entryId, io.netty.buffer.ByteBuf buffer) throws java.io.IOException
Description copied from interface:SkipListFlusher
Process an entry.- Specified by:
process
in interfaceSkipListFlusher
- Parameters:
ledgerId
- Ledger ID.entryId
- The entry id this entry.buffer
- Entry ByteBuffer- Throws:
java.io.IOException
-
flush
public void flush() throws java.io.IOException
Description copied from interface:LedgerStorage
Flushes all data in the storage. Once this is called, add data written to the LedgerStorage up until this point has been persisted to perminant storage- Specified by:
flush
in interfaceLedgerStorage
- Throws:
java.io.IOException
-
onSizeLimitReached
public void onSizeLimitReached(CheckpointSource.Checkpoint cp) throws java.io.IOException
Description copied from interface:CacheCallback
Process notification that cache size limit reached.- Specified by:
onSizeLimitReached
in interfaceCacheCallback
- Throws:
java.io.IOException
-
onRotateEntryLog
public void onRotateEntryLog()
Description copied from interface:DefaultEntryLogger.EntryLogListener
Rotate a new entry log to write.- Specified by:
onRotateEntryLog
in interfaceDefaultEntryLogger.EntryLogListener
-
getStateManager
BookieStateManager getStateManager()
-
getEntryLogger
public DefaultEntryLogger getEntryLogger()
-
getActiveLedgersInRange
public java.lang.Iterable<java.lang.Long> getActiveLedgersInRange(long firstLedgerId, long lastLedgerId) throws java.io.IOException
Description copied from interface:CompactableLedgerStorage
Get an iterator over a range of ledger ids stored in the bookie.- Specified by:
getActiveLedgersInRange
in interfaceCompactableLedgerStorage
- Parameters:
firstLedgerId
- first ledger id in the sequence (included)lastLedgerId
- last ledger id in the sequence (not included)- Returns:
- Throws:
java.io.IOException
-
updateEntriesLocations
public void updateEntriesLocations(java.lang.Iterable<EntryLocation> locations) throws java.io.IOException
Description copied from interface:CompactableLedgerStorage
Update the location of several entries.- Specified by:
updateEntriesLocations
in interfaceCompactableLedgerStorage
- Parameters:
locations
- the list of locations to update- Throws:
java.io.IOException
-
flushEntriesLocationsIndex
public void flushEntriesLocationsIndex() throws java.io.IOException
Description copied from interface:CompactableLedgerStorage
Flush the entries locations index for the compacted entries.- Specified by:
flushEntriesLocationsIndex
in interfaceCompactableLedgerStorage
- Throws:
java.io.IOException
-
getUnderlyingLedgerStorage
public LedgerStorage getUnderlyingLedgerStorage()
- Specified by:
getUnderlyingLedgerStorage
in interfaceLedgerStorage
-
forceGC
public void forceGC()
Description copied from interface:LedgerStorage
Force trigger Garbage Collection.- Specified by:
forceGC
in interfaceLedgerStorage
-
forceGC
public void forceGC(boolean forceMajor, boolean forceMinor)
Description copied from interface:LedgerStorage
Force trigger Garbage Collection with forceMajor or forceMinor parameter.- Specified by:
forceGC
in interfaceLedgerStorage
-
suspendMinorGC
public void suspendMinorGC()
- Specified by:
suspendMinorGC
in interfaceLedgerStorage
-
suspendMajorGC
public void suspendMajorGC()
- Specified by:
suspendMajorGC
in interfaceLedgerStorage
-
resumeMinorGC
public void resumeMinorGC()
- Specified by:
resumeMinorGC
in interfaceLedgerStorage
-
resumeMajorGC
public void resumeMajorGC()
- Specified by:
resumeMajorGC
in interfaceLedgerStorage
-
isMajorGcSuspended
public boolean isMajorGcSuspended()
- Specified by:
isMajorGcSuspended
in interfaceLedgerStorage
-
isMinorGcSuspended
public boolean isMinorGcSuspended()
- Specified by:
isMinorGcSuspended
in interfaceLedgerStorage
-
localConsistencyCheck
public java.util.List<LedgerStorage.DetectedInconsistency> localConsistencyCheck(java.util.Optional<com.google.common.util.concurrent.RateLimiter> rateLimiter) throws java.io.IOException
Description copied from interface:LedgerStorage
Performs internal check of local storage logging any inconsistencies.- Specified by:
localConsistencyCheck
in interfaceLedgerStorage
- Parameters:
rateLimiter
- Provide to rate of entry checking. null for unlimited.- Returns:
- List of inconsistencies detected
- Throws:
java.io.IOException
-
isInForceGC
public boolean isInForceGC()
Description copied from interface:LedgerStorage
Whether force triggered Garbage Collection is running or not.- Specified by:
isInForceGC
in interfaceLedgerStorage
- Returns:
- true -- force triggered Garbage Collection is running, false -- force triggered Garbage Collection is not running
-
getGarbageCollectionStatus
public java.util.List<GarbageCollectionStatus> getGarbageCollectionStatus()
Description copied from interface:LedgerStorage
Get Garbage Collection status. Since DbLedgerStorage is a list of storage instances, we should return a list.- Specified by:
getGarbageCollectionStatus
in interfaceLedgerStorage
-
getListOfEntriesOfLedger
public java.util.PrimitiveIterator.OfLong getListOfEntriesOfLedger(long ledgerId) throws java.io.IOException
Description copied from interface:LedgerStorage
Returns the primitive long iterator for entries of the ledger, stored in this LedgerStorage. The returned iterator provide weakly consistent state of the ledger. It is guaranteed that entries of the ledger added to this LedgerStorage by the time this method is called will be available but modifications made after method invocation may not be available.- Specified by:
getListOfEntriesOfLedger
in interfaceLedgerStorage
- Parameters:
ledgerId
- - id of the ledger- Returns:
- the list of entries of the ledger available in this ledgerstorage.
- Throws:
java.io.IOException
-
setLimboState
public void setLimboState(long ledgerId) throws java.io.IOException
Description copied from interface:LedgerStorage
Set a ledger to limbo state. When a ledger is in limbo state, we cannot answer any requests about it. For example, if a client asks for an entry, we cannot say we don't have it because it may have been written to us in the past, but we are waiting for data integrity checks to copy it over.- Specified by:
setLimboState
in interfaceLedgerStorage
- Throws:
java.io.IOException
-
hasLimboState
public boolean hasLimboState(long ledgerId) throws java.io.IOException
Description copied from interface:LedgerStorage
Check whether a ledger is in limbo state.- Specified by:
hasLimboState
in interfaceLedgerStorage
- Throws:
java.io.IOException
- See Also:
LedgerStorage.setLimboState(long)
-
clearLimboState
public void clearLimboState(long ledgerId) throws java.io.IOException
Description copied from interface:LedgerStorage
Clear the limbo state of a ledger.- Specified by:
clearLimboState
in interfaceLedgerStorage
- Throws:
java.io.IOException
- See Also:
LedgerStorage.setLimboState(long)
-
getStorageStateFlags
public java.util.EnumSet<LedgerStorage.StorageState> getStorageStateFlags() throws java.io.IOException
Description copied from interface:LedgerStorage
Get the storage state flags currently set for the storage instance.- Specified by:
getStorageStateFlags
in interfaceLedgerStorage
- Throws:
java.io.IOException
-
setStorageStateFlag
public void setStorageStateFlag(LedgerStorage.StorageState flags) throws java.io.IOException
Description copied from interface:LedgerStorage
Set a storage state flag for the storage instance. Implementations must ensure this method is atomic, and the flag is persisted to storage when the method returns.- Specified by:
setStorageStateFlag
in interfaceLedgerStorage
- Throws:
java.io.IOException
-
clearStorageStateFlag
public void clearStorageStateFlag(LedgerStorage.StorageState flags) throws java.io.IOException
Description copied from interface:LedgerStorage
Clear a storage state flag for the storage instance. Implementations must ensure this method is atomic, and the flag is persisted to storage when the method returns.- Specified by:
clearStorageStateFlag
in interfaceLedgerStorage
- Throws:
java.io.IOException
-
-