Package org.objectweb.howl.log

Class LogBufferManager

  • class LogBufferManager
extends LogObject
    Provides a generalized buffer manager for journals and loggers.

    log records are written to disk as blocks of data. Block size is a multiple of 512 data to assure optimum disk performance.

    • Field Detail

      • haveIOException

        private boolean haveIOException
        boolean is set true when an IOException is returned by a write or force to a log file.

        Any attempt to write or force after haveIOException becomes true should result in an IOException being returned to the caller.

      • ioexception

        private java.io.IOException ioexception
        The last IOException returned to the logger

      • bufferManagerLock

        private final java.lang.Object bufferManagerLock
        mutex for synchronizing access to buffers list.

        also synchronizes access to fqPut in routines that put LogBuffers into the forceQueue[].

      • forceManagerLock

        private final java.lang.Object forceManagerLock
        mutex for synchronizing threads through the portion of force() that forces the channel.

      • forceRequired

        final boolean forceRequired
        indicates if a force() must be called in the flush() method.

        Set false in constructor if config.getLogFileMode() is "rwd".

      • fillBuffer

        private LogBuffer fillBuffer
        The LogBuffer that is currently being filled.

      • freeBuffer

        private LogBuffer[] freeBuffer
        array of LogBuffer objects available for filling

      • bufferList

        private LogBuffer[] bufferList
        array of all LogBuffer objects allocated.

        Used to find and debug buffers that are not in the freeBuffer list if logger hangs waiting for buffers to be returned to the freeBuffer pool.

      • nextIndex

        short nextIndex
        workerID into freeBuffer list maintained in getBuffer.

      • waitForBuffer

        private long waitForBuffer
        number of times there were no buffers available.

        The FlushManager thread monitors this field to determine if the buffer pool needs to be grown.

      • noRoomInBuffer

        private long noRoomInBuffer
        number of times buffer was forced because it is full.

      • growPoolCounter

        private int growPoolCounter
        number of times buffer size was increased because of threads waiting for buffers.

      • nextFillBSN

        int nextFillBSN
        next block sequence number for fillBuffer.

      • nextWriteBSN

        int nextWriteBSN
        next BSN to be written to log.

        synchronized by forceManagerLock

      • prevWriteTOD

        long prevWriteTOD
        LogBuffer.tod from previous buffer written.

        maintained in force() method. Used to check against decrement in TOD field. Added to help investigate BUG 303907

      • buffersWaitingForce

        int buffersWaitingForce
        number of buffers waiting to be forced.

        synchronized by bufferManagerLock.

        incremented in put() and decremented in releaseBuffer(). When a thread calls put() with sync parameter set true, and buffersWaitingForce is also zero, then put() causes the buffer to be forced immediately. This strategy minimizes latency in situations of low load, such as a single thread running.

      • lastForceBSN

        int lastForceBSN
        last BSN forced to log.

        synchronized by forceManagerLock

      • forceCount

        private long forceCount
        number of times channel.force() called.

      • writeCount

        private long writeCount
        number of times channel.write() called.

      • minBuffersForced

        private int minBuffersForced
        minimum number of buffers forced by channel.force().

      • maxBuffersForced

        private int maxBuffersForced
        maximum number of buffers forced by channel.force()

      • totalForceTime

        private long totalForceTime
        total amount of time spent in channel.force();

      • totalWriteTime

        private long totalWriteTime
        total amount of time spent in channel.write();

      • maxWriteTime

        private long maxWriteTime
        maximum time (ms) for any single write

      • totalWaitForWriteLockTime

        private long totalWaitForWriteLockTime
        total amount of time (ms) spent waiting for the forceMangerLock

      • totalTimeBetweenForce

        private long totalTimeBetweenForce
        total time between channel.force() calls

      • minTimeBetweenForce

        private long minTimeBetweenForce

      • maxTimeBetweenForce

        private long maxTimeBetweenForce

      • lastForceTOD

        private long lastForceTOD
        time of last force used to compute totalTimeBetweenForce

      • threadsWaitingForce

        private int threadsWaitingForce
        number of threads waiting for a force

      • maxThreadsWaitingForce

        private int maxThreadsWaitingForce

      • totalThreadsWaitingForce

        private long totalThreadsWaitingForce

      • threadsWaitingForceThreshold

        private int threadsWaitingForceThreshold

      • forceOnTimeout

        long forceOnTimeout

      • forceNoWaitingThreads

        long forceNoWaitingThreads

      • forceHalfOfBuffers

        long forceHalfOfBuffers

      • forceMaxWaitingThreads

        long forceMaxWaitingThreads

      • forceOnFileSwitch

        long forceOnFileSwitch

      • flushManagerName

        private static final java.lang.String flushManagerName
        name of flush manager thread
        See Also:
        Constant Field Values

      • forceQueue

        private LogBuffer[] forceQueue
        queue of buffers waiting to be written. The queue guarantees that buffers are written to disk in BSN order. Buffers are placed into the forceQueue using the fqPut workerID, and removed from the forceQueue using the fqGet workerID. Access to these two workerID members is synchronized using separate objects to allow most threads to be storing log records while a single thread is blocked waiting for a physical force.

        Buffers are added to the queue when put() detects the buffer is full, and when the FlushManager thread detects that a buffer has waited too long to be written. The fqPut member is the workerID of the next location in forceQueue to put a LogBuffer that is to be written. fqPut is protected by bufferManagerLock .

        Buffers are removed from the queue in force() and written to disk. The fqGet member is an workerID to the next buffer to remove from the forceQueue. fqGet is protected by forceManagerLock .

        The size of forceQueue[] is one larger than the size of freeBuffer[] so that fqPut == fqGet always means the queue is empty.

      • fqPut

        private int fqPut
        next put workerID into forceQueue .

        synchronized by bufferManagerLock.

      • fqGet

        private int fqGet
        next get workerID from forceQueue .

        synchronized by forceManagerLock.

    • Constructor Detail

      • LogBufferManager

        LogBufferManager​(Configuration config)
        Parameters:
        config - Configuration object

    • Method Detail

      • elapsedTime

        final long elapsedTime​(long startTime)
        compute elapsed time for an event
        Parameters:
        startTime - time event began
        Returns:
        elapsed time (System.currentTimeMillis() - startTime)

      • force

        private void force​(boolean timeout)
            throws java.io.IOException,
                   java.lang.InterruptedException
        forces buffer to disk.

        batches multiple buffers into a single force when possible.

        Design Note:
        It was suggested that using forceManagerLock to control writes from the forceQueue[] and forces would reduce overlap due to the amount of time that forceManagerLock is shut while channel.force() is active.

        Experimented with using two separate locks to manage the channel.write() and the channel.force() calls, but it appears that thread calling channel.force() will block another thread trying to call channel.write() so both locks end up being shut anyway. Since two locks did not provide any measurable benefit, it seems best to use a single forceManagerLock to keep the code simple.

        Throws:
        java.io.IOException
        java.lang.InterruptedException

      • sync

        private void sync​(LogBuffer logBuffer)
           throws java.io.IOException,
                  java.lang.InterruptedException
        Waits for logBuffer to be forced to disk.

        No monitors are owned when routine is entered.

        Prior to calling sync(), the thread called put() with sync param set true to register the fact that the thread would wait for the force.

        Throws:
        java.io.IOException
        java.lang.InterruptedException

      • releaseBuffer

        private void releaseBuffer​(LogBuffer buffer)
        decrements count of threads waiting on this buffer.

        If count goes to zero, buffer is returned to the freeBuffer list, and any threads waiting for a free buffer are notified.

        Parameters:
        buffer - LogBuffer to be released
        See Also:
        buffersWaitingForce

      • getLogBuffer

        LogBuffer getLogBuffer​(int index)
                throws java.lang.ClassNotFoundException
        return a new instance of LogBuffer.

        Actual LogBuffer implementation class is specified by configuration.

        Returns:
        a new instance of LogBuffer
        Throws:
        java.lang.ClassNotFoundException

      • fqAdd

        void fqAdd​(LogBuffer buffer)
        Add a buffer to the forceQueue.

        PRECONDITION: bufferManagerLock owned by caller

        Parameters:
        buffer - LogBuffer to be added to the forceQueue

      • put

        long put​(short type,
         byte[][] data,
         boolean sync)
  throws LogRecordSizeException,
         LogFileOverflowException,
         java.lang.InterruptedException,
         java.io.IOException
        writes data byte[][] to log and returns a log key.

        waits for IO to complete if sync is true.

        MG 27/Jan/05 modified code to force buffer if caller has set sync == true, and there are no buffers waiting to be written. This causes buffers to be written immediately in a single threaded and/or low volume situation. Change suggested by developers at ApacheCon and at ObjectWebCon. This feature is disabled by default and is enabled by setting the log configuration property XXX to true.

        Returns:
        token reference (log key) for record just written
        Throws:
        LogRecordSizeException - when size of byte[] is larger than the maximum possible record for the configured buffer size.
        LogFileOverflowException
        java.lang.InterruptedException
        java.io.IOException
        See Also:
        buffersWaitingForce

      • forceCurrentBuffer

        void forceCurrentBuffer()
                 throws java.io.IOException
        Force the current buffer to disk before starting a replay().
        Throws:
        java.io.IOException

      • replay

        void replay​(ReplayListener listener,
            long mark,
            boolean replayCtrlRecords)
     throws LogConfigurationException,
            InvalidLogKeyException
        Replays log from requested mark forward to end of log.

        Blocks caller until replay completes due to end of log, or an exception is passed to listener.onError().

        Parameters:
        listener - ReplayListener to receive notifications for each log record.
        mark - log key for the first record to be replayed.

        If mark is zero then the entire active log is replayed.

        replayCtrlRecords - indicates whether to return control records.

        used by utility routines such as CopyLog.

        Throws:
        InvalidLogKeyException - if the requested key is not found in the log.
        LogConfigurationException
        See Also:
        Logger.replay(ReplayListener, long)

      • open

        void open()
   throws java.lang.ClassNotFoundException
        Allocate pool of IO buffers for Logger.

        The LogBufferManager class is a generalized manager for any type of LogBuffer. The class name for the type of LogBuffer to use is specified by configuration parameters.

        Throws:
        java.lang.ClassNotFoundException - if the configured LogBuffer class cannot be found.

      • close

        void close()
        Shutdown any threads that are started by this LogBufferManager instance.

      • init

        void init​(LogFileManager lfm,
          int bsn)
        perform initialization following reposition of LogFileManager.
        Parameters:
        lfm - LogFileManager used by the buffer manager to obtain log files for writing buffers.
        bsn - last Block Sequence Number written by Logger.

      • flushAll

        void flushAll()
       throws java.io.IOException
        flush active buffers to disk and wait for all LogBuffers to be returned to the freeBuffer pool.

        May be called multiple times.

        Throws:
        java.io.IOException

      • doubleToString

        private java.lang.String doubleToString​(double val,
                                        int decimalPlaces)
        convert a double to String with fixed number of decimal places
        Parameters:
        val - double to be converted
        decimalPlaces - number of decimal places in output
        Returns:
        String result of conversion

      • getStats

        java.lang.String getStats()
        Returns an XML node containing statistics for the LogBufferManager.

        The nested element contains entries for each LogBuffer object in the buffer pool.

        Returns:
        a String containing statistics.

      • bsnFromMark

        int bsnFromMark​(long mark)
        returns the BSN value portion of a log key mark .
        Parameters:
        mark - log key or log mark to extract BSN from.
        Returns:
        BSN portion of mark

      • markFromBsn

        long markFromBsn​(int bsn,
                 int offset)
        generate a log mark (log key).
        Parameters:
        bsn - Block Sequence Number.
        offset - offset within block.

        May be zero to allow access to the beginning of a block.

        Returns:
        a log key.

      • getWaitForBuffer

        public final long getWaitForBuffer()
        provides synchronized access to waitForBuffer
        Returns:
        the current value of waitForBuffer