Package org.apache.lucene.codecs.simpletext

Class SimpleTextBKDWriter

java.lang.Object

org.apache.lucene.codecs.simpletext.SimpleTextBKDWriter

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

final class SimpleTextBKDWriter
extends java.lang.Object
implements java.io.Closeable

Forked from BKDWriter and simplified/specialized for SimpleText's usage

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private class	SimpleTextBKDWriter.OneDimensionBKDWriter

Field Summary

Fields

Modifier and Type	Field	Description
protected int	bytesPerDim	How many bytes each value in each dimension takes.
private int	bytesPerDoc	How many bytes each docs takes in the fixed-width offline format
static java.lang.String	CODEC_NAME
(package private) int[]	commonPrefixLengths
static float	DEFAULT_MAX_MB_SORT_IN_HEAP	Default maximum heap to use, before spilling to (slower) disk
static int	DEFAULT_MAX_POINTS_IN_LEAF_NODE	Default maximum number of point in each leaf block
protected FixedBitSet	docsSeen
private boolean	finished
static int	MAX_DIMS	Maximum number of dimensions (2 * max index dimensions)
static int	MAX_INDEX_DIMS	Maximum number of dimensions
private int	maxDoc
(package private) double	maxMBSortInHeap
protected byte[]	maxPackedValue	Maximum per-dim values, packed
protected int	maxPointsInLeafNode
private int	maxPointsSortInHeap
protected byte[]	minPackedValue	Minimum per-dim values, packed
protected int	numDataDims	How many dimensions we are storing at the leaf (data) nodes
protected int	numIndexDims	How many dimensions we are indexing in the internal nodes
protected int	packedBytesLength	numDims * bytesPerDim
protected int	packedIndexBytesLength	numIndexDims * bytesPerDim
protected long	pointCount
private PointWriter	pointWriter
(package private) BytesRefBuilder	scratch
(package private) byte[]	scratch1
(package private) byte[]	scratch2
(package private) BytesRef	scratchBytesRef1
(package private) BytesRef	scratchBytesRef2
(package private) byte[]	scratchDiff
(package private) TrackingDirectoryWrapper	tempDir
(package private) java.lang.String	tempFileNamePrefix
private IndexOutput	tempInput
private long	totalPointCount	An upper bound on how many points the caller will add (includes deletions)
static int	VERSION_COMPRESSED_DOC_IDS
static int	VERSION_COMPRESSED_VALUES
static int	VERSION_CURRENT
static int	VERSION_IMPLICIT_SPLIT_DIM_1D
static int	VERSION_START

Constructor Summary

Constructors

Constructor	Description
SimpleTextBKDWriter(int maxDoc, Directory tempDir, java.lang.String tempFileNamePrefix, int numDataDims, int numIndexDims, int bytesPerDim, int maxPointsInLeafNode, double maxMBSortInHeap, long totalPointCount)

Method Summary

Modifier and Type	Method	Description
void	add(byte[] packedValue, int docID)
private void	build(int nodeID, int leafNodeOffset, MutablePointValues reader, int from, int to, IndexOutput out, byte[] minPackedValue, byte[] maxPackedValue, byte[] splitPackedValues, long[] leafBlockFPs, int[] spareDocIds)
private void	build(int nodeID, int leafNodeOffset, BKDRadixSelector.PathSlice points, IndexOutput out, BKDRadixSelector radixSelector, byte[] minPackedValue, byte[] maxPackedValue, byte[] splitPackedValues, long[] leafBlockFPs, int[] spareDocIds)	The array (sized numDims) of PathSlice describe the cell we have currently recursed to.
private void	checkMaxLeafNodeCount(int numLeaves)
void	close()
private void	computeCommonPrefixLength(HeapPointWriter heapPointWriter, byte[] commonPrefix)
long	finish(IndexOutput out)	Writes the BKD tree to the provided IndexOutput and returns the file offset where index was written.
long	getPointCount()	How many points have been added so far
private void	newline(IndexOutput out)
private void	rotateToTree(int nodeID, int offset, int count, byte[] index, java.util.List<byte[]> leafBlockStartValues)
private static int	runLen(java.util.function.IntFunction<BytesRef> packedValues, int start, int end, int byteOffset)
protected int	split(byte[] minPackedValue, byte[] maxPackedValue)
private HeapPointWriter	switchToHeap(PointWriter source)	Pull a partition back into heap once the point count is low enough while recursing.
private boolean	valueInBounds(BytesRef packedValue, byte[] minPackedValue, byte[] maxPackedValue)	Called only in assert
private boolean	valueInOrder(long ord, int sortedDim, byte[] lastPackedValue, byte[] packedValue, int packedValueOffset, int doc, int lastDoc)
private boolean	valuesInOrderAndBounds(int count, int sortedDim, byte[] minPackedValue, byte[] maxPackedValue, java.util.function.IntFunction<BytesRef> values, int[] docs, int docsOffset)
private java.lang.Error	verifyChecksum(java.lang.Throwable priorException, PointWriter writer)	Called on exception, to check whether the checksum is also corrupt in this source, and add that information (checksum matched or didn't) as a suppressed exception.
static void	verifyParams(int numDims, int numIndexDims, int maxPointsInLeafNode, double maxMBSortInHeap, long totalPointCount)
private void	write(IndexOutput out, java.lang.String s)
private void	write(IndexOutput out, BytesRef b)
long	writeField(IndexOutput out, java.lang.String fieldName, MutablePointValues reader)	Write a field from a MutablePointValues.
private long	writeField1Dim(IndexOutput out, java.lang.String fieldName, MutablePointValues reader)
private long	writeFieldNDims(IndexOutput out, java.lang.String fieldName, MutablePointValues values)
private void	writeIndex(IndexOutput out, long[] leafBlockFPs, byte[] splitPackedValues)	Subclass can change how it writes the index.
private void	writeInt(IndexOutput out, int x)
protected void	writeLeafBlockDocs(IndexOutput out, int[] docIDs, int start, int count)
protected void	writeLeafBlockPackedValues(IndexOutput out, int[] commonPrefixLengths, int count, int sortedDim, java.util.function.IntFunction<BytesRef> packedValues)
private void	writeLeafBlockPackedValuesRange(IndexOutput out, int[] commonPrefixLengths, int start, int end, java.util.function.IntFunction<BytesRef> packedValues)
private void	writeLong(IndexOutput out, long x)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CODEC_NAME

public static final java.lang.String CODEC_NAME

See Also:

Constant Field Values

VERSION_START

public static final int VERSION_START

See Also:

Constant Field Values

VERSION_COMPRESSED_DOC_IDS

public static final int VERSION_COMPRESSED_DOC_IDS

See Also:

Constant Field Values

VERSION_COMPRESSED_VALUES

public static final int VERSION_COMPRESSED_VALUES

See Also:

Constant Field Values

VERSION_IMPLICIT_SPLIT_DIM_1D

public static final int VERSION_IMPLICIT_SPLIT_DIM_1D

See Also:

Constant Field Values

VERSION_CURRENT

public static final int VERSION_CURRENT

See Also:

Constant Field Values

bytesPerDoc

private final int bytesPerDoc

How many bytes each docs takes in the fixed-width offline format

DEFAULT_MAX_POINTS_IN_LEAF_NODE

public static final int DEFAULT_MAX_POINTS_IN_LEAF_NODE

Default maximum number of point in each leaf block

See Also:

Constant Field Values

DEFAULT_MAX_MB_SORT_IN_HEAP

public static final float DEFAULT_MAX_MB_SORT_IN_HEAP

Default maximum heap to use, before spilling to (slower) disk

See Also:

Constant Field Values

MAX_DIMS

public static final int MAX_DIMS

Maximum number of dimensions (2 * max index dimensions)

See Also:

Constant Field Values

MAX_INDEX_DIMS

public static final int MAX_INDEX_DIMS

Maximum number of dimensions

See Also:

Constant Field Values

numDataDims

protected final int numDataDims

How many dimensions we are storing at the leaf (data) nodes

numIndexDims

protected final int numIndexDims

How many dimensions we are indexing in the internal nodes

bytesPerDim

protected final int bytesPerDim

How many bytes each value in each dimension takes.

packedBytesLength

protected final int packedBytesLength

numDims * bytesPerDim

packedIndexBytesLength

protected final int packedIndexBytesLength

numIndexDims * bytesPerDim

scratch

final BytesRefBuilder scratch

tempDir

final TrackingDirectoryWrapper tempDir

tempFileNamePrefix

final java.lang.String tempFileNamePrefix

maxMBSortInHeap

final double maxMBSortInHeap

scratchDiff

final byte[] scratchDiff

scratch1

final byte[] scratch1

scratch2

final byte[] scratch2

scratchBytesRef1

final BytesRef scratchBytesRef1

scratchBytesRef2

final BytesRef scratchBytesRef2

commonPrefixLengths

final int[] commonPrefixLengths

docsSeen

protected final FixedBitSet docsSeen

pointWriter

private PointWriter pointWriter

finished

private boolean finished

tempInput

private IndexOutput tempInput

maxPointsInLeafNode

protected final int maxPointsInLeafNode

maxPointsSortInHeap

private final int maxPointsSortInHeap

minPackedValue

protected final byte[] minPackedValue

Minimum per-dim values, packed

maxPackedValue

protected final byte[] maxPackedValue

Maximum per-dim values, packed

pointCount

protected long pointCount

totalPointCount

private final long totalPointCount

An upper bound on how many points the caller will add (includes deletions)

maxDoc

private final int maxDoc

Constructor Detail

SimpleTextBKDWriter

public SimpleTextBKDWriter(int maxDoc,
                           Directory tempDir,
                           java.lang.String tempFileNamePrefix,
                           int numDataDims,
                           int numIndexDims,
                           int bytesPerDim,
                           int maxPointsInLeafNode,
                           double maxMBSortInHeap,
                           long totalPointCount)
                    throws java.io.IOException

Throws:

java.io.IOException

Method Detail

verifyParams

public static void verifyParams(int numDims,
                                int numIndexDims,
                                int maxPointsInLeafNode,
                                double maxMBSortInHeap,
                                long totalPointCount)

add

public void add(byte[] packedValue,
                int docID)
         throws java.io.IOException

Throws:

java.io.IOException

getPointCount

public long getPointCount()

How many points have been added so far

writeField

public long writeField(IndexOutput out,
                       java.lang.String fieldName,
                       MutablePointValues reader)
                throws java.io.IOException

Write a field from a MutablePointValues. This way of writing points is faster than regular writes with BKDWriter.add(byte[], int) since there is opportunity for reordering points before writing them to disk. This method does not use transient disk in order to reorder points.

Throws:

java.io.IOException

writeFieldNDims

private long writeFieldNDims(IndexOutput out,
                             java.lang.String fieldName,
                             MutablePointValues values)
                      throws java.io.IOException

Throws:

java.io.IOException

writeField1Dim

private long writeField1Dim(IndexOutput out,
                            java.lang.String fieldName,
                            MutablePointValues reader)
                     throws java.io.IOException

Throws:

java.io.IOException

rotateToTree

private void rotateToTree(int nodeID,
                          int offset,
                          int count,
                          byte[] index,
                          java.util.List<byte[]> leafBlockStartValues)

checkMaxLeafNodeCount

private void checkMaxLeafNodeCount(int numLeaves)

finish

public long finish(IndexOutput out)
            throws java.io.IOException

Writes the BKD tree to the provided IndexOutput and returns the file offset where index was written.

Throws:

java.io.IOException

writeIndex

private void writeIndex(IndexOutput out,
                        long[] leafBlockFPs,
                        byte[] splitPackedValues)
                 throws java.io.IOException

Subclass can change how it writes the index.

Throws:

java.io.IOException

writeLeafBlockDocs

protected void writeLeafBlockDocs(IndexOutput out,
                                  int[] docIDs,
                                  int start,
                                  int count)
                           throws java.io.IOException

Throws:

java.io.IOException

writeLeafBlockPackedValues

protected void writeLeafBlockPackedValues(IndexOutput out,
                                          int[] commonPrefixLengths,
                                          int count,
                                          int sortedDim,
                                          java.util.function.IntFunction<BytesRef> packedValues)
                                   throws java.io.IOException

Throws:

java.io.IOException

writeLeafBlockPackedValuesRange

private void writeLeafBlockPackedValuesRange(IndexOutput out,
                                             int[] commonPrefixLengths,
                                             int start,
                                             int end,
                                             java.util.function.IntFunction<BytesRef> packedValues)
                                      throws java.io.IOException

Throws:

java.io.IOException

runLen

private static int runLen(java.util.function.IntFunction<BytesRef> packedValues,
                          int start,
                          int end,
                          int byteOffset)

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable

Throws:

java.io.IOException

verifyChecksum

private java.lang.Error verifyChecksum(java.lang.Throwable priorException,
                                       PointWriter writer)
                                throws java.io.IOException

Called on exception, to check whether the checksum is also corrupt in this source, and add that information (checksum matched or didn't) as a suppressed exception.

Throws:

java.io.IOException

valueInBounds

private boolean valueInBounds(BytesRef packedValue,
                              byte[] minPackedValue,
                              byte[] maxPackedValue)

Called only in assert

split

protected int split(byte[] minPackedValue,
                    byte[] maxPackedValue)

switchToHeap

private HeapPointWriter switchToHeap(PointWriter source)
                              throws java.io.IOException

Pull a partition back into heap once the point count is low enough while recursing.

Throws:

java.io.IOException

build

private void build(int nodeID,
                   int leafNodeOffset,
                   MutablePointValues reader,
                   int from,
                   int to,
                   IndexOutput out,
                   byte[] minPackedValue,
                   byte[] maxPackedValue,
                   byte[] splitPackedValues,
                   long[] leafBlockFPs,
                   int[] spareDocIds)
            throws java.io.IOException

Throws:

java.io.IOException

build

private void build(int nodeID,
                   int leafNodeOffset,
                   BKDRadixSelector.PathSlice points,
                   IndexOutput out,
                   BKDRadixSelector radixSelector,
                   byte[] minPackedValue,
                   byte[] maxPackedValue,
                   byte[] splitPackedValues,
                   long[] leafBlockFPs,
                   int[] spareDocIds)
            throws java.io.IOException

The array (sized numDims) of PathSlice describe the cell we have currently recursed to.

Throws:

java.io.IOException

computeCommonPrefixLength

private void computeCommonPrefixLength(HeapPointWriter heapPointWriter,
                                       byte[] commonPrefix)

valuesInOrderAndBounds

private boolean valuesInOrderAndBounds(int count,
                                       int sortedDim,
                                       byte[] minPackedValue,
                                       byte[] maxPackedValue,
                                       java.util.function.IntFunction<BytesRef> values,
                                       int[] docs,
                                       int docsOffset)
                                throws java.io.IOException

Throws:

java.io.IOException

valueInOrder

private boolean valueInOrder(long ord,
                             int sortedDim,
                             byte[] lastPackedValue,
                             byte[] packedValue,
                             int packedValueOffset,
                             int doc,
                             int lastDoc)

write

private void write(IndexOutput out,
                   java.lang.String s)
            throws java.io.IOException

Throws:

java.io.IOException

writeInt

private void writeInt(IndexOutput out,
                      int x)
               throws java.io.IOException

Throws:

java.io.IOException

writeLong

private void writeLong(IndexOutput out,
                       long x)
                throws java.io.IOException

Throws:

java.io.IOException

write

private void write(IndexOutput out,
                   BytesRef b)
            throws java.io.IOException

Throws:

java.io.IOException

newline

private void newline(IndexOutput out)
              throws java.io.IOException

Throws:

java.io.IOException