Struct FlatLastNonNull 

pub struct FlatLastNonNull {
    field_column_start: usize,
    filter_deleted: bool,
    buffer: Option<BatchLastRow>,
    contains_delete: bool,
}

Dedup strategy that keeps the last non-null field for the same key.

Fields

field_column_start: usize

The start index of field columns:

filter_deleted: bool

Filter deleted rows; otherwise preserve every delete barrier.

buffer: Option<BatchLastRow>

Buffered batch to check whether the next batch have duplicated rows with this batch. Fields in the last row of this batch may be updated by the next batch. In preserve mode, duplicate keys may be separated by delete markers.

contains_delete: bool

Whether the last row range contains a delete operation. If so, we don’t need to update null fields.

Implementations

impl FlatLastNonNull

pub fn new(field_column_start: usize, filter_deleted: bool) -> Self

Creates a strategy with the given filter_deleted flag. When false, every delete barrier is preserved and Puts are folded only between barriers so later reads cannot fill fields from a deleted row. Inputs for partial merges must form a sequence-contiguous group to avoid folding fields across an intermediate version left outside the merge.

fn dedup_one_batch( batch: RecordBatch, field_column_start: usize, prev_batch_contains_delete: bool, filter_deleted: bool, ) -> Result<(RecordBatch, bool)>

Remove duplications from the batch without considering the previous and next rows. Returns a tuple containing the deduplicated batch and a boolean indicating whether the last range contains deleted rows.

fn dedup_by_partitions( batch: RecordBatch, partitions: &Partitions, field_column_start: usize, op_types: &UInt8Array, first_range_contains_delete: bool, filter_deleted: bool, ) -> Result<(RecordBatch, bool)>

Remove depulications for each partition. Returns a tuple containing the deduplicated batch and a boolean indicating whether the last range contains deleted rows.

fn compute_field_indices( ranges: &[Range<usize>], field_array: &ArrayRef, op_types: &UInt8Array, first_range_contains_delete: bool, ) -> UInt64Array

Returns an array of indices of the latest non null value for each input range. If all values in a range are null, the returned index is unspecific. Stops when encountering a delete operation and ignores all subsequent rows.

fn last_range_has_delete(ranges: &[Range<usize>], op_types: &UInt8Array) -> bool

Checks whether the last range contains a delete operation.

Trait Implementations

impl RecordBatchDedupStrategy for FlatLastNonNull

fn push_batch( &mut self, batch: RecordBatch, metrics: &mut DedupMetrics, ) -> Result<Option<RecordBatch>>

Pushes a batch to the dedup strategy. Returns rows that no longer need to be updated by subsequent input. A strategy for partial merges may retain duplicates separated by Deletes.

fn finish(&mut self, metrics: &mut DedupMetrics) -> Result<Option<RecordBatch>>

Finishes the deduplication process and returns any remaining batch.