Skip to content

Batch processor: mode support #1570

@jmacd

Description

@jmacd

The batch processor currently forces all of its inputs into the OtapArrowRecords format, batches in terms of OTAP structures, and outputs the same format. For users sending OTLP inputs, batching, and sending OTLP outputs, this will be inefficient. While Arrow batching may yield more compressed batches, this is not the simplest way to batch OTLP data.

Since we have an OTLP bytes representation, we should support batching in the OTLP bytes representation.

Proposing a new mode field in the batch_processor configuration, with values:

  • "otap": current behavior batches in OTAP format (default)
  • "preserve": new behavior batches in the arriving format
  • "otlp": new behavior batches in OTLP format

The new options will enable batching OTLP bytes directly. Fortunately, batching OTLP bytes is trivial, we simply concatenate the arrays. In the OTLP batching mode we will not support splitting, only merging.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions