-
Notifications
You must be signed in to change notification settings - Fork 65
Open
Description
The batch processor currently forces all of its inputs into the OtapArrowRecords format, batches in terms of OTAP structures, and outputs the same format. For users sending OTLP inputs, batching, and sending OTLP outputs, this will be inefficient. While Arrow batching may yield more compressed batches, this is not the simplest way to batch OTLP data.
Since we have an OTLP bytes representation, we should support batching in the OTLP bytes representation.
Proposing a new mode field in the batch_processor configuration, with values:
- "otap": current behavior batches in OTAP format (default)
- "preserve": new behavior batches in the arriving format
- "otlp": new behavior batches in OTLP format
The new options will enable batching OTLP bytes directly. Fortunately, batching OTLP bytes is trivial, we simply concatenate the arrays. In the OTLP batching mode we will not support splitting, only merging.
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
Status
No status