OTAP dataflow logging SDK requirements

[jmacd]

From a review thread about debug and info-level logging.

We need to adopt a logging SDK for the OTAP dataflow engine to make our code base more accessible.

As an OpenTelemetry project, I think we should defer to the best practice recommended by OpenTelemetry-Rust.

While we have justified the use of a special metric SDK internally, the justification used ("multivariate", in particular) do not apply to logs and spans. We need to plan for emitting logs and/or spans in this code base, and this is an issue to discuss requirements.

The tokio tracing crate is recommended by OpenTelemetry-Rust. We believe it is compatible with our thread-per-core design, so long as the SDK is configured in a per-thread way.

Are there requirements to avoid Sync and Send traits?

This raises questions about contextual enrichment the way an OpenTelemetry logging SDK might do. While the otap-dataflow engine has a Context type where we could propagate the OTel span context, this would likely have to be introduced in the otap_df_otap::pdata::Context before we could benefit from pipeline-level tracing.

[cijothomas]

To get us started with something (instead of println! statements!), I propose to do the following:

1. Introduce logging macros, similar to what OTel Rust does. (https://github.com/open-telemetry/opentelemetry-rust/blob/main/opentelemetry/src/global/internal_logging.rs)
2. This will use tracing to begin with, but free to change as we learn more and get more feedback.
3. Engine can read a simple config for logging (min_severity_level and destination - only stdout to start with), and use this to setup a fmt subscriber from tracing that outputs logs/events to stdout. To begin with, I'll set this up as global affecting entire process. In a future iteration, we can try doing this on a per thread basis (after measuring the impact on perf/contention etc.)

This will give us a basic picture of how logging would work. In a future iteration, we can discuss leveraging OTel SDK and output the logs via OTLP or elsewhere (like OS_native facilities like etw/linux trace points).

I'll send a PR with the above shortly to facilitate easier discussion.

(Hoping to get some basic logging in, before folks start holidays!)

[lquerel]

What matters most to me, in order of priority:

• minimal impact on the engine's performance and adherence to the thread-per-core and NUMA-aware design
• options to configure the log and trace reporting level and to disable them entirely if needed, ideally either globally or per pipeline instance

[cijothomas]

• minimal impact on the engine's performance and adherence to the thread-per-core and NUMA-aware design

tracing itself is no-op by default. It is the subscribers can do things with the events (like sending to stdout or file or opentelemetry/otlp etc.). We need to write our own custom subscriber that does per-thread aggregation, and flush_periodically (similar to the metrics crate we have now.)
We have written such things for etw/user_events, and based on our testing, they have no contention, and is nicely scaling with number of CPU cores.
https://github.com/open-telemetry/opentelemetry-rust-contrib/blob/main/stress/src/etw_logs.rs#L11-L14
We should be able to make them NUMA-aware (my knowledge is limited here, so have to explore)

• options to configure the log and trace reporting level and to disable them entirely if needed, ideally either globally or per pipeline instance

This is totally possible (yes to global or per pipeline/thread etc.). And the cost of not-enabled is very low/zero. (https://github.com/open-telemetry/opentelemetry-rust/blob/main/opentelemetry-appender-tracing/benches/logs.rs#L9). It is also possible to disable some levels via compile time flags achieving truly zero cost.

tracing uses a lot of statics, and I opened #1575 to track if this will cause concerns.

[lalitb]

We should be able to make them NUMA-aware (my knowledge is limited here, so have to explore)

For a NUMA-aware setup, each collector thread pinned to a specific core should create its own LoggerProvider. That basically means every core gets its own little otel-rust pipeline: logger to processor to exporter. And everything stays local to that core - buffers, atomics, reference counts - so we don’t end up paying for cross-NUMA hops or fighting over cache lines.

The cons is we burn more resources. Every core ends up with its own export thread, its own buffers, and if using OTLP, its own client connection.

[cijothomas]

We should be able to make them NUMA-aware (my knowledge is limited here, so have to explore)

For a NUMA-aware setup, each collector thread pinned to a specific core should create its own LoggerProvider. That basically means every core gets its own little otel-rust pipeline: logger to processor to exporter. And everything stays local to that core - buffers, atomics, reference counts - so we don’t end up paying for cross-NUMA hops or fighting over cache lines.

The cons is we burn more resources. Every core ends up with its own export thread, its own buffers, and if using OTLP, its own client connection.

We should be able to make it happen without creating N LoggerProvider. We can do something similar to how metrics are flowing today - each thread accumulates logs into its own buffer, and occasionally push them to the global one, where there is a OTel SDK listening. Will try to prototype this and share.