Skip to content

Conversation

@jack-berg
Copy link
Member

@jack-berg jack-berg commented Dec 8, 2025

Marked as a draft because this depends on corresponding code from opentelemetry-java and because unit tests are needed, but this is functionally complete.

Related PRs:

The java logging ecosystem is fractured, with a variety of competing libraries for recording logs and configuring how they are processed.

For this discussion, I'll use the following terminology:

  • Log API: A library meant to be called by application code and libraries to record / emit log records.
  • Log Bridge: A library that converts from one logging API / SDK to another.
  • Log SDK: A library that processes logs. Common tasks include filtering, formatting / templating, export (file, stream, network location, etc).

Problem

The OpenTelemetry API is not a good participant in the logging ecosystem. For all other tools, you can choose any log SDK, and configure logs from any log API to route to it. However, if you call the OpenTelemetry log API, logs can only be routed to the OpenTelemetry log SDK.

This matrix summarizes how different log APIs are routed to different log SDKs:

OpenTelemetry API SLF4J API JUL API Log4j2 API Log4j1 / Reload4j API
OpenTelemetry SDK
Logback SDK
JUL SDK
Log4j2 SDK ✅ (
Log4j1 / Reload4j SDK

See "Appendix: log ecosystem" for more details.

OpenTelemetry log API calls should be routable to both the OpenTelemetry SDK and any other log SDK. This will clear the path for recommending the OpenTelemetry log API for end users, rather than only bridges.

Proposed solution

Provide capabilities to bridge the OpenTelemetry log SDK to the SLf4J API, while detecting and preventing cycles.

  • SLF4J is the standard logging facade for Java. All the log SDKs have implementations that allow SLF4J calls to be routed to them. If OpenTelemetry log records can be routed to SLF4J, then they can be routed to any log SDK.
  • Cycle prevention is key. Since we also aim to bridge calls from log SDKs with appenders (i.e. Logback and Log4j2 appenders), we're exposed to infinite cycles if we're not care: SLF4J API -> Log4j2 SDK -> OpenTelemetry Log4j2 Appender -> OpenTelemetry Log API -> OpenTelemetry Log SDK -> SLF4J API.
  • The solution looks different based on whether or not the javaagent is installed.
  • With javaagent installed:
    • Use bytecode instrumentation to detect if the application has SLF4J 2+ on classpath. If so, jump through some hoops to allow OpenTelemetryInstaller to add Slf4jBridgeAgentProcessor, a LogRecordProcessor which records logs to the SLF4J in the application classpath.
    • Avoid cycles using CallDepth tracking.
  • Without javaagent installed:
    • Provide a new LogRecordProcessor called Slf4jBridgeProcessor. Users must explicitly configure their OpenTelemetrySdk to use this processor.
    • This processor bridges logs from OpenTelemetrySdk to SLF4J, and uses Context to avoid cycles.

See "Appendix: call paths" for details about how code flows in various scenarios.

Demonstrations

To demonstrate the proposed solution, I've created two new demos in opentelemetry-java-examples: open-telemetry/opentelemetry-java-examples#954

Each records logs from the various log APIs (OpenTelemetry log API, SLF4J API, JUL API, Log4j2 API, Log4j1 API) and is configured to export to OTLP using the OpenTelemetry SDK, and to the console using a traditional log SDK.

Each can be run with or without the javaagent.

  • Log4j2 configures the application to use the log4j2 SDK
    • Log4j2 is the SLF4J implementation via log4j2-slf4j-impl
    • Log4j1 API is rerouted to SLF4J via log4j-over-slf4j
    • JUL is rerouted to SLF4J via jul-to-slf4j
    • OpenTelemetry log SDK is rerouted to SLF4J via Slf4jBridgeAgentProcessor / Slf4jBridgeProcessor
    • Log4j2 SDK is rerouted to OpenTelemetry log API via opentelemetry-log4j-appender-2.17
  • Logback configures the application to use the logback SDK
    • Logback is the SLF4J implementation via log4j2-classic
    • Log4j2 is rerouted to SLF4J implementation via log4j-to-slf4j
    • Log4j1 API is rerouted to SLF4J via log4j-over-slf4j
    • JUL is rerouted to SLF4J via jul-to-slf4j
    • OpenTelemetry log SDK is rerouted to SLF4J via Slf4jBridgeAgentProcessor / Slf4jBridgeProcessor
    • Logback SDK is rerouted to OpenTelemetry log API via opentelemetry-logback-appender-1.0

Open questions

  • How to reuse SLF4J bridge code agent and non-agent cases? We want the bridge code to be consistent but in the current proposal we maintain two copies.
  • Should otel.loopback context be a java specific or a spec level concept reserving one of the bits of LogRecord.flags? Its fine to start as a java specific concept and promote to spec later.

Appendix: log ecosystem

Popular log APIs in the Java ecosystem.
Name Maven coordinates Description Usage
OpenTelemetry API io.opentelemetry:opentelemetry-api OpenTelemetry log APIs, along with metrics and traces opentelemetry.getLogsBridge().get("my-logger").logRecordBuilder().setSeverity(INFO).setBody("Hello world").emit();
SLF4J API org.slf4j:slf4j-api Standard logging facade for java API LoggerFactory.getLogger("my-logger").info("Hello world");
JUL API built into java since 1.4 JUL - Java utility logger Logger.getLogger("my-logger").info("Hello world");
Log4j2 API org.apache.logging.log4j:log4j-api Log4j2 API LogManager.getLogger("my-logger").info("Hello world");
Log4j1 API / SDK log4j:log4j Log4j1 API / SDK (EOL) Logger.getLogger("my-logger").info("Hello world");
Reload4j API / SDK ch.qos.reload4j:reload4j:1.2.26 Reload4j API / SDK - drop in replacement for Log4j1 Logger.getLogger("my-logger").info("Hello world"); (identical API as Log4j1)
Bridges to convert from one log API / SDK tool to another.
Source Target Maven coordinates Description Usage
Log4j2 SDK OpenTelemetry API io.opentelemetry.instrumentation:opentelemetry-log4j-appender-2.17 Bridge logs from Log4j2 SDK to OpenTelemetry log API by implementing Appender interface referenced in Log4j2 SDK config Include <AppenderRef ref="OpenTelemetryAppender" /> in log4j2.xml, and call OpenTelemetryAppender.install(OpenTelemetry)
Logback SDK OpenTelemetry API io.opentelemetry.instrumentation:opentelemetry-logback-appender-1.0 Bridge logs from Logback SDK to OpenTelemetry log API by implementing Appender interface referenced in Logback SDK config Include <appender name="OpenTelemetry" class="*.OpenTelemetryAppender" /> in logback.xml, and call OpenTelemetryAppender.install(OpenTelemetry)
Log4j1 API OpenTelemetry API io.opentelemetry.javaagent:opentelemetry-javaagent Bridge logs from Log4j1 API to OpenTelemetry log API using bytecode manipulation Include -javaagent=/path/to/opentelemetry-javaagent.jar
Log4j2 API OpenTelemetry API io.opentelemetry.javaagent:opentelemetry-javaagent Bridge logs from Log4j2 API to OpenTelemetry log API using bytecode manipulation Include -javaagent=/path/to/opentelemetry-javaagent.jar
Logback API OpenTelemetry API io.opentelemetry.javaagent:opentelemetry-javaagent Bridge logs from Logback API to OpenTelemetry log API using bytecode manipulation Include -javaagent=/path/to/opentelemetry-javaagent.jar
JUL API OpenTelemetry API io.opentelemetry.javaagent:opentelemetry-javaagent Bridge logs from JUL API to OpenTelemetry log API using bytecode manipulation Include -javaagent=/path/to/opentelemetry-javaagent.jar
JUL API OpenTelemetry API io.opentelemetry.javaagent:opentelemetry-javaagent Bridge logs from JUL API to OpenTelemetry log API using bytecode manipulation Include -javaagent=/path/to/opentelemetry-javaagent.jar
JUL API SLF4J API org.slf4j:jul-to-slf4j Bridge logs from JUL to SLF4J by adding Handler to JUL root logger Call SLF4JBridgeHandler.removeHandlersForRootLogger(), SLF4JBridgeHandler.install()
Log4j1 API SLF4J API org.slf4j:log4j-over-slf4j Bridge logs from Log4j1 SDK to SLF4J by reimplementing Log4j1 API classes Replace log4j.jar with log4j-over-slf4j.jar on classpath
Log4j2 API SLF4J API org.apache.logging.log4j:log4j-to-slf4j Bridge logs from Log4j2 SDK to SLF4J by implementing the Log42J API Provider interface used to bind the Log4j2 API to an implementation Include log4j-to-slf4j.jar on classpath
SLF4J API JUL SDK org.slf4j:slf4j-jdk14 Bridge logs from SLF4J to JUL by implementing SLF4J API SLF4JServiceProvider interface used to bind SLF4J API to an implementation Include slf4j-jdk14.jar on classpath
SLF4J API Reload4j SDK / Log4j1 SDK org.slf4j:slf4j-reload4j Bridge logs from SLF4J to Reload4j or Log4j1 by implementing SLF4J API SLF4JServiceProvider interface used to bind SLF4J API to an implementation Include slf4j-reload4j.jar on classpath
SLF4J API Logback SDK ch.qos.logback:logback-classic Bridge logs from SLF4J to Logback by implementing SLF4J API SLF4JServiceProvider interface used to bind SLF4J API to an implementation Include logback-classic.jar on classpath
SLF4J API Log4j2 SDK org.apache.logging.log4j:log4j-slf4j2-impl (org.apache.logging.log4j:log4j-slf4j-impl for older versions) Bridge logs from SLF4J to Log4j2 by implementing SLF4J API SLF4JServiceProvider interface used to bind SLF4J API to an implementation Include log4j-slf4j2-impl.jar on classpath

|

Popular log SDKs in the Java ecosystem.
Name Maven coordinates Description
OpenTelemetry SDK io.opentelemetry:opentelemetry-sdk OpenTelemetry log SDK, along with metrics and traces
Logback SDK ch.qos.logback:logback-core Logback SDK
Log4j2 SDK org.apache.logging.log4j:log4j-core Log4j2 SDK
Log4j1 API / SDK log4j:log4j Log4j1 API / SDK (EOL)
Reload4j API / SDK ch.qos.reload4j:reload4j:1.2.26 Reload4j API / SDK - drop in replacement for Log4j1
JUL SDK built into java since 1.4 JUL SDK
Summary of how log APIs are routed to log SDKs.
OpenTelemetry API SLF4J API JUL API Log4j2 API Log4j1 / Reload4j API
OpenTelemetry SDK ✅ (opentelemetry-sdk) ✅ if log4j2 / logback SDK (logback-classic, opentelemetry-logback-appender-1.0 OR log4j-slf4j-impl, opentelemetry-log4j-appender-2.17) ✅ if log4j2 / logback SDK (jul-to-slf4j + SLF4J API -> OpenTelemetry SDK), or otel javaagent ✅ (opentelemetry-log4j-appender-2.17), or otel javaagent ✅ if log4j2 / logback SDK (log4j-over-slf4j + SLF4J API -> OpenTelemetry SDK), or otel javaagent
Logback SDK ✅ (logback-classic) ✅ (jul-to-slf4j, logback-classic) ✅ (log4j-to-slf4j, logback-classic) ✅ (log4j-over-slf4j, logback-classic)
JUL SDK ✅ (slf4j-jdk14) ✅ (built in) ✅ (log4j-to-slf4j, slf4j-jdk14) ✅ (log4j-over-slf4j, slf4j-jdk14)
Log4j2 SDK ✅ (log4j-slf4j2-impl) ✅ (jul-to-slf4j, log4j-slf4j2-impl) ✅ (log4j-core) ✅ (log4j-over-slf4j, log4j-slf4j2-impl)
Log4j1 / Reload4j SDK ✅ (slf4j-reload4j) ✅ (jul-to-slf4j, slf4j-reload4j) ✅ (log4j-to-slf4j, slf4j-reload4j) ✅ (built in)

Appendix: call paths

In all cases we want the log record to be exported by the OpenTelemetry SDK va OTLP, and to the console appender.

Click to expand

No javaaagent, SLF4J API called by user:

  • user calls slf4j-api
    • calls log4j-slf4j-impl
      • calls log4j-core
        • calls opentelemetry-log4j-appender
          • calls opentelemetry-api w/ otel.loopback=true
            • calls opentelemetry-sdk
              • calls opentelemetry-exporter-otlp
              • calls Slf4jBridgeProcessor
                • calls slf4j-api terminates call w/ otel.loopback=true
        • calls log4j-core console appender ✅

No javaaagent, JUL API called by user:

  • user calls java.util.logging
    • calls jul-to-slf4j
      • calls slf4j-api
        • calls log4j-slf4j-impl
        • calls log4j-core
          • calls opentelemetry-log4j-appender
            • calls opentelemetry-api w/ otel.loopback=true
              • calls opentelemetry-sdk
                • calls opentelemetry-exporter-otlp
                • calls Slf4jBridgeProcessor
                  • calls slf4j-api terminates call w/ otel.loopback=true
          • calls log4j-core console appender ✅

No javaaagent, OpenTelemetry log API called by user:

  • user calls opentelemetry-api
    • calls opentelemetry-sdk
      • calls opentelemetry-exporter-otlp
      • calls Slf4jBridgeProcessor
        • calls slf4j-api w/ otel.loopback=true
          • calls log4j-slf4j-impl
            • calls log4j-core
              • calls opentelemetry-log4j-appender terminates calls w/ otel.loopback=true
              • calls log4j-core console appender ✅

Javaaagent installed, SLF4J API called by user:

  • user calls slf4j-api
    • calls log4j-slf4j-impl
      • calls log4j-core
        • intercepted by log4j-appender-2.17:javaagent
          • calls opentelemetry-api w/ CallDepth(LoggerProvider)=1
            • calls opentelemetry-sdk
              • calls opentelemetry-exporter-otlp
              • calls Slf4jBridgeAgentProcessor
                • calls slf4j-api terminates calls w/ CallDepth(LoggerProvider)!=0)
        • calls log4j-core console appender ✅

Javaaagent installed, JUL API called by user:

  • user calls java.util.logging
    • calls jul-to-slf4j
      • calls slf4j-api
        • calls log4j-slf4j-impl
          • calls log4j-core
            • intercepted by log4j-appender-2.17:javaagent
              • calls opentelemetry-api w/ CallDepth(LoggerProvider)=1
                • calls opentelemetry-sdk
                  • calls opentelemetry-exporter-otlp
                  • calls Slf4jBridgeAgentProcessor
                    • calls slf4j-api terminates calls w/ CallDepth(LoggerProvider)!=0)
            • calls log4j-core console appender ✅

Javaaagent installed, OpenTelemetry log API called by user:

  • user calls opentelemetry-api
    • intercepted by opentelemetry-api:javaagent
      • calls opentelemetry-sdk
        • calls opentelemetry-exporter-otlp
        • calls Slf4jBridgeAgentProcessor
          • calls slf4j-api w/ CallDepth(LoggerProvider)=1
            • calls log4j-slf4j-impl
              • calls log4j-core
                • intercepted by log4j-appender-2.17:javaagent terminates calls w/ CallDepth(LoggerProvider)!=0
                • calls log4j-core console appender ✅

Appendix: key links

Click to expand

@github-actions github-actions bot added the test native This label can be applied to PRs to trigger them to run native tests label Dec 8, 2025
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is all unrelated code to bridge the new GlobalOpenTelemetry.isSet(), which makes it easier to demonstrate how this works because the examples can easily toggle with agent installed and not installed. Decent starting point since we'll probably want this in the next release.

@Advice.OnMethodExit(suppress = Throwable.class)
public static void onExit() {
if (Slf4jBridgeInstallerFlags.IS_INSTALLED.compareAndSet(false, true)) {
Slf4jLogRecorderImpl.install();
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's happening here:

  • This instrumentation has access to the slf4j-api from the application classpath
  • The OpenTelemetryInstaller from javaagent-tooling needs to register a LogRecordProcessor which calls routs to the application classpath slf4j-api
  • So we establish a common minimal interface Slf4jLogRecorder in javaagent-bootstrap which can be accessed from both the instrumentation and the OpenTelemetryInstaller
  • Each time the LogRecordProcessor#onEmit is called, it calls Slf4jLogRecorderHolder.get() to get a reference to a Slf4jLogRecorder and record the log.
  • This Slf4jLogRecorder instance starts as a noop, but this instrumentation here detects the first usage of slf4j-api in the application, and installs calls Slf4jLogRecordHolder.initialize(Slf4jLogRecordHolder) with a proper instance connected to the slf4j-api in the application classpath.

Voila!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If / when we agree on the basic concepts here, I can polish this up and it ready to merge:

  • Better testing
  • Find a way to have one definition mapping from otel to SLF4J. Currently, I maintain two copies of the same code, one here and one in opentelemetry-java for the non agent case.
  • Figure out the configuration story. Currently, the LogRecordProcessor is always added. Should be configurable and need to think about the defaults.
  • And what about declarative config? Declarative config makes it easy to add this processor, but:
    1. We want to this to be easy / discoverable and the starter templates won't include this java specific processor.
    2. I think this is too important to require adopting declarative config as a prerequisite.
  • Maybe the answer is to have a java agent / distribution option like:
    export OTEL_JAVAAGENT_SLF4J_BRIDGE=true
    
    or for delcarative config:
    distribution:
      javaaagent:
        slf4j_bridge_enabled: true
    

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test native This label can be applied to PRs to trigger them to run native tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant