-
Notifications
You must be signed in to change notification settings - Fork 1k
Proposal for making OpenTelemetry log API calls accessible in other log SDKs #15572
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is all unrelated code to bridge the new GlobalOpenTelemetry.isSet(), which makes it easier to demonstrate how this works because the examples can easily toggle with agent installed and not installed. Decent starting point since we'll probably want this in the next release.
| @Advice.OnMethodExit(suppress = Throwable.class) | ||
| public static void onExit() { | ||
| if (Slf4jBridgeInstallerFlags.IS_INSTALLED.compareAndSet(false, true)) { | ||
| Slf4jLogRecorderImpl.install(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's happening here:
- This instrumentation has access to the
slf4j-apifrom the application classpath - The
OpenTelemetryInstallerfromjavaagent-toolingneeds to register aLogRecordProcessorwhich calls routs to the application classpathslf4j-api - So we establish a common minimal interface
Slf4jLogRecorderinjavaagent-bootstrapwhich can be accessed from both the instrumentation and theOpenTelemetryInstaller - Each time the
LogRecordProcessor#onEmitis called, it callsSlf4jLogRecorderHolder.get()to get a reference to aSlf4jLogRecorderand record the log. - This
Slf4jLogRecorderinstance starts as a noop, but this instrumentation here detects the first usage ofslf4j-apiin the application, and installs callsSlf4jLogRecordHolder.initialize(Slf4jLogRecordHolder)with a proper instance connected to theslf4j-apiin the application classpath.
Voila!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If / when we agree on the basic concepts here, I can polish this up and it ready to merge:
- Better testing
- Find a way to have one definition mapping from otel to SLF4J. Currently, I maintain two copies of the same code, one here and one in
opentelemetry-javafor the non agent case. - Figure out the configuration story. Currently, the
LogRecordProcessoris always added. Should be configurable and need to think about the defaults. - And what about declarative config? Declarative config makes it easy to add this processor, but:
- We want to this to be easy / discoverable and the starter templates won't include this java specific processor.
- I think this is too important to require adopting declarative config as a prerequisite.
- Maybe the answer is to have a java agent / distribution option like:
or for delcarative config:
export OTEL_JAVAAGENT_SLF4J_BRIDGE=truedistribution: javaaagent: slf4j_bridge_enabled: true
Marked as a draft because this depends on corresponding code from opentelemetry-java and because unit tests are needed, but this is functionally complete.
Related PRs:
The java logging ecosystem is fractured, with a variety of competing libraries for recording logs and configuring how they are processed.
For this discussion, I'll use the following terminology:
Problem
The OpenTelemetry API is not a good participant in the logging ecosystem. For all other tools, you can choose any log SDK, and configure logs from any log API to route to it. However, if you call the OpenTelemetry log API, logs can only be routed to the OpenTelemetry log SDK.
This matrix summarizes how different log APIs are routed to different log SDKs:
See "Appendix: log ecosystem" for more details.
OpenTelemetry log API calls should be routable to both the OpenTelemetry SDK and any other log SDK. This will clear the path for recommending the OpenTelemetry log API for end users, rather than only bridges.
Proposed solution
Provide capabilities to bridge the OpenTelemetry log SDK to the SLf4J API, while detecting and preventing cycles.
OpenTelemetryInstallerto addSlf4jBridgeAgentProcessor, aLogRecordProcessorwhich records logs to the SLF4J in the application classpath.CallDepthtracking.LogRecordProcessorcalledSlf4jBridgeProcessor. Users must explicitly configure theirOpenTelemetrySdkto use this processor.Contextto avoid cycles.See "Appendix: call paths" for details about how code flows in various scenarios.
Demonstrations
To demonstrate the proposed solution, I've created two new demos in
opentelemetry-java-examples: open-telemetry/opentelemetry-java-examples#954Each records logs from the various log APIs (OpenTelemetry log API, SLF4J API, JUL API, Log4j2 API, Log4j1 API) and is configured to export to OTLP using the OpenTelemetry SDK, and to the console using a traditional log SDK.
Each can be run with or without the javaagent.
log4j2-slf4j-impllog4j-over-slf4jjul-to-slf4jSlf4jBridgeAgentProcessor/Slf4jBridgeProcessoropentelemetry-log4j-appender-2.17log4j2-classiclog4j-to-slf4jlog4j-over-slf4jjul-to-slf4jSlf4jBridgeAgentProcessor/Slf4jBridgeProcessoropentelemetry-logback-appender-1.0Open questions
otel.loopbackcontext be a java specific or a spec level concept reserving one of the bits of LogRecord.flags? Its fine to start as a java specific concept and promote to spec later.Appendix: log ecosystem
Popular log APIs in the Java ecosystem.
io.opentelemetry:opentelemetry-apiopentelemetry.getLogsBridge().get("my-logger").logRecordBuilder().setSeverity(INFO).setBody("Hello world").emit();org.slf4j:slf4j-apiLoggerFactory.getLogger("my-logger").info("Hello world");Logger.getLogger("my-logger").info("Hello world");org.apache.logging.log4j:log4j-apiLogManager.getLogger("my-logger").info("Hello world");log4j:log4jLogger.getLogger("my-logger").info("Hello world");ch.qos.reload4j:reload4j:1.2.26Logger.getLogger("my-logger").info("Hello world");(identical API as Log4j1)Bridges to convert from one log API / SDK tool to another.
io.opentelemetry.instrumentation:opentelemetry-log4j-appender-2.17Appenderinterface referenced in Log4j2 SDK config<AppenderRef ref="OpenTelemetryAppender" />inlog4j2.xml, and callOpenTelemetryAppender.install(OpenTelemetry)io.opentelemetry.instrumentation:opentelemetry-logback-appender-1.0Appenderinterface referenced in Logback SDK config<appender name="OpenTelemetry" class="*.OpenTelemetryAppender" />inlogback.xml, and callOpenTelemetryAppender.install(OpenTelemetry)io.opentelemetry.javaagent:opentelemetry-javaagent-javaagent=/path/to/opentelemetry-javaagent.jario.opentelemetry.javaagent:opentelemetry-javaagent-javaagent=/path/to/opentelemetry-javaagent.jario.opentelemetry.javaagent:opentelemetry-javaagent-javaagent=/path/to/opentelemetry-javaagent.jario.opentelemetry.javaagent:opentelemetry-javaagent-javaagent=/path/to/opentelemetry-javaagent.jario.opentelemetry.javaagent:opentelemetry-javaagent-javaagent=/path/to/opentelemetry-javaagent.jarorg.slf4j:jul-to-slf4jHandlerto JUL root loggerSLF4JBridgeHandler.removeHandlersForRootLogger(),SLF4JBridgeHandler.install()org.slf4j:log4j-over-slf4jlog4j.jarwithlog4j-over-slf4j.jaron classpathorg.apache.logging.log4j:log4j-to-slf4jProviderinterface used to bind the Log4j2 API to an implementationlog4j-to-slf4j.jaron classpathorg.slf4j:slf4j-jdk14SLF4JServiceProviderinterface used to bind SLF4J API to an implementationslf4j-jdk14.jaron classpathorg.slf4j:slf4j-reload4jSLF4JServiceProviderinterface used to bind SLF4J API to an implementationslf4j-reload4j.jaron classpathch.qos.logback:logback-classicSLF4JServiceProviderinterface used to bind SLF4J API to an implementationlogback-classic.jaron classpathorg.apache.logging.log4j:log4j-slf4j2-impl(org.apache.logging.log4j:log4j-slf4j-implfor older versions)SLF4JServiceProviderinterface used to bind SLF4J API to an implementationlog4j-slf4j2-impl.jaron classpath|
Popular log SDKs in the Java ecosystem.
io.opentelemetry:opentelemetry-sdkch.qos.logback:logback-coreorg.apache.logging.log4j:log4j-corelog4j:log4jch.qos.reload4j:reload4j:1.2.26Summary of how log APIs are routed to log SDKs.
opentelemetry-sdk)logback-classic,opentelemetry-logback-appender-1.0ORlog4j-slf4j-impl,opentelemetry-log4j-appender-2.17)jul-to-slf4j+ SLF4J API -> OpenTelemetry SDK), or otel javaagentopentelemetry-log4j-appender-2.17), or otel javaagentlog4j-over-slf4j+ SLF4J API -> OpenTelemetry SDK), or otel javaagentlogback-classic)jul-to-slf4j,logback-classic)log4j-to-slf4j,logback-classic)log4j-over-slf4j,logback-classic)slf4j-jdk14)log4j-to-slf4j,slf4j-jdk14)log4j-over-slf4j,slf4j-jdk14)log4j-slf4j2-impl)jul-to-slf4j,log4j-slf4j2-impl)log4j-core)log4j-over-slf4j,log4j-slf4j2-impl)slf4j-reload4j)jul-to-slf4j,slf4j-reload4j)log4j-to-slf4j,slf4j-reload4j)Appendix: call paths
In all cases we want the log record to be exported by the OpenTelemetry SDK va OTLP, and to the console appender.
Click to expand
No javaaagent, SLF4J API called by user:
slf4j-apilog4j-slf4j-impllog4j-coreopentelemetry-log4j-appenderopentelemetry-apiw/otel.loopback=trueopentelemetry-sdkopentelemetry-exporter-otlp✅Slf4jBridgeProcessorslf4j-apiterminates call w/otel.loopback=truelog4j-coreconsole appender ✅No javaaagent, JUL API called by user:
java.util.loggingjul-to-slf4jslf4j-apilog4j-slf4j-impllog4j-coreopentelemetry-log4j-appenderopentelemetry-apiw/otel.loopback=trueopentelemetry-sdkopentelemetry-exporter-otlp✅Slf4jBridgeProcessorslf4j-apiterminates call w/otel.loopback=truelog4j-coreconsole appender ✅No javaaagent, OpenTelemetry log API called by user:
opentelemetry-apiopentelemetry-sdkopentelemetry-exporter-otlp✅Slf4jBridgeProcessorslf4j-apiw/otel.loopback=truelog4j-slf4j-impllog4j-coreopentelemetry-log4j-appenderterminates calls w/otel.loopback=truelog4j-coreconsole appender ✅Javaaagent installed, SLF4J API called by user:
slf4j-apilog4j-slf4j-impllog4j-corelog4j-appender-2.17:javaagentopentelemetry-apiw/CallDepth(LoggerProvider)=1opentelemetry-sdkopentelemetry-exporter-otlp✅Slf4jBridgeAgentProcessorslf4j-apiterminates calls w/CallDepth(LoggerProvider)!=0)log4j-coreconsole appender ✅Javaaagent installed, JUL API called by user:
java.util.loggingjul-to-slf4jslf4j-apilog4j-slf4j-impllog4j-corelog4j-appender-2.17:javaagentopentelemetry-apiw/CallDepth(LoggerProvider)=1opentelemetry-sdkopentelemetry-exporter-otlp✅Slf4jBridgeAgentProcessorslf4j-apiterminates calls w/CallDepth(LoggerProvider)!=0)log4j-coreconsole appender ✅Javaaagent installed, OpenTelemetry log API called by user:
opentelemetry-apiopentelemetry-api:javaagentopentelemetry-sdkopentelemetry-exporter-otlp✅Slf4jBridgeAgentProcessorslf4j-apiw/CallDepth(LoggerProvider)=1log4j-slf4j-impllog4j-corelog4j-appender-2.17:javaagentterminates calls w/CallDepth(LoggerProvider)!=0log4j-coreconsole appender ✅Appendix: key links
Click to expand