Skip to content

Service call chain broken based on gRPC protocol #3120

@yitian108

Description

@yitian108

Describe the bug

When I used OpenTelemetry go instrumentation to perform automatic instrumentation on the GRPC program to verify the client->middle->middle->server call chain, three services based on k8s were launched on December 11, 2025 at T08:51:03. From the visualization of the call chain from Tempo, it can be seen that the call logic from 08:51:03 to 08:53:33 basically conforms to the client->middle->middle->server call chain. However, from 08:53:33 onwards, each trace on the client side only has one span of its own, and there is no longer a middle or server side. Moreover, some traces from 08:51:03 to 08:53:33 have duplicate middle spans.

Environment

  • OS: AlmaLinux 9.6
  • Go Version: 1.24.10
  • Version: otel/autoinstrumentation-go:latest (v0.23.0)

To Reproduce

Steps to reproduce the behavior:

  1. kubectl apply -f grpc-auto-demo.yaml
  2. select grpc-auto-client-svc/grpc-auto-middle-svc/grpc-auto-server-svc
  3. query and inspect each traceID
  4. See only one span for client and no other spans for middle/server after startup a few minutes

Expected behavior

should be a full call chains client->middle->middle->server all the time rather than only lasting a few minutes from starting the pods

Additional context

please see the attachments

go mods.txt
grpc-auto-demo.yaml
otel-logs.log
Image
Image

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions