Is it possible to have traces go to aws xray trace dashboard?

[IhsanMujdeci]

When using the aws-otel-nodejs layer from AWS we can export the otel directly to AWS.
Is this possible with the solution or is it only for external non aws sources only?

Thanks

[mheffner]

Hi @IhsanMujdeci

Yes, we have support for exporting traces to AWS Xray. The instructions for configuration can be found in the Rotel repo: here.

For example, a minimal environment config would look like:

ROTEL_EXPORTER="aws-xray"
ROTEL_XRAY_EXPORTER_REGION=<region>

Give that a shot and let us know how it goes.

[IhsanMujdeci]

@mheffner thanks mate! It worked. Doing some benchmarks for work right now.
Another question probably silly, how do we enable AWS auto instrumentation?
Is it same as normal otel layer config?

[mheffner]

@IhsanMujdeci great to hear!

I haven't tried Rotel with the AWS Nodejs layer, but we do have an example of using it with the OT Nodejs layer: here. In theory it should work similarly for the AWS layer, we are just replacing the collector.

Do you know what the AWS Nodejs provides over the OpenTelemetry Nodejs layer? I'm curious what AWS auto instrumentation provides.

[IhsanMujdeci]

Hey, when I said I had it working it was a false alarm. I had xray http instrumented and that what showed up.
I can't seem to get telemetry to appear on xray trace console.
I've tried experimenting with environment variables but no luck as of yet.

[rjenkins]

Hi @IhsanMujdeci, can you check and see if the IAM Role you are using has the xray:PutTraceSegments action permission?

[rjenkins]

@IhsanMujdeci during development and testing over here I've been using the built-in AWSXrayWriteOnlyAccess however you should probably only need the xray:PutTraceSegments action. Example Role with attached permission policies and action.

[IhsanMujdeci]

Yes is has the permissions, I have double checked permissions and normal ADOT layer and plain xray are emitting if i configure my lambda.
Side question, in the Configuration portion of the docs are all of the options to be prepended with "ROTEL"?

[IhsanMujdeci]

To be clear, regular xray traces appear to be working. Like showing me lambda to lambda connections. But not the extra sauce like HTTP and Dynamo when using rotel

[rjenkins]

Yes is has the permissions, I have double checked permissions and normal ADOT layer and plain xray are emitting if i configure my lambda. Side question, in the Configuration portion of the docs are all of the options to be prepended with "ROTEL"?

Correct, the env variables are prefixed with ROTEL, for XRay you should only have to set two. One to select the exporter and one for the region.

ROTEL_EXPORTER=aws-xray
ROTEL_XRAY_EXPORTER_REGION=us-east-1 <-- replace with your region.

[rjenkins]

To be clear, regular xray traces appear to be working. Like showing me lambda to lambda connections. But not the extra sauce like HTTP and Dynamo when using rotel

If you can grab a copy of the rotel logs and share any error messages related to the XRay exporter that might be helpful.

[IhsanMujdeci]

-- | --
  | 2025-06-03T02:27:42.331Z | INFO Starting Rotel. grpc_endpoint="127.0.0.1:4317" http_endpoint="127.0.0.1:4318"
  | 2025-06-03T02:27:42.331Z | INFO No exporters are configured for metrics, OTLP receiver will be configured to not accept metrics
  | 2025-06-03T02:27:42.331Z | INFO No exporters are configured for logs, OTLP receiver will be configured to not accept logs
  | 2025-06-03T02:27:42.331Z | TELEMETRY Name: rotel-extension State: Subscribed Types: [Platform, Function, Extension]
  | 2025-06-03T02:27:42.332Z | INFO AWS X-Ray only supports a batch size of 50 segments, setting batch max size to 50
  | 2025-06-03T02:27:42.334Z | INFO Rotel Lambda Extension started in 60ms
  | 2025-06-03T02:27:42.547Z | Registering OpenTelemetry
  | 2025-06-03T02:27:42.570Z | OpenTelemetry wrapper init completed in 153 ms
  | 2025-06-03T02:27:42.678Z | OpenTelemetry loader init completed in 108 ms
  | 2025-06-03T02:27:42.681Z | (node:7) ExperimentalWarning: Use `importAttributes` instead of `importAssertions`
  | 2025-06-03T02:27:42.681Z | (Use `node --trace-warnings ...` to show where the warning was created)
  | 2025-06-03T02:27:45.286Z | EXTENSION Name: rotel-extension State: Ready Events: [INVOKE, SHUTDOWN]
  | 2025-06-03T02:27:45.288Z | START RequestId: 45ecdef9-13bd-41f1-b3d3-19e8945b5b66 Version: $LATEST
... application logs
2025-06-03T02:27:46.437Z	45ecdef9-13bd-41f1-b3d3-19e8945b5b66	ERROR	
{
    "stack": "Error: PeriodicExportingMetricReader: metrics export failed (error OTLPExporterError: Not Found)\n    at d._doRun (/opt/773.wrapper.js:1:3565)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async d._runOnce (/opt/773.wrapper.js:1:2894)\n    at async d.onForceFlush (/opt/773.wrapper.js:1:3784)\n    at async d.forceFlush (/opt/773.wrapper.js:1:2000)\n    at async ee.forceFlush (/opt/773.wrapper.js:1:21040)\n    at async Promise.all (index 0)\n    at async ne.forceFlush (/opt/773.wrapper.js:1:24412)\n    at async Promise.all (index 1)",
    "message": "PeriodicExportingMetricReader: metrics export failed (error OTLPExporterError: Not Found)",
    "name": "Error"
}

[rjenkins]

To be clear, regular xray traces appear to be working. Like showing me lambda to lambda connections. But not the extra sauce like HTTP and Dynamo when using rotel

Thanks @IhsanMujdeci another question: Does the HTTP library you're using and DynamoDB tracing layer generate OTel traces, or are they XRay. Right now the XRay support in rotel only handles receiving OTel traces, converting into XRay format, and exporting to XRay. If those libraries are generating XRay traces, we don't have XRay receiver support at the moment.

[IhsanMujdeci]

Great question because I assumed with the layers of

  const rotelLayerARN = 'arn:aws:lambda:ap-southeast-2:418653438961:layer:rotel-extension-amd64-alpha:24'
  const otelLayer = 'arn:aws:lambda:ap-southeast-2:184161586896:layer:opentelemetry-nodejs-0_14_0:1'

That the instrumentation would just pick it up and generate OTEL traces and throw them out.
I will give you more verbose logs soon.

Also thanks a lot guys for responding so quickly!

Side question. Have you tried OTEL with no collector approach? It feels to be promising. Even with ROTEL layer we seem to be getting ~3 seconds init time with cold starts. Better than ADOT.

[rjenkins]

-- | --
  | 2025-06-03T02:27:42.331Z | INFO Starting Rotel. grpc_endpoint="127.0.0.1:4317" http_endpoint="127.0.0.1:4318"
  | 2025-06-03T02:27:42.331Z | INFO No exporters are configured for metrics, OTLP receiver will be configured to not accept metrics
  | 2025-06-03T02:27:42.331Z | INFO No exporters are configured for logs, OTLP receiver will be configured to not accept logs
  | 2025-06-03T02:27:42.331Z | TELEMETRY Name: rotel-extension State: Subscribed Types: [Platform, Function, Extension]
  | 2025-06-03T02:27:42.332Z | INFO AWS X-Ray only supports a batch size of 50 segments, setting batch max size to 50
  | 2025-06-03T02:27:42.334Z | INFO Rotel Lambda Extension started in 60ms
  | 2025-06-03T02:27:42.547Z | Registering OpenTelemetry
  | 2025-06-03T02:27:42.570Z | OpenTelemetry wrapper init completed in 153 ms
  | 2025-06-03T02:27:42.678Z | OpenTelemetry loader init completed in 108 ms
  | 2025-06-03T02:27:42.681Z | (node:7) ExperimentalWarning: Use `importAttributes` instead of `importAssertions`
  | 2025-06-03T02:27:42.681Z | (Use `node --trace-warnings ...` to show where the warning was created)
  | 2025-06-03T02:27:45.286Z | EXTENSION Name: rotel-extension State: Ready Events: [INVOKE, SHUTDOWN]
  | 2025-06-03T02:27:45.288Z | START RequestId: 45ecdef9-13bd-41f1-b3d3-19e8945b5b66 Version: $LATEST
... application logs
2025-06-03T02:27:46.437Z	45ecdef9-13bd-41f1-b3d3-19e8945b5b66	ERROR	
{
    "stack": "Error: PeriodicExportingMetricReader: metrics export failed (error OTLPExporterError: Not Found)\n    at d._doRun (/opt/773.wrapper.js:1:3565)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async d._runOnce (/opt/773.wrapper.js:1:2894)\n    at async d.onForceFlush (/opt/773.wrapper.js:1:3784)\n    at async d.forceFlush (/opt/773.wrapper.js:1:2000)\n    at async ee.forceFlush (/opt/773.wrapper.js:1:21040)\n    at async Promise.all (index 0)\n    at async ne.forceFlush (/opt/773.wrapper.js:1:24412)\n    at async Promise.all (index 1)",
    "message": "PeriodicExportingMetricReader: metrics export failed (error OTLPExporterError: Not Found)",
    "name": "Error"
}

Thanks for this @IhsanMujdeci. So It looks like the X-Ray exporter has started and is configured fine. I don't see any errors here that are related to failure to post to XRay. If rotel cannot publish to XRay you'll see an error that looks something like below.

2025-05-20T19:24:21.934753Z ERROR Failed to export to X-Ray: Http(Parts { status: 400, version: HTTP/2.0, headers: {"date": "Tue, 20 May 2025 19:24:21 GMT", "content-type": "application/json", "content-lengt
h": "16", "x-amzn-requestid": "af94bf04-7f99-4410-8ee8-8be16c7fd607", "strict-transport-security": "max-age=31540000; includeSubDomains", "x-amzn-errortype": "SerializationException", "x-content-type-options
": "nosniff"} }, None) telemetry_type="traces" 

This other error doesn't relate directly to XRay afaict but we should investigate. @mheffner are you familiar with that error above for PeriodicExportingMetricReader.

[rjenkins]

Great question because I assumed with the layers of

  const rotelLayerARN = 'arn:aws:lambda:ap-southeast-2:418653438961:layer:rotel-extension-amd64-alpha:24'
  const otelLayer = 'arn:aws:lambda:ap-southeast-2:184161586896:layer:opentelemetry-nodejs-0_14_0:1'

That the instrumentation would just pick it up and generate OTEL traces and throw them out. I will give you more verbose logs soon.

Also thanks a lot guys for responding so quickly!

Side question. Have you tried OTEL with no collector approach? It feels to be promising. Even with ROTEL layer we seem to be getting ~3 seconds init time with cold starts. Better than ADOT.

Happy to help @IhsanMujdeci. Feel free to hop in our discord as well if you'd like to chat and discuss more in real-time. FWIW there are a few folks with lots of Lambda experience there. Re: cold start time, not sure what all is in your stack but our latest benchmarks indicate rotel should add only 200-400ms to the startup time, with an average of less than 200.

Great to hear it's quicker than ADOT though 🙌