Extract trace context when lambda is called through ALB

[kimmoahokas]

Expected Behavior

Lambda function called by AWS Application Load Balancer connects traces properly to upstream service

Actual Behavior

Traces are not connected

Steps to Reproduce the Problem

1. Configure traced service to call AWS ALB
2. Configure ALB to call lambda
3. Instrument the lambda with datadog-lambda-js

Specifications

• Datadog Lambda Layer version: arn:aws:lambda:eu-west-1:464622532012:layer:Datadog-Node20-x:115
• Node version: Node 20
• Datadog Lambda Extension version: arn:aws:lambda:eu-west-1:464622532012:layer:Datadog-Extension-ARM:65

The event format is documented at https://docs.aws.amazon.com/lambda/latest/dg/services-alb.html

[purple4reina]

Hey @kimmoahokas, thanks for reaching out. I'm curious though, from my understanding of ALB (you can see an example event here) our extractors should be able to handle them out of the box. We even have a test for this here. Basically, whenever there's a header key in the event payload, we'll look through it when extracting trace context.

Instead, I'm wondering if your issue is upstream. We should first check to make sure the caller is correctly injecting trace context. You can check this by inspecting the inbound lambda payload by adding DD_CAPTURE_LAMBDA_PAYLOAD=true and then looking at the aws.lambda span on any trace. Would you mind doing that and sharing an example inbound event payload with us? (of course redact anything sensitive)

[kimmoahokas]

Hello @purple4reina oh, thanks for the suggestion. I'm sure our upstream component adds trace headers (It's scala play play_ws client). But I will investigate this more after the holidays.

Mostly as a reminder for myself, this particular ALB+Lambda combination uses ALB Multi-Value headers which apparently changes the incomign event structure. So that might be the culprit here. Anyway, I'll get back to this on January since this is not a critical problem for us at the moment.

[purple4reina]

Ah yes, multi-value headers. I really suspect that is the culprit here. Thanks for that docs link. We've added support for multi-value headers for other languages, but not js yet. I'll create an internal feature request and add it to our future road map.

[kimmoahokas]

Thank you so much @purple4reina