Not all requests are Traced

[theobouwman]

Hi I am using this code to use Otel tracing with FastAPI on Cloud Run, but not all my requests are traced as you can see.
Grafana shows 42 requests but GCP Trace only should 23 traces...

 SQLAlchemyInstrumentor().instrument(engine=db._engine)

    resource = Resource.create(
        {
            "service.name": service_name,
        }
    )

    set_global_textmap(CloudTraceFormatPropagator())

    tracer_provider = TracerProvider(resource=resource)
    cloud_trace_exporter = CloudTraceSpanExporter()
    tracer_provider.add_span_processor(
        # BatchSpanProcessor buffers spans and sends them in batches in a
        # background thread. The default parameters are sensible, but can be
        # tweaked to optimize your performance
        BatchSpanProcessor(cloud_trace_exporter)
    )

    meter_provider = MeterProvider(
        metric_readers=[
            PeriodicExportingMetricReader(
                CloudMonitoringMetricsExporter(), export_interval_millis=5000
            )
        ],
        resource=resource,
    )

    trace.set_tracer_provider(tracer_provider)
    metrics.set_meter_provider(meter_provider)

    tracer = trace.get_tracer(__name__)
    meter = metrics.get_meter(__name__)

    FastAPIInstrumentor.instrument_app(
        app,
        tracer_provider=tracer_provider,
        meter_provider=meter_provider
    )

We also have a 1second latency when request code in, and this is what I want to solve by looking at the Traces...

[nickelghost]

This is probably a Cloud Run / Google specific issue, as I get similar results on Cloud Run using the Go SDK.

[nabheet]

Are you also getting these errors? I have even increased the interval to 2mins and it didn't help. Can you please advise?

[theobouwman]

We turned away from GCP tracing as we got a huge bill increase, moved to Sentry. But we managed to trace all requests eventually. We used Cloud Run and the container was killed before it could push its traces. Also you need to check if the instance has permission to write the traces

[nabheet]

Finally found the "issue" - It turns out that the default sampler is parentbased_always_on. After setting the environment variable OTEL_TRACES_SAMPLER to always_on, I am now seeing all the traces.

https://anecdotes.dev/opentelemetry-on-google-cloud-unraveling-the-mystery-f61f044c18be
https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/

[nabheet]

And I also had to add retry functionality to the Cloud Monitoring metric service client:

from google.cloud.monitoring_v3.services.metric_service import MetricServiceClient
from google.api_core.retry import Retry
from google.api_core.retry.retry_base import if_exception_type
from google.api_core.exceptions import (
    BadGateway,
    InvalidArgument,
    DeadlineExceeded,
    InternalServerError,
    TooManyRequests,
    ServiceUnavailable,
    RetryError,
)

_RETRIABLE_TYPES = (
    BadGateway,
    InvalidArgument,
    DeadlineExceeded,
    InternalServerError,
    TooManyRequests,
    ServiceUnavailable,
)

class CustomMetricServiceClient(MetricServiceClient):
    def create_time_series(
        self,
        *args,
        **kwargs,
    ) -> None:
        try:
            super().create_time_series(
                *args,
                **kwargs,
                retry=Retry(predicate=if_exception_type(*_RETRIABLE_TYPES)),
            )
        except RetryError as ex:
            logger.debug("Failed to write metrics to Cloud Monitoring", ex)

[nabheet]

Hope this helps other people encountering the same "issue".