You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: content/docs/reference/tracing.mdx
+92-65
Original file line number
Diff line number
Diff line change
@@ -16,109 +16,136 @@ import TabItem from '@theme/TabItem';
16
16
17
17
## Summary
18
18
19
-
Tracing tracks the progression of a single user request as it is handled by Pomerium.
19
+
Pomerium has comprehensive support for OpenTelemetry tracing, allowing detailed introspection into requests and authorization flows. You can use tracing to debug errors and latency issues in your applications.
20
20
21
-
Each unit of work is called a Span in a trace. Spans include metadata about the work, including the time spent in the step (latency), status, time events, attributes, links. You can use tracing to debug errors and latency issues in your applications, including in downstream connections.
22
-
23
-
## How to configure
21
+
## Configuration
24
22
25
23
<Tabs>
26
24
<TabItemvalue="Core"label="Core">
27
25
28
-
#### Shared Tracing Settings
26
+
### Environment Variables
27
+
28
+
The recommended way to configure tracing is by using the standard OpenTelemetry environment variables:
The main variables used to configure tracing in Pomerium are the following:
34
+
35
+
| Name | Description | Default |
36
+
| :-- | :-- | :-- |
37
+
|[`OTEL_TRACES_EXPORTER`](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#exporter-selection)| Trace exporter to be used. <br/> Valid values are `"otlp"` or `"none"`|`"none"`|
38
+
|[`OTEL_EXPORTER_OTLP_ENDPOINT`](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_endpoint) or <br/> [`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_traces_endpoint)| See [Endpoint Configuration](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#endpoint-configuration). |
39
+
|[`OTEL_EXPORTER_OTLP_PROTOCOL`](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_protocol) or <br/> [`OTEL_EXPORTER_OTLP_TRACES_PROTOCOL`](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_traces_protocol)| See [Protocol Configuration](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#protocol-configuration). <br/> Valid values are `"grpc"` or `"http/protobuf"`. <br/>If unset, Pomerium will attempt to determine the protocol based on the endpoint port number (the standard ports are 4317 for GRPC, 4318 for HTTP), otherwise it will default to `"http/protobuf"`. | (auto) |
40
+
|[`OTEL_TRACES_SAMPLER_ARG`](https://opentelemetry.io/docs/languages/sdk-configuration/general/#otel_traces_sampler_arg)| Sampling probability, a number in the \[0..1\] range, e.g. `1.0` (sample all traces) or `0.25` (sample 25% of traces) |`1.0`|
41
+
42
+
### Config file
43
+
44
+
Tracing can also be configured using the Pomerium config file if desired:
| tracing_provider | The name of the tracing provider. (e.g. Jaeger, Zipkin) | ✅ |
33
-
| tracing_sample_rate | Percentage of requests to sample in decimal notation. Default is `0.0001`, or .01% | ❌ |
53
+
</TabItem>
54
+
<TabItemvalue="Enterprise"label="Enterprise">
34
55
35
-
Set `tracing_sample_rate = 1` if you want to see all requests in the tracings.
56
+
1. In the Enterprise Console, navigate to Settings > Tracing
36
57
37
-
#### Datadog
58
+
2. In the "Tracing Provider" dropdown, select "OTLP"
38
59
39
-
Datadog is a real-time monitoring system that supports distributed tracing and monitoring.
60
+
3. Enter your desired sample rate and OTLP endpoint
40
61
41
-
| Config Key | Description | Required |
42
-
| :-- | :-- | --- |
43
-
| tracing_datadog_address |`host:port` address of the Datadog Trace Agent. Defaults to `localhost:8126`| ❌ |
62
+
4. Optionally, enter a protocol ("grpc" or "http/protobuf"). If the endpoint uses port 4317 or 4318, the protocol will be selected automatically. Port 4317 is the standard for OTLP GRPC, and 4318 for OTLP HTTP.
**Warning** At this time, the Jaeger protocol does not capture spans inside the Proxy Service. Please use the Zipkin protocol with Jaeger for full support.
66
+
</TabItem>
67
+
</Tabs>
48
68
49
-
[Jaeger](https://www.jaegertracing.io/) is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and troubleshooting microservices-based distributed systems, including:
69
+
## Examples
50
70
51
-
- Distributed context propagation
52
-
- Distributed transaction monitoring
53
-
- Root cause analysis
54
-
- Service dependency analysis
55
-
- Performance / latency optimization
71
+
### Using Jaeger to visualize trace data
56
72
57
-
| Config Key | Description | Required |
58
-
| :-- | :-- | --- |
59
-
| tracing_jaeger_collector_endpoint | Url to the Jaeger HTTP Thrift collector. | ✅ |
60
-
| tracing_jaeger_agent_endpoint | Send spans to jaeger-agent at this address. | ✅ |
73
+
[Jaeger](https://www.jaegertracing.io/) is a popular open-source tracing platform. It can be used to collect trace data and visualize it in the browser.
61
74
62
-
For quick local testing, use Jaeger all-in-one, which is an executable designed to launch the Jaeger UI, jaeger-collector, jaeger-query, and jaeger-agent, with an in-memory storage component.
75
+
1. Run Jaeger in all-in-one mode with Docker:
63
76
64
77
```bash
65
-
docker run -d --name jaeger \
66
-
-e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
67
-
-e COLLECTOR_OTLP_ENABLED=true \
68
-
-p 6831:6831/udp \
69
-
-p 6832:6832/udp \
70
-
-p 5778:5778 \
78
+
$ docker run -d --name jaeger \
71
79
-p 16686:16686 \
72
80
-p 4317:4317 \
73
81
-p 4318:4318 \
74
-
-p 14250:14250 \
75
-
-p 14268:14268 \
76
-
-p 14269:14269 \
77
-
-p 9411:9411 \
78
-
jaegertracing/all-in-one:1.45
79
-
82
+
jaegertracing/jaeger:latest
80
83
```
81
84
82
-
Pomerium settings
85
+
2. Run Pomerium with OpenTelemetry environment variables set:
Open Jaeger UI at `http://localhost:16686` in the browser to view Pomerium traces.
91
+
3. Navigate to a Pomerium route defined in the config file
91
92
92
-
#### Zipkin
93
+
4. Open your browser to http://localhost:16686 to view traces in the Jaeger UI.
93
94
94
-
Zipkin is an open-source distributed tracing system and protocol.
95
+
### Tracing errors
95
96
96
-
Many tracing backends support Zipkin either directly or through intermediary agents, including Jaeger. For full tracing support, we recommend using the Zipkin tracing protocol.
97
+
A typo in the OAuth2 issuer URL configuration is a common mistake that can lead to unexpected errors. A user attempting to navigate to a Pomerium route that requires authentication might see an error page instead of being redirected to the Identity Provider. In the Jaeger UI, traces that contain errors are highlighted and easy to find:
Clicking on this trace will show us the original unauthenticated request (`GET https://verify.localhost.pomerium.io/`) and that it was redirected to sign in. When attempting to initiate the auth flow, an error was encountered, which was recorded in the trace:
109
+
If upstream applications also have OpenTelemetry support, traces will propagate through Pomerium to those applications and the combined trace data will be visible.
The [Securing Grafana with Pomerium](../guides/grafana.mdx) guide can help you get started with a new Grafana deployment.
118
116
119
-
</TabItem>
120
-
</Tabs>
117
+
To enable OpenTelemetry traces in Grafana, set the environment variable `GF_TRACING_OPENTELEMETRY_OTLP_ADDRESS` to the same ip:port (without scheme) as the OTLP endpoint configured in Pomerium.
118
+
119
+
Alternatively, this can be setin the Grafana config file:
120
+
121
+
```ini
122
+
# grafana.ini
123
+
[tracing.opentelemetry.otlp]
124
+
address = x.x.x.x:4317
125
+
```
126
+
127
+
Note that at the time of writing, Grafana only supports exporting OTLP traces with the GRPC protocol. Most tracing backends, like Jaeger or the OTel Collector, support both protocols however.
128
+
129
+
With tracing enabled in both Pomerium and Grafana, navigate to your Grafana route. After a few seconds, the combined traces should be visible in Jaeger:
130
+
131
+
132
+
133
+
The bottom trace (occurred first) is the initial unauthenticated request to Pomerium. The top trace is the authenticated request, after the user signed in and was redirected. This trace includes spans exported by Grafana itself, which we can see in detail:
Grafana exports very detailed traces, which can be helpful in debugging complex issues. The combined trace data helps easily visualize the request flow between Pomerium and Grafana, or any other upstream application.
138
+
139
+
### Visualizing the Pomerium auth flow
140
+
141
+
Pomerium can trace a request's entire journey through the authentication process, across multiple individual redirects between Pomerium services and the Identity Provider.
142
+
143
+
For example, this trace shows an unauthenticated request (`GET https://verify.localhost.pomerium.io/`) that triggered a sequence of redirects to perform the auth flow:
144
+
145
+
146
+
147
+
The trace above ends with a final redirect to repeat the original request, but this time the user is authenticated:
121
148
122
-
### Examples
149
+
123
150
124
-
151
+
This trace ends with the proxied request to the upstream server.
0 commit comments