add initial tracing docs

Author: kralicky

content/docs/reference/img/tracing/auth-flow-2.png

@@ -16,109 +16,106 @@ import TabItem from '@theme/TabItem';
 
 ## Summary
 
-Tracing tracks the progression of a single user request as it is handled by Pomerium.
+Pomerium has comprehensive support for OpenTelemetry tracing, allowing detailed introspection into requests and authorization flows.
+You can use tracing to debug errors and latency issues in your applications.
 
-Each unit of work is called a Span in a trace. Spans include metadata about the work, including the time spent in the step (latency), status, time events, attributes, links. You can use tracing to debug errors and latency issues in your applications, including in downstream connections.
-
-## How to configure
+## Configuration
 
 <Tabs>
 <TabItem value="Core" label="Core">
 
-#### Shared Tracing Settings
+### Environment Variables
+
+The recommended way to configure tracing is by using the standard OpenTelemetry environment variables:
+- [SDK environment variables](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#general-sdk-configuration)
+- [OTLP exporter environment variables](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/)
+
+The main variables used to configure tracing in Pomerium are the following:
 
-| Config Key | Description | Required |
-| :-- | :-- | --- |
-| tracing_provider | The name of the tracing provider. (e.g. Jaeger, Zipkin) | ✅ |
-| tracing_sample_rate | Percentage of requests to sample in decimal notation. Default is `0.0001`, or .01% | ❌ |
+| Name | Description | Default |
+| :--- | :---------- | :------ |
+| [`OTEL_TRACES_EXPORTER`](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#exporter-selection) | Trace exporter to be used. <br/> Valid values are `"otlp"` or `"none"` | `"none"` |
+| [`OTEL_EXPORTER_OTLP_ENDPOINT`](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_endpoint) or <br/> [`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_traces_endpoint) | See [Endpoint Configuration](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#endpoint-configuration). |
+| [`OTEL_EXPORTER_OTLP_PROTOCOL`](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_protocol) or <br/> [`OTEL_EXPORTER_OTLP_TRACES_PROTOCOL`](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_traces_protocol) | See [Protocol Configuration](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#protocol-configuration). <br/> Valid values are `"grpc"` or `"http/protobuf"`. <br/>If unset, Pomerium will attempt to determine the protocol based on the endpoint port number (the standard ports are 4317 for GRPC, 4318 for HTTP), otherwise it will default to `"http/protobuf"`. | (auto) |
+| [`OTEL_TRACES_SAMPLER_ARG`](https://opentelemetry.io/docs/languages/sdk-configuration/general/#otel_traces_sampler_arg) | Sampling probability, a number in the \[0..1\] range, e.g. `1.0` (sample all traces) or `0.25` (sample 25% of traces) | `1.0` |
 
-Set `tracing_sample_rate = 1` if you want to see all requests in the tracings.
+### Config file
 
-#### Datadog
+Tracing can also be configured using the Pomerium config file if desired:
 
-Datadog is a real-time monitoring system that supports distributed tracing and monitoring.
+| Config Key | Equivalent Environment Variable |
+| :--- | :---------- |
+| `tracing_provider` | [`OTEL_TRACES_EXPORTER`](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#exporter-selection) |
+| `tracing_otlp_endpoint` | [`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_traces_endpoint) |
+| `tracing_otlp_protocol` | [`OTEL_EXPORTER_OTLP_TRACES_PROTOCOL`](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_traces_protocol) |
+| `tracing_sample_rate` | [`OTEL_TRACES_SAMPLER_ARG`](https://opentelemetry.io/docs/languages/sdk-configuration/general/#otel_traces_sampler_arg) |
 
-| Config Key | Description | Required |
-| :-- | :-- | --- |
-| tracing_datadog_address | `host:port` address of the Datadog Trace Agent. Defaults to `localhost:8126` | ❌ |
+</TabItem>
+<TabItem value="Enterprise" label="Enterprise">
 
-#### Jaeger (partial)
+1. In the Enterprise Console, navigate to Settings > Tracing
 
-**Warning** At this time, the Jaeger protocol does not capture spans inside the Proxy Service. Please use the Zipkin protocol with Jaeger for full support.
+2. In the "Tracing Provider" dropdown, select "OTLP"
 
-[Jaeger](https://www.jaegertracing.io/) is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and troubleshooting microservices-based distributed systems, including:
+3. Enter your desired sample rate and OTLP endpoint
+
+4. Optionally, enter a protocol ("grpc" or "http/protobuf"). If the endpoint uses port 4317 or 4318, the protocol will be selected automatically. Port 4317 is the standard for OTLP GRPC, and 4318 for OTLP HTTP.
+
+![Enterprise tracing config](./img/tracing/tracing-otlp.png)
+</TabItem>
+</Tabs>
 
-- Distributed context propagation
-- Distributed transaction monitoring
-- Root cause analysis
-- Service dependency analysis
-- Performance / latency optimization
+## Examples
 
-| Config Key | Description | Required |
-| :-- | :-- | --- |
-| tracing_jaeger_collector_endpoint | Url to the Jaeger HTTP Thrift collector. | ✅ |
-| tracing_jaeger_agent_endpoint | Send spans to jaeger-agent at this address. | ✅ |
+### Using Jaeger to visualize trace data
 
-For quick local testing, use Jaeger all-in-one, which is an executable designed to launch the Jaeger UI, jaeger-collector, jaeger-query, and jaeger-agent, with an in-memory storage component.
+[Jaeger](https://www.jaegertracing.io/) is a popular open-source tracing platform. It can be used to collect trace data and visualize it in the browser.
+
+1. Run Jaeger in all-in-one mode with Docker:
 
 ```bash
-docker run -d --name jaeger \
-  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
-  -e COLLECTOR_OTLP_ENABLED=true \
-  -p 6831:6831/udp \
-  -p 6832:6832/udp \
-  -p 5778:5778 \
+$ docker run -d --name jaeger \
   -p 16686:16686 \
   -p 4317:4317 \
   -p 4318:4318 \
-  -p 14250:14250 \
-  -p 14268:14268 \
-  -p 14269:14269 \
-  -p 9411:9411 \
-  jaegertracing/all-in-one:1.45
-
+  jaegertracing/jaeger:latest
 ```
 
-Pomerium settings
-
-```yaml
-tracing_provider: jaeger
-tracing_jaeger_collector_endpoint: http://localhost:14268/api/traces
-tracing_jaeger_agent_endpoint: localhost:6831
+2. Run Pomerium with OpenTelemetry environment variables set:
+```bash
+$ OTEL_TRACES_EXPORTER=otlp OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 pomerium --config path/to/your/config.yaml`
 ```
 
-Open Jaeger UI at `http://localhost:16686` in the browser to view Pomerium traces.
+3. Navigate to a Pomerium route defined in the config file
 
-#### Zipkin
+4. Open your browser to http://localhost:16686 to view traces in the Jaeger UI.
 
-Zipkin is an open-source distributed tracing system and protocol.
+### Tracing errors
 
-Many tracing backends support Zipkin either directly or through intermediary agents, including Jaeger. For full tracing support, we recommend using the Zipkin tracing protocol.
+A typo in the OAuth2 issuer URL configuration is a common mistake that can lead to unexpected errors.
+A user attempting to navigate to a Pomerium route that requires authentication might see an error page instead of being redirected to the Identity Provider.
+In the Jaeger UI, traces that contain errors are highlighted and easy to find:
 
-| Config Key              | Description                      | Required |
-| :---------------------- | :------------------------------- | -------- |
-| tracing_zipkin_endpoint | Url to the Zipkin HTTP endpoint. | ✅       |
+![Jaeger trace list](./img/tracing/jaeger-trace-list-err.png)
 
-</TabItem>
-<TabItem value="Enterprise" label="Enterprise">
+Clicking on this trace will show us the original unauthenticated request (`GET https://verify.localhost.pomerium.io/`) and that it was redirected to sign in.
+When attempting to initiate the auth flow, an error was encountered, which was recorded in the trace:
 
-Configure **Tracing** in the Console:
+![Jaeger error trace](./img/tracing/error-flow.png)
 
-1. Select a **Tracing Provider** and set **Sample Tracing Rate**
+Clicking on the span that recorded the error will show the error message - we are missing a trailing slash in the issuer URL!
 
-![Select tracing provider](./img/tracing/tracing-providers.png)
+### Visualizing the Pomerium auth flow
 
-![Select tracing provider and sample rate](./img/tracing/default-tracing.png)
+Pomerium can trace a request's entire journey through the authentication process, across multiple individual redirects between Pomerium services and the Identity Provider.
 
-2. Configure tracing **Endpoints**
+For example, this trace shows an unauthenticated request (`GET https://verify.localhost.pomerium.io/`) that triggered a sequence of redirects to perform the auth flow:
 
-![Set Jaeger endpoints](./img/tracing/jaeger-endpoints.png)
+![Auth flow](./img/tracing/auth-flow.png)
 
-![Set Zipkin endpoint](./img/tracing/zipkin-endpoint.png)
+The trace above ends with a final redirect to repeat the original request, but this time the user is authenticated:
 
-</TabItem>
-</Tabs>
+![Auth flow 2](./img/tracing/auth-flow-2.png)
 
-### Examples
+This trace ends with the proxied request to the upstream server.
 
-![jaeger example trace](img/jaeger.png)