No Metrics or Traces

[gtokman]

Great work on this stack!

I'm struglling to get the Metrics and Traces to show up in the Grafana UI. I followed your steps here:

Using [Locomotive](https://railway.com/template/jP9r-f) for Loki
You can easily ingest all of your railway logs into Loki from any service using [Locomotive](https://railway.com/template/jP9r-f). Just spin up their template, drop in your Railway API key, the ID of the services you want to monitor, and a link to your new Loki instance and logs will start flowing! no code changes needed anywhere!

Using standard observability tooling
To send data from your other Railway applications to this observability stack:

In your application's Railway service, add environment variables that reference the internal URLs:
LOKI_URL=${{Grafana.LOKI_INTERNAL_URL}}
PROMETHEUS_URL=${{Grafana.PROMETHEUS_INTERNAL_URL}}
TEMPO_URL=${{Grafana.TEMPO_INTERNAL_URL}}
Configure your application's logging, metrics, or tracing libraries to use these URLs
Your application data will automatically appear in your Grafana dashboards

I setup my telementry in my Node app, but I don't see them in Grafana. Do i need to configure a otel-collector-config.yaml for Locomotive to send to the correct place?

    opentelemetry({
      spanProcessors: [
        new BatchSpanProcessor(
          new OTLPTraceExporter({
            url: process.env.TEMPO_URL!,
          })
        ),
      ],
      serviceName: "api",
    })
  )

[MykalMachon]

Hey @gtokman, thanks!
I definitely didn't do a great job of providing documentation on using Grafana Tempo in this stack.

I'll get a better example setup in /examples/api and merge it in. I'll also explicitly set the tempo protocol endpoints in /tempo/tempo.yml so it's hopefully a bit more clear where you can ingest your traces 👍🏻

As far as metrics go, if you're hoping to use prometheus you will have to fork this repo and alter the prometheus/prom.yml file to pull from your metrics endpoints. Since prometheus is pull based unfortunately that's the only option 😄

You'll have to read more on the locomotive docs to see how to push logs- they have a solid guide on setting up Loki with logs.

[MykalMachon]

Hey @gtokman I think you should be good to go now.

Fixing up Tempo to expose ingest properly

I made a slight change to the tempo service's configuration to make sure you can push in traces via Railway. To update, you should be able to go into the tempo service > go to the settings tab > find the "upstream repo" setting and click "check for updates". This should trigger a new update for the service.

Providing an explicit example of using Tempo in node.js

If you check out examples/api/tracer.js in this repo you should be able to see a working ingest using the open telemetry packages and this stack. You'll just need to import the tracer.js file in your entrypoint javascript file and make sure you send some spans over (there's examples of that in that folder as well)

I was able to get the example api app working and pushing spans into tempo via railway just now but let me know if you're still having issues!