Getting started with Heroku OTEL and Grafana Cloud

Now that Fir is in GA, we wanted to share a quick guide to help you explore what we think will be one of the most powerful features of the new stack - Native OpenTelemetry (traces, metrics, logs).

From the OpenTelemetry Concepts and Heroku docs:

We designed the Fir generation of the Heroku platform to simplify application observability by integrating OpenTelemetry directly into its core. It offers developers a streamlined way to collect and export telemetry data without extensive configuration.

This is definitely not a deep exploration of building a production grade setup, our aim was sharing the fastest way to explore and let you get thinking about the possibilities with this level of data available to you.

Grafana Cloud

You can use a variety of applications to view OpenTelemetry data, but the fastest is probably to use Grafana Cloud. This provides:

• The ability to make use of traces, metrics and logs together in an interconnected suite.
• Get started fast with their hosted offering but retain the option to run the open source versions on Heroku ourselves in if we want to.
• The ability to send events directly from the Heroku Collector to the Grafana Cloud OTLP endpoint, without needing to set up our own collector or agent (but again have the option to do so later).

Our Test Architecture

We created a simple little chat app. It’s a bit of a contrived setup so that we could use a few different pieces of the Heroku Platform, allowing us to see how the OpenTelemetry for each looked, and how it worked together. We have:

• The chat application, a Node.js app running on a single dyno, publishes chat messages to Heroku Kafka and sends OTEL data from its backend.
• The second application, also a Node.js app running a single dyno, retrieves chat messages from Heroku Kafka and writes them to Heroku Postgres. It also sends OTEL data from its backend.

Instrumenting the code

We followed the Open Telemetry guide for automatic instrumentation in node, available here.

The only changes we made were to use the OLTPTraceExporter and OLTPMetricExporter instead of the console exporters.

The full code for our instrumentation.js was the following:

/*instrumentation.js*/
// Require dependencies
const { NodeSDK } = require("@opentelemetry/sdk-node");
const {
  OTLPTraceExporter,
} = require("@opentelemetry/exporter-trace-otlp-proto");
const {
  OTLPMetricExporter,
} = require("@opentelemetry/exporter-metrics-otlp-proto");
const {
  getNodeAutoInstrumentations,
} = require("@opentelemetry/auto-instrumentations-node");
const { PeriodicExportingMetricReader } = require("@opentelemetry/sdk-metrics");

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter(),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter(),
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

We then set the OTEL_SERVICE_NAME environment variable on both applications to set the logical name of the service.

If the service name was not specified, the OpenTelemetry SDKs will fallback to unknown_service: concatenated with process.executable.name, e.g. unknown_service:node

Connecting Heroku to Grafana Cloud

We essentially follow a combination of the Manual OpenTelemetry setup for advanced users section of the Grafana docs plus Working with Heroku Telemetry Drains, but here it is stepped out:

• Sign In (after signing up if needed) to the Grafana Cloud Portal.
• From your organization Overview, select or click Launch to open a Grafana Cloud stack.

• With a stack selected (in our case kilterset), click Configure from the OpenTelemetry tile.

• View the OTLP Endpoint screen, where we’ll gather all the details needed to setup a Heroku Telemetry Drain to send traces, metrics and logs to Grafana.

• On this screen, generate a Password / API Token by clicking Generate now. Which will give you the Environment Variables which we’ll make use of next.
  
  • OTEL_EXPORTER_OTLP_ENDPOINT is the endpoint URL we’ll use to send telemetry to.
  • OTEL_EXPORTER_OTLP_HEADERS provides a Base64 encoded concatenation of the Instance ID and Password/API Token for passing in to authenticate us.
• From the Heroku CLI, add a Telemetry Drain with the heroku telemetry:add command:
  $ heroku telemetry:add "OTEL_EXPORTER_OTLP_ENDPOINT" --space heroku-space-name --signals traces,logs,metrics --transport http --headers '{"OTEL_EXPORTER_OTLP_HEADERS}'

Note: above we’re adding the whole Space, you can also do it per app with the --app flag instead.

Once done, you should be able to check the Logs, Metrics and Traces tabs in Grafana and see telemetry flowing through. Some examples below:

There’s a lot more you can do from here, but this is enough to get you you up and running quickly and start the gears turning on how you can make use of this capability.

You might also want to consider making use of your own OTEL Collector to give you some more control, or setup some filters, dashboards, etc to get the insights you want.

As soon as we got this working we couldn’t help ourselves plugging other apps in and answering recent questions we had. This is another example of a really powerful capability made so easy to use by Heroku.

‍