Observability with OpenTelemetry

GoApp is instrumented with OpenTelemetry (OTel) to provide comprehensive observability into the application's performance through traces and metrics. This allows developers to monitor, debug, and understand the application's behavior in real-time.

The entire observability stack is configured and managed within the internal/pkg/apm package.

Overview

The apm package initializes and configures the necessary OpenTelemetry providers and exporters. It sets up:

• Tracer Provider: For distributed tracing.
• Meter Provider: For collecting metrics.

These providers are then used to create middleware and instrument specific parts of the application, like the HTTP server and the database client.

Distributed Tracing

Distributed tracing allows you to follow a request as it travels through different parts of the application and even across different services. In GoApp, tracing is implemented in two key areas:

1. HTTP Server Middleware

The HTTP server uses the go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp package to create a middleware. This middleware automatically starts a new trace span for each incoming HTTP request, measures its duration, and captures relevant attributes like the HTTP method, path, and status code.

Configuration can be seen in cmd/server/http/http.go:

// cmd/server/http/http.go
otelopts := []otelhttp.Option{
    // Exclude health checks from traces
    otelhttp.WithFilter(func(req *http.Request) bool {
        return !strings.HasPrefix(req.URL.Path, "/-/")
    }),
    // Format span names to reduce cardinality
    otelhttp.WithSpanNameFormatter(func(operation string, r *http.Request) string {
        wctx := webgo.Context(r)
        if wctx == nil {
            return r.URL.Path
        }
        return wctx.Route.Pattern
    }),
}

apmMw := apm.NewHTTPMiddleware(otelopts...)
router.Use(func(w http.ResponseWriter, r *http.Request, hf http.HandlerFunc) {
    apmMw(hf).ServeHTTP(w, r)
})

2. PostgreSQL Driver Instrumentation

Database queries are also traced using the github.com/exaring/otelpgx library. It wraps the pgx database driver to create a child span for every SQL query executed. This allows you to see exactly how much time is spent in the database for any given request.

Configuration is in internal/pkg/postgres/postgres.go:

// internal/pkg/postgres/postgres.go
pgxconfig.Tracer = otelpgx.NewTracer(
    otelpgx.WithTracerProvider(apm.Global().GetTracerProvider()),
)

Metrics

The application collects metrics and exposes them for scraping by a Prometheus server.

Prometheus Exporter

The internal/pkg/apm/prometheus.go file sets up a Prometheus exporter. This exporter starts a separate HTTP server (by default on port 9090) and exposes a /metrics endpoint.

// internal/pkg/apm/prometheus.go
func prometheusScraper(opts *Options) {
    mux := http.NewServeMux()
    mux.Handle("/-/metrics", promhttp.Handler())
    server := &http.Server{
        Handler:           mux,
        Addr:              fmt.Sprintf(":%d", opts.PrometheusScrapePort),
    }
    // ... server starts listening
}

Metrics collected include standard ones from the otelhttp middleware (e.g., request counts, latency histograms) and any custom metrics defined within the application.

Configuration

The observability features are configured via the apm.Options struct, which is populated from the main application config.

• PrometheusScrapePort: The port for the Prometheus /metrics endpoint (e.g., 9090).
• TracesSampleRate: A value between 0.0 and 100.0 to control the percentage of traces that are sampled and exported.
• CollectorURL: The endpoint of an OpenTelemetry Collector where traces should be sent (e.g., http://otel-collector:4318).
• UseStdOut: If true, traces and metrics are printed to standard output instead of being exported, which is useful for local debugging.