Distributed Tracing

Distributed tracing follows a request across services: each service adds spans to a shared trace, linked by headers on outbound calls. In this agent it is enabled by default. This page explains header formats, sampling configuration (including the environment variable forms), span limits, and Infinite Tracing.

Turning It On or Off

Setting Default Env var
distributed_tracing.enabled true NEW_RELIC_DISTRIBUTED_TRACING_ENABLED
span_events.enabled true NEW_RELIC_SPAN_EVENTS_ENABLED

With distributed tracing disabled, getTraceMetadata() returns {}, linking metadata has no trace.id, and insertDistributedTraceHeaders() does nothing.

Header Formats

For outbound calls made with instrumented clients, the agent adds W3C Trace Context headers (traceparent, tracestate). It can also add the older New Relic newrelic header:

Setting Default Effect
distributed_tracing.exclude_newrelic_header true When true, the newrelic header is not sent; only W3C headers are. Set to false to send both, for example when calling services with old New Relic agents that only understand newrelic.

Note: The schema description of this setting says "By default (when false) both W3C TraceContext ... and New Relic formats will be sent", but the actual default is true. Trust the default: out of the box, only W3C headers are sent.

On inbound requests, traceparent is preferred; the newrelic header is only used when no traceparent is present. Trace context from other New Relic accounts is accepted only when it matches your trusted_account_key (sent by the server; configurable locally only in serverless mode).

Sampling

Not every trace is kept. The agent makes a sampling decision per trace, using one of several samplers depending on the situation:

Sampler setting Applies when
distributed_tracing.sampler.root The transaction starts a new trace (no incoming trace context).
distributed_tracing.sampler.remote_parent_sampled An incoming traceparent has the sampled flag set. Takes precedence over the next one.
distributed_tracing.sampler.remote_parent_not_sampled An incoming traceparent has the sampled flag 0.

Each sampler accepts:

Value Meaning
'adaptive' (default) New Relic's adaptive sampler, targeting a number of sampled transactions per minute.
'always_on' Always sample. For remote parents, the local transaction gets priority 2.
'always_off' Never sample.
{ trace_id_ratio_based: { ratio: 0.25 } } Sample a fixed fraction (0 to 1) of traces, decided from the trace ID.
{ adaptive: { sampling_target: 20 } } Adaptive sampling with a target specific to this sampler (an integer from 1 to 120).

The global adaptive target is distributed_tracing.sampler.adaptive_sampling_target (default 10 transactions per minute, range 1–120). The server can also send a sampling_target on connect.

exports.config = {
  distributed_tracing: {
    sampler: {
      adaptive_sampling_target: 20,
      root: { trace_id_ratio_based: { ratio: 0.1 } },
      remote_parent_sampled: 'always_on',
      remote_parent_not_sampled: 'always_off'
    }
  }
}

In a configuration file, the schema enforces these shapes: a sampling_target outside 1–120, a trace_id_ratio_based without ratio, or a string other than the three above fails validation — and a failed validation disables the agent. Run npx newrelic validate-config after changing samplers.

Samplers from environment variables

A sampler variable takes a string. For the two object forms, set the sampler to the type name and give the parameter in a second variable named after the sampler path:

# Ratio-based root sampler at 25%
export NEW_RELIC_DISTRIBUTED_TRACING_SAMPLER_ROOT=trace_id_ratio_based
export NEW_RELIC_DISTRIBUTED_TRACING_SAMPLER_ROOT_TRACE_ID_RATIO_BASED_RATIO=0.25

# Adaptive remote-parent-sampled sampler with its own target
export NEW_RELIC_DISTRIBUTED_TRACING_SAMPLER_REMOTE_PARENT_SAMPLED=adaptive
export NEW_RELIC_DISTRIBUTED_TRACING_SAMPLER_REMOTE_PARENT_SAMPLED_ADAPTIVE_SAMPLING_TARGET=30

Fallbacks to know:

  • trace_id_ratio_based without the ..._TRACE_ID_RATIO_BASED_RATIO variable falls back to adaptive.
  • adaptive with a ..._ADAPTIVE_SAMPLING_TARGET outside 1–120 falls back to plain adaptive; without the variable it stays plain adaptive.
  • The ratio is parsed with parseFloat and the target with parseInt.

Partial granularity (core tracing)

The schema also contains distributed_tracing.sampler.partial_granularity.* and full_granularity.enabled. Core tracing is currently disabled in the code: full_granularity.enabled is always forced to true, partial_granularity.enabled to false, and the other partial granularity settings are accepted but have no effect.

Span Limits

Setting Default Effect
span_events.max_samples_stored 2000 Span events kept per harvest. The server can adjust this through span_event_harvest_config.
span_events.attributes.* — Include/exclude rules for span attributes.

Infinite Tracing

Infinite Tracing streams every span to a trace observer over gRPC, where tail-based sampling happens, instead of sampling in the agent. Setting the observer host turns it on:

exports.config = {
  infinite_tracing: {
    trace_observer: {
      host: 'your-trace-observer-host',
      port: 443
    },
    span_events: {
      queue_size: 10000,
      batch_size: 750
    }
  }
}
Setting Default Effect
infinite_tracing.trace_observer.host '' Trace observer host. Non-empty enables Infinite Tracing.
infinite_tracing.trace_observer.port 443 Trace observer port.
infinite_tracing.span_events.queue_size 10000 Spans held in memory before dropping.
infinite_tracing.span_events.batch_size 750 Spans per batch.
infinite_tracing.batching true Send spans in batches (recordSpanBatch) instead of one by one (recordSpan).
infinite_tracing.compression true Compress gRPC payloads.

Infinite Tracing is forced off in serverless mode (the host and port are cleared and Infinite tracing is explicitly disabled in serverless_mode. is logged).

Manual Propagation

For transports the agent does not instrument, propagate context yourself with insertDistributedTraceHeaders and acceptDistributedTraceHeaders on the transaction handle. See Propagating trace context manually.

Serverless Mode

In serverless mode, distributed tracing needs account_id. Without it, the agent logs Using distributed tracing in serverless mode requires account_id be defined ... Disabling distributed tracing. With it, trusted_account_key defaults to account_id and primary_application_id to Unknown. See Serverless and Cloud.

See Also