Distributed Tracing
Distributed tracing follows a request across services: each service adds spans to a shared trace, linked by headers on outbound calls. In this agent it is enabled by default. This page explains header formats, sampling configuration (including the environment variable forms), span limits, and Infinite Tracing.
Turning It On or Off
| Setting | Default | Env var |
|---|---|---|
distributed_tracing.enabled |
true |
NEW_RELIC_DISTRIBUTED_TRACING_ENABLED |
span_events.enabled |
true |
NEW_RELIC_SPAN_EVENTS_ENABLED |
With distributed tracing disabled, getTraceMetadata() returns {}, linking metadata has no trace.id, and insertDistributedTraceHeaders() does nothing.
Header Formats
For outbound calls made with instrumented clients, the agent adds W3C Trace Context headers (traceparent, tracestate). It can also add the older New Relic newrelic header:
| Setting | Default | Effect |
|---|---|---|
distributed_tracing.exclude_newrelic_header |
true |
When true, the newrelic header is not sent; only W3C headers are. Set to false to send both, for example when calling services with old New Relic agents that only understand newrelic. |
Note: The schema description of this setting says "By default (when false) both W3C TraceContext ... and New Relic formats will be sent", but the actual default is
true. Trust the default: out of the box, only W3C headers are sent.
On inbound requests, traceparent is preferred; the newrelic header is only used when no traceparent is present. Trace context from other New Relic accounts is accepted only when it matches your trusted_account_key (sent by the server; configurable locally only in serverless mode).
Sampling
Not every trace is kept. The agent makes a sampling decision per trace, using one of several samplers depending on the situation:
| Sampler setting | Applies when |
|---|---|
distributed_tracing.sampler.root |
The transaction starts a new trace (no incoming trace context). |
distributed_tracing.sampler.remote_parent_sampled |
An incoming traceparent has the sampled flag set. Takes precedence over the next one. |
distributed_tracing.sampler.remote_parent_not_sampled |
An incoming traceparent has the sampled flag 0. |
Each sampler accepts:
| Value | Meaning |
|---|---|
'adaptive' (default) |
New Relic's adaptive sampler, targeting a number of sampled transactions per minute. |
'always_on' |
Always sample. For remote parents, the local transaction gets priority 2. |
'always_off' |
Never sample. |
{ trace_id_ratio_based: { ratio: 0.25 } } |
Sample a fixed fraction (0 to 1) of traces, decided from the trace ID. |
{ adaptive: { sampling_target: 20 } } |
Adaptive sampling with a target specific to this sampler (an integer from 1 to 120). |
The global adaptive target is distributed_tracing.sampler.adaptive_sampling_target (default 10 transactions per minute, range 1–120). The server can also send a sampling_target on connect.
exports.config = {
distributed_tracing: {
sampler: {
adaptive_sampling_target: 20,
root: { trace_id_ratio_based: { ratio: 0.1 } },
remote_parent_sampled: 'always_on',
remote_parent_not_sampled: 'always_off'
}
}
}
In a configuration file, the schema enforces these shapes: a sampling_target outside 1–120, a trace_id_ratio_based without ratio, or a string other than the three above fails validation — and a failed validation disables the agent. Run npx newrelic validate-config after changing samplers.
Samplers from environment variables
A sampler variable takes a string. For the two object forms, set the sampler to the type name and give the parameter in a second variable named after the sampler path:
# Ratio-based root sampler at 25%
export NEW_RELIC_DISTRIBUTED_TRACING_SAMPLER_ROOT=trace_id_ratio_based
export NEW_RELIC_DISTRIBUTED_TRACING_SAMPLER_ROOT_TRACE_ID_RATIO_BASED_RATIO=0.25
# Adaptive remote-parent-sampled sampler with its own target
export NEW_RELIC_DISTRIBUTED_TRACING_SAMPLER_REMOTE_PARENT_SAMPLED=adaptive
export NEW_RELIC_DISTRIBUTED_TRACING_SAMPLER_REMOTE_PARENT_SAMPLED_ADAPTIVE_SAMPLING_TARGET=30
Fallbacks to know:
trace_id_ratio_basedwithout the..._TRACE_ID_RATIO_BASED_RATIOvariable falls back toadaptive.adaptivewith a..._ADAPTIVE_SAMPLING_TARGEToutside 1–120 falls back to plainadaptive; without the variable it stays plainadaptive.- The ratio is parsed with
parseFloatand the target withparseInt.
Partial granularity (core tracing)
The schema also contains distributed_tracing.sampler.partial_granularity.* and full_granularity.enabled. Core tracing is currently disabled in the code: full_granularity.enabled is always forced to true, partial_granularity.enabled to false, and the other partial granularity settings are accepted but have no effect.
Span Limits
| Setting | Default | Effect |
|---|---|---|
span_events.max_samples_stored |
2000 |
Span events kept per harvest. The server can adjust this through span_event_harvest_config. |
span_events.attributes.* |
— | Include/exclude rules for span attributes. |
Infinite Tracing
Infinite Tracing streams every span to a trace observer over gRPC, where tail-based sampling happens, instead of sampling in the agent. Setting the observer host turns it on:
exports.config = {
infinite_tracing: {
trace_observer: {
host: 'your-trace-observer-host',
port: 443
},
span_events: {
queue_size: 10000,
batch_size: 750
}
}
}
| Setting | Default | Effect |
|---|---|---|
infinite_tracing.trace_observer.host |
'' |
Trace observer host. Non-empty enables Infinite Tracing. |
infinite_tracing.trace_observer.port |
443 |
Trace observer port. |
infinite_tracing.span_events.queue_size |
10000 |
Spans held in memory before dropping. |
infinite_tracing.span_events.batch_size |
750 |
Spans per batch. |
infinite_tracing.batching |
true |
Send spans in batches (recordSpanBatch) instead of one by one (recordSpan). |
infinite_tracing.compression |
true |
Compress gRPC payloads. |
Infinite Tracing is forced off in serverless mode (the host and port are cleared and Infinite tracing is explicitly disabled in serverless_mode. is logged).
Manual Propagation
For transports the agent does not instrument, propagate context yourself with insertDistributedTraceHeaders and acceptDistributedTraceHeaders on the transaction handle. See Propagating trace context manually.
Serverless Mode
In serverless mode, distributed tracing needs account_id. Without it, the agent logs Using distributed tracing in serverless mode requires account_id be defined ... Disabling distributed tracing. With it, trusted_account_key defaults to account_id and primary_application_id to Unknown. See Serverless and Cloud.