AI Monitoring

When an application calls a supported LLM SDK (OpenAI, Anthropic, Google Gen AI, Amazon Bedrock, LangChain, LangGraph, Google ADK, the Model Context Protocol SDK — see Supported Modules), the agent records Llm* events, spans, and metrics. This page explains how AI monitoring is switched on, whether message content is recorded, and the three API methods that enrich AI data.

Is AI Monitoring On?

Config#isAiMonitoringEnabled() decides, in this order:

  1. If the connect reply contains collect_ai: false (AI monitoring disabled for the account), it is off.
  2. If ai_monitoring.enabled is explicitly true or false, that value wins. This setting is deprecated and logs a deprecation warning when set.
  3. Otherwise ai_monitoring.basic_telemetry.enabled decides. Its default is true.

So with default settings AI monitoring is on, unless your account disables it. High security mode forces ai_monitoring.enabled to false.

Is Message Content Recorded?

Prompts, completions, and embedding inputs are only included when content recording is on (Config#shouldRecordAiContent()):

ai_monitoring.record_content.enabled Content recorded?
true Yes (if AI monitoring is on).
false No.
unset (default) Only when you set the deprecated ai_monitoring.enabled: true locally and the server did not send ai_monitoring.enabled. This preserves the old behavior for existing users.

Migrating from the deprecated flag? The deprecation warning (logged and emitted as a Node.js DeprecationWarning) points to ai_monitoring.basic_telemetry.enabled as the replacement and advises also setting ai_monitoring.record_content.enabled to true if you want to keep recording message content.

exports.config = {
  ai_monitoring: {
    basic_telemetry: { enabled: true },
    record_content: { enabled: true },
    streaming: { enabled: true }
  }
}

ai_monitoring.streaming.enabled (default true) controls instrumentation of streamed responses; when false, streams are not instrumented.

recordLlmFeedbackEvent(params)

Records end-user feedback (thumbs up/down, a rating) about an LLM response, as a custom event of type LlmFeedbackMessage.

Parameter Type Description
traceId string Required. The trace ID of the request that produced the response, from getTraceMetadata().
category string A tag for the feedback (for example informative).
rating string How useful the response was (for example 👍 or 3).
message string Optional. Free-text feedback. Default ''.
metadata object Optional. Extra key-value pairs added to the event.
// When generating the answer
const { traceId } = newrelic.getTraceMetadata()
res.json({ answer, traceId })

// Later, when the user rates it (in another request)
app.post('/feedback', (req, res) => {
  newrelic.recordLlmFeedbackEvent({
    traceId: req.body.traceId,
    category: 'helpfulness',
    rating: req.body.rating,
    message: req.body.comment,
    metadata: { model: 'gpt-4o' }
  })
  res.sendStatus(204)
})

The event has the fields id (random), trace_id, category, rating, message, and ingest_source: 'Node', plus your metadata. Because it is recorded with recordCustomEvent, it obeys the custom event rules: it is dropped in high security mode, when api.custom_events_enabled is false, and metadata values must be strings, numbers, or booleans. Metadata keys that collide with the event fields are overwritten by the event fields.

The call is ignored, with a warning, when traceId is missing, when AI monitoring is off, or outside a transaction (recordLlmFeedbackEvent must be called within the scope of a transaction).

setLlmTokenCountCallback(callback)

Registers a synchronous function that returns token counts for LLM content. It is typically needed when content recording is off and the SDK response does not include usage counts, but you still want token metrics.

const { encoding_for_model } = require('tiktoken') // any tokenizer you choose

newrelic.setLlmTokenCountCallback((model, content) => {
  const enc = encoding_for_model(model)
  return enc.encode(content).length
})
Callback argument Description
model The model name (for example gpt-4o-mini).
content The prompt or completion text.

It must return an integer. An async function or a non-function is rejected (Llm token count callback must be a synchronous function, callback will not be registered.). Each call replaces the previous callback.

withLlmCustomAttributes(context, callback)

Runs callback with extra attributes that are added to every LLM event recorded inside it, and returns the callback's return value.

const OpenAI = require('openai')
const client = new OpenAI()

await newrelic.withLlmCustomAttributes({ 'llm.conversationId': convId, tenant: 'acme' }, async () => {
  return client.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: 'Tell me about Node.js.' }]
  })
})

Rules applied to context:

  • Keys must start with llm.. Other keys are renamed (tenant becomes llm.tenant, with a warning).
  • Values that are objects or functions are dropped with Invalid attribute type for <key>. Skipped.
  • The context object you pass is mutated by these rules.
  • Calls can be nested: inner contexts are merged over outer ones.
  • Outside a transaction, it logs a warning and simply calls callback() without attributes. Without a function callback, it returns undefined.

See Also