Callbacks and events

RTVI clients listen for messages and events from the bot via the transport. This allows you to respond to changes in state, errors, and other events.

Callbacks

RTVI clients exposes callback hooks that can be defined on the constructor.

const rtviClient = new RTVIClient({
  // ...
  callbacks: {
    onBotReady: bot_ready_handler_func,
  },
});

Messages and errors

onGenericMessage · data:unknown

If the client receives an unknown message type from the transport (see messages and events), it can be handled here.

onMessageError · message:RTVIMessage

Response error when an action fails or an unknown message type is sent from the client.

onError · message:RTVIMessage

Error signalled by the bot. This could be due to a malformed config update or an unknown action dispatch or the ability to complete the requested action.

State and connectivity

onTransportStateChanged · state:TransportState

Provides a TransportState string representing the connectivity state of the local client. See transports for state explanation.

onConnected

Local user successfully established a connection to the transport.

onDisconnected

Local user disconnected from the transport, either intentionally by calling rtviClient.disconnect() or due to an error.

onBotConnected

Bot connected to the transport and is configuring. Note: bot connectivity does not infer that its pipeline is yet ready to run. Please use onBotReady instead.

onParticipantJoined · participant: Participant

A non-bot participant joined the session.

onParticipantLeft · participant: Participant

A non-bot participant left the session. Note: excluded local participant.

onBotReady · botReadyData:BotReadyData

The bot has been instantiated, its pipeline is configured, and it is receiving user media and interactions. This method is passed a BotReadyData object, which contains a config RTVIClientConfigOption[] array and the RTVI version number. It is recommended to hydrate your application with the passed config in case the bot alters any configuration based on its implementation. Since the bot is remote and may be using a different version of RTVI than the client, you can use the passed version string to check for compatibility.

onBotDisconnected · participant: Participant

Bot disconnected from the transport. This may occur due to session expiry, a pipeline error or because the local participant left the session.

Configuration

onConfigUpdated · config:RTVIClientConfigOption[]

Sent when the bots configuration changes. This is most likely in response to a user updateConfig action, but can occur within a bots pipeline in response to actions or messages.

onConfigDescribe · configDescription:unknown

A list of available configuration options for each service. Sent in response to a user describeConfig() call.

"config": [
  {
      "service": "llm",
      "options": [
        { "name": "model", "type": "string" },
        { "name": "messages", "type": "array" },
        ...
      ]
  },
  {
      "service": "tts",
      "options": [
        { "name": "voice", "type": "string" },
        ...
      ]
  },
  ...
]

Configuration

onActionsAvailable · actions: unknown

Response to describeActions() call. Lists available actions for the bot and their associated arguments and types.

Media and devices

onAvailableMicsUpdated · mics:MediaDeviceInfo[]

Lists available local media microphone devices. Triggered when a new device becomes available or in response to rtviClient.initDevices().

onAvailableCamsUpdated · cams:MediaDeviceInfo[]

Lists available local media camera devices. Triggered when a new device becomes available or in response to rtviClient.initDevices().

onMicUpdated · mic:MediaDeviceInfo

User selected a new microphone as their selected/active device.

onCamUpdated · cam:MediaDeviceInfo

User selected a new camera as their selected/active device.

onTrackStarted · track: MediaStreamTrack, participant:Participant

Media track from a local or remote participant was started and playable. Can be either an audio or video track.

onTrackStopped · track: MediaStreamTrack, participant:Participant

Media track from a local or remote participant was stopped and no longer playable.

Audio and Voice Activity

onLocalAudioLevel · level:number

Local audio gain level (0 to 1).

onRemoteAudioLevel · level: number, participant: Participant

Remote audio gain level (0 to 1). Note: if more than one participant is connected to the transport, the participant property details the associated peer.

onBotStartedSpeaking

The bot started speaking/sending speech audio.

onBotStoppedSpeaking

The bot stopped speaking/sending speech audio.

onUserStartedSpeaking

The local user started speaking. This method is more reliable than using audio gain and is the result of the bot’s VAD (voice activity detection) model. This provides a more accurate result in noisy environments.

onUserStoppedSpeaking

The local user stopped speaking, indicated by the VAD model.

Transcription

onUserTranscript · transcript:TranscriptData

Transcribed local user input (both partial and final).

Callback receives a TranscriptData object:

export type TranscriptData = {
  text: string;
  final: boolean;
  timestamp: string;
  user_id: string;
};

onBotTranscript · text:BotLLMTextData

Finalized bot output text generated by the LLM. Sentence aggregated.

onBotLLmText · text:BotLLMTextData

Streamed LLM token response text generated by the LLM service. Please note that this event is not dispatched when in a connected state. This callback is intended to be used for offline, single-turn actions, where not TTS service is connected.

When connected:

• If you want to align words as they are spoken by the bot, use onBotTtsText instead.

• If you want the full text of the LLM response, use onBotTranscript.

onBotLlmStarted

LLM service inference started.

onBotLlmStopped

LLM service inference concluded.

onBotTtsText · text: BotTTSTextData

If your TTS service supports streamed responses over sockets, the text parameter contains the words from TTS service as they are spoken If you are using a HTTP based TTS service, the text parameter will contain the full text of the TTS response.

onBotTtsStarted

TTS service started inference.

onBotTtsStarted

TTS service inference concluded.

Other

onMetrics · default: "data:PipecatMetricsData"

Pipeline data provided by Pipecat. Please see Pipecat documentation for more information.

onStorageItemStored · default: "data: StorageItemStoredData"

A message or messages item was saved to the storage / database. Assumed you have a storage context processor as part of your bot pipeline. Data references the saved message.

Events

Callbacks methods are also defined as events that map to messages/actions received from the bot. The following events are available:

MessageError = "messageError";
Error = "error";

Connected = "connected";
Disconnected = "disconnected";
TransportStateChanged = "transportStateChanged";

Config = "config";
ConfigDescribe = "configDescribe";
ActionsAvailable = "actionsAvailable";

ParticipantConnected = "participantConnected";
ParticipantLeft = "participantLeft";
TrackStarted = "trackStarted";
TrackedStopped = "trackStopped";

AvailableCamsUpdated = "availableCamsUpdated";
AvailableMicsUpdated = "availableMicsUpdated";
CamUpdated = "camUpdated";
MicUpdated = "micUpdated";

BotConnected = "botConnected";
BotReady = "botReady";
BotDisconnected = "botDisconnected";
BotStartedSpeaking = "botStartedSpeaking";
BotStoppedSpeaking = "botStoppedSpeaking";
RemoteAudioLevel = "remoteAudioLevel";

UserStartedSpeaking = "userStartedSpeaking";
UserStoppedSpeaking = "userStoppedSpeaking";
LocalAudioLevel = "localAudioLevel";

UserTranscript = "userTranscript";
UserText = "userText";
BotTranscript = "botTranscript";

BotText = "botText";
BotLlmStarted = "botLlmStarted";
BotLlmStopped = "botLlmStopped";

BotTtsText = "botTtsText";
BotTtsStarted = "botTtsStarted";
BotTtsStopped = "botTtsStopped";

LLMFunctionCall = "llmFunctionCall";
LLMFunctionCallStart = "llmFunctionCallStart";
LLMJsonCompletion = "llmJsonCompletion";

Metrics = "metrics";
StorageItemStored = "storageItemStored";

Handlers for these can be bound on the client like so:

import { RTVIEvent } from "realtime-ai";

function handleBotReady() {
  console.log("Bot is ready!");
}

// Bind an event handler
rtviClient.on(RTVIEvent.BotReady, handleBotReady);
// or
rtviClient.on("botReady", handleBotReady);

// Unbind an event handler
rtviClient.off(RTVIEvent.BotReady, handleBotReady);