Client Methods

The RTVI clients expose a number of core methods. These are documented below.

RTVI implementations will often publish helper libraries that extend the core client with additional methods and events. For more on helpers, see here.

Starting and stopping a session

start()

Start a session. This method sends a web request to baseUrl and initializes your transport class.

await voiceClient.start();

Calling start() asynchronously will resolve when the bot and client signal that they are ready. See messages and events. If you want to call start() without await, you can use the onBotReady callback or BotReady event to know when you can interact with the bot.

Attempting to call start() when the transport is not in a ready state will throw an error.

This method can be try / catched to handle errors at startup:

try {
  await voiceClient.start();
} catch (error) {
  console.error(error);
}

disconnect()

Disconnects from the active session. In most implementations, the bot will exit when the client disconnects. (Note however, that this is implementation-specific. The RTVI standard does not mandate bot behavior.)

await voiceClient.disconnect();

Bot configuration

Methods to query and update the configuration of the bot the client is connected to.

See: configuration.

getBotConfig()

Returns a Promise that resolves with the bot’s current configuration.

config = await voiceClient.getBotConfig();

A typical config might look like this.

[
  {
    "service": "vad",
    "options": [{ "name": "params", "value": { "stop_secs": 3.0 } }]
  },
  {
    "service": "tts",
    "options": [
      { "name": "voice", "value": "79a125e8-cd45-4c13-8a67-188112f4dd22" }
    ]
  },
  {
    "service": "llm",
    "options": [
      {
        "name": "model",
        "value": "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo"
      },
      {
        "name": "initial_messages",
        "value": [
          {
            "role": "system",
            "content": `You are a assistant called ExampleBot. You can ask me anything.
              Keep responses brief and legible.
              Your responses will converted to audio. Please do not include any special characters in your response other than '!' or '?'.
              Start by briefly introducing yourself.`
          }
        ]
      },
      { "name": "run_on_config", "value": true }
    ]
  }
]

The client’s current config state can be accessed with voiceClient.config. This property is hydrated with the bot’s config after start() is called, and after any updateConfig request.

updateConfig()

Update the bot’s config. Passing a partial config is fine. Only the individual options that are passed to the updateConfig() call will be updated. You can omit any services you don’t want to make changes to. You can omit any options within a service that you don’t want to change.

Returns a Promise that resolves with the full updated configuration.

On resolve, will trigger the onConfigUpdate callback and ConfigUpdated event.

On reject, will trigger the onMessageError callback and MessageError event.

config · VoiceClientConfigOption[] · required

VoiceClientConfigOption[] partial object with the new configuration

interrupt · boolean · default: "false"

Should the new config be immediately applied (interrupting the bot), or wait until the current pipeline concludes.

const new_config = await voiceClient.updateConfig(
  [{ service: "vad", options: [{ stop_secs: 0.5 }] }],
  true | false
);

If called asynchronously, will resolve with either a config or error-response message.

describeConfig()

Get the available config options for the bot the client is connected to.

Returns a Promise that resolve’s with the bot’s configuration description.

configuration_metadescription = await voiceClient.describeConfig();

getServiceOptionsFromConfig()

VoiceClientConfigOption | undefined

serviceKey · string · required

Service key / string identifier e.g. llm

Config helper method that returns configuration options for specified service key.

Returns new instance (copy) of the service options array.

const llmServiceConfig: VoiceClientConfigOption =
  voiceClient.getServiceOptionsFromConfig("llm");
// > [{ service: "llm", options: [ { name: "model", value: "GPT-4o" }, ... ] }]

Returns undefined when an invalid service key is provided.

getServiceOptionValueFromConfig()

unknown | undefined

serviceKey · string · required

Service key / string identifier e.g. llm

option · string · required

Service option to retrieve e.g. model

Config helper method that returns service options for specified service key ad option name.

Returns new instance (copy) of the client config.

const llmModel = voiceClient.getServiceOptionsFromConfig("llm", "model");
// > "GPT-4o"

Returns undefined when an invalid service key or unknown option value is provided.

setServiceOptionInConfig()

VoiceClientConfigOption[]

serviceKey · string · required

Service key / string identifier e.g. llm

serviceKey · ConfigOption | ConfigOption[] · required

Service option value to update, or array of service options.

config · VoiceClientConfigOption[]

Config object to update, otherwise uses a clone of the current voice client config.

Config helper method that sets a new value for a service option in the config.

Returns new instance of config with updated option(s) for specified service key and option name

Does not mutate the voice client config.

You can pass a single ConfigOption object…

const newConfig = voiceClient.setServiceOptionInConfig("llm", {
  name: "model",
  value: "new-model",
});
// > [{ service: "llm", options: [ { name: "model", value: "new-model" }, ... ] }]

await voiceClient.updateConfig(newConfig);

…or an array of ConfigOption objects.

const newConfig = voiceClient.setServiceOptionInConfig("llm", [
  { name: "model", value: "new-model" },
  { name: "initial_messages", value: [...] },
]);

// > [{
// >    service: "llm", options: [
// >      { name: "model", value: "new-model" },
// >      { name: "initial_messages", value: [ ... ] },
// >      ...]
// > }]

await voiceClient.updateConfig(newConfig);

You can pass an optional config object to update, otherwise uses a clone of the current voice client config.

const workInProgressConfig = [...];
const newConfig = setServiceOptionInConfig("llm", [
  { name: "model", value: "new-model" },
  { name: "initial_messages", value: [...] },
], workInProgressConfig);

await voiceClient.updateConfig(newConfig);

setConfigOptions()

VoiceClientConfigOption[]

configOptions · VoiceClientConfigOption[] · required

Array of service options to update.

config · VoiceClientConfigOption[]

Config object to update, otherwise uses a clone of the current voice client config.

Config helper method that sets new values for multiple service options in the config.

Convenience method for calling setServiceOptionInConfig multiple times.

You can pass an optional config object to update, otherwise uses a clone of the current voice client config.

Returns new instance (copy) of the client config.

const newConfig = voiceClient.setConfigOptions([
  { service: "llm", options: [
    { name: "model", value: "new-model" },
    { name: "initial_messages", value: [...] },
  ] },
  { service: "tts", options: [
    { name: "voice", value: "new-voice" },
  ] },
]);

await updateConfig(newConfig);

Messages

Dispatches a VoiceMessage data object to the bot.

See: messages and events.

sendMessage()

message · VoiceMessage

A VoiceMessage object containing the type of message and data object to dispatch.

Messages are sychronous events that are dispatched to the bot. They do not return a Promise and cannot be awaited.

Actions

Dispatches a service-specific action to the bot.

See: actions.

action()

Dispatch an action message to the bot.

What actions a bot responds to depend on how the bot is implemented. So RTVI defines an actions protocol, but does not define any specific actions. Helper libraries will often install convenience methods that make it less verbose (and more type-safe) to call actions.

action · ActionData

The action to dispatch.

Returns Promise<VoiceMessageActionResponse>.

// send a tts:say action using RTVI primitives

response = await voiceClient.action({
  service: "tts",
  action: "say",
  arguments: [{ name: "text", value: "Say 'Hello world!'" }],
});

// a bot helper library might install convenience methods so that
// the bot's supported actions could be invoked like this, instead
response = await voiceClient.say("Hello world!");

describeActions()

Describe the actions implemented by the bot that the client is connected to.

Returns a Promise that resolves with the bot’s actions description.

actions_metadescription = await voiceClient.describeActions();

Devices (microphones and cameras)

initDevices()

Initialize the media device selection machinery in the web browser. This method can be called before start() to test and switch between camera and microphone sources.

await voiceClient.initDevices();

getAllMics()

Returns a list of available microphones in the form of MediaDeviceInfo[].

mic_device_list = voiceClient.getAllMics();

getAllCams()

Returns a list of available cameras in the form of MediaDeviceInfo[].

cam = voiceClient.getAllCams();

updateMic()

Switches to the microphone identified by the provided micId, which should map a deviceId in the list returned from getAllMics().

micId · string

deviceId

voiceClient.updateMic(deviceId);

updateCam()

Switches to the camera identified by the provided camId, which should map a deviceId in the list returned from getAllCams().

camId · string

deviceId

voiceClient.updateMic(deviceId);

enableMic()

Turn on or off (unmute or mute) the client mic input.

enable · boolean

voiceClient.enableMic(true);

enableCam()

Turn on or off the client cam input.

enable · boolean

voiceClient.enableCam(true);

Tracks (audio and video)

tracks()

Returns available MediaStreamTrack objects for both the client and the bot.

live_tracks_list = voiceClient.tracks()

// return type is:
{
  local: {
    audio?: MediaStreamTrack;
    video?: MediaStreamTrack;
  },
  bot?: {
    audio?: MediaStreamTrack;
    video?: MediaStreamTrack;
  }
}