Use Language Translation Session

After establishing a WebRTC connection in Connect With WebRTC, you can initialize a conversation for simultaneous language translation with Sanas.

First, start by sending a reset message on the WebRTC data channel:

{
  type: 'reset'
  reset: {
    id: string                 // This should be an arbitrary UUID generated by you
    lang_in: string | null     // The original language code in ISO 639-1 format, automatically inferred if null
    lang_out: string           // The translation language code in ISO 639-1 format
    glossary?: string[] | null // An optional list of brand names to preserve
    detect_languages: bool     // Enable language identification pipeline to get language probabilities at regular intervals
  }
}

Then, wait to receive a ready message back from the server on the WebRTC data channel with a matching id. The message should look like so:

{
  type: 'ready'
  ready: {
    id: string            // The id that you specified in the reset message
  }
}

You are now ready to start feeding audio through the WebRTC audio channel! You will then start receiving audio back.

Additionally, you will receive the following messages on the data channel for transcripts and translations:

{
  type: 'transcription'
  transcription: {
    lang: str | null      // Automatically identified language code in ISO 639-1 format
    complete: Word[]      // Delta of words that will never change. Only sent once, it is your jobs to store them once received to recreate the full utterance.
    partial: Word[]       // Words that might still change. Word will be repeatedly sent here until complete.
    utterance_idx: string // Index representing what utterance is being spoken. Increments with each new utterance. Useful if client wants to display each utterance in its own chat bubble.
  }
}

{
  type: 'translation'
  transcription: {
    complete: Word[]      // Words that will never change. Only sent once, it is your jobs to store them once received.
    partial: Word[]       // Words that might still change. Word will be repeatedly sent here until complete.
    utterance_idx: string // Index representing what utterance is being spoken. Increments with each new utterance. Useful if client wants to display each utterance in its own chat bubble.
  }
}

{
  type: 'speech_delimiter'
  speech_delimiter: {
    time: number            // Timestamp in seconds representing the current playback position in the audio stream
    transcription: {
      utterance_idx: number // Approximate index of the source utterance currently being rendered
      word_idx: number      // Approximate word-level offset; helps track which part of the source text aligns with the current audio
      char_idx: number      // Approximate character-level offset within the word for high-granularity UI highlighting
    }
    translation: {
      utterance_idx: number // The specific utterance index currently being played in the output audio
      word_idx: number      // The exact word index where the audio playback currently sits; marks the boundary between 'spoken' and 'pending'
      char_idx: number.     // The exact character index within the word; provides a frame-accurate split point for the translation stream
    }
  }
}

{
  type: 'languages' // This message is only returned when `detect_languages` is enabled in the reset message
  languages: {
    languages: IdentifiedLanguage[] // List of languages with their probabilities of being identified
  }
}

With the Word object like so:

{
  word: string        // The text content of the word
  start: number       // When the word started speaking, in seconds since the start of the session
  end: number         // When the word finished speaking, in seconds since the start of the session
  probability: number // (beta) Certainty of a word being correct
}

And with the IdentifiedLanguage object like so:

{
  short_code: string  // ISO 639-1 language code
  name: number        // English name for this language
  probability: number // Certainty of a language being detected (0-1)
}

To end a session, simply stop the WebRTC connection.