Quickly Voice Your Apps with NVIDIA NIM Microservices for Speech and Translation

Use the following script to start the ASR NIM: 

export CONTAINER_NAME=parakeet-ctc-1.1b-asr

docker run -it --rm --name=$CONTAINER_NAME \
  --runtime=nvidia \
  --gpus '"device=0"' \
  --shm-size=8GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NIM_MANIFEST_PROFILE=9136dd64-4777-11ef-9f27-37cfd56fa6ee \
  -e NIM_HTTP_API_PORT=9000 \
  -e NIM_GRPC_API_PORT=50051 \
  -p 9000:9000 \
  -p 50051:50051 \
  -v "$LOCAL_NIM_CACHE:/home/nvs/.cache/nim" \
  nvcr.io/nim/nvidia/parakeet-ctc-1.1b-asr:1.0.0

If the LOCAL_NIM_CACHE directory is empty (such as the first time you execute this command), it may take 20-30 minutes to complete. During this time, NIM will download the acoustic (offline, streaming optimized for latency, and streaming optimized for throughput) and punctuation models as .tar.gz files, then uncompress them within the container. 

You might want to change the NIM_MANIFEST_PROFILE parameter, depending on your GPU type. By default, the NIM container downloads ONNX-formatted model files that run on any sufficiently advanced NVIDIA GPU. However, if you change this parameter appropriately when starting the ASR or TTS NIM, you’ll instead download NVIDIA TensorRT-formatted model files that have been optimized for NVIDIA H100, A100, or L40 GPUs. This results in faster inference on one of the supported GPUs. Optimized NIM support for additional GPU types is forthcoming. 

Table 1 shows the supported NIM_MANIFEST_PROFILE values for the ASR NIM for the Parakeet CTC Riva 1.1B En-US model:  

You can also find this table in the Supported Models section of the Getting Started page in the ASR NIM documentation. Note the slight difference between the model name and the container name. Unlike some other NIM microservices, the speech and translation NIM microservices do not support the list-model-profiles utility, meaning you can’t access the list of valid NIM_MANIFEST_PROFILE values through the docker CLI. 

Launch the TTS NIM

After stopping the ASR NIM, either with Ctrl+C in the same terminal or with docker stop or docker container stop in a different terminal, launch the TTS NIM:

export CONTAINER_NAME=fastpitch-hifigan-tts

docker run -it --rm --name=$CONTAINER_NAME \
  --runtime=nvidia \
  --gpus '"device=0"' \
  --shm-size=8GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NIM_MANIFEST_PROFILE=3c8ee3ee-477f-11ef-aa12-1b4e6406fad5 \
  -e NIM_HTTP_API_PORT=9000 \
  -e NIM_GRPC_API_PORT=50051 \
  -p 9000:9000 \
  -p 50051:50051 \
  -v "$LOCAL_NIM_CACHE:/home/nvs/.cache/nim" \
  nvcr.io/nim/nvidia/fastpitch-hifigan-tts:1.0.0

From Table 2, choose a suitable NIM_MANIFEST_PROFILE value for the FastPitch HifiGAN Riva En-US model: 

GPU	NIM_MANIFEST_PROFILE
Generic	3c8ee3ee-477f-11ef-aa12-1b4e6406fad5
NVIDIA H100	bbce2a3a-4337-11ef-84fe-e7f5af9cc9af
NVIDIA A100	5ae1da8e-43f3-11ef-9505-e752a24fdc67
NVIDIA L40	713858f8-43f3-11ef-86ee-4f6374fce1aa

You can also find this table in the Supported Models section of the Getting Started page in the TTS NIM documentation. Note the slight difference between the model name and the container name. 

Starting the TTS NIM should be much faster than starting the ASR NIM, because the constituent models take up far less space in total. However, because we’re using the same LOCAL_NIM_CACHE directory for both the ASR and TTS NIMs, the TTS NIM will launch both the ASR and TTS models.

Connect the speech NIM microservices to the RAG pipeline

The RAG web app is part of the NVIDIA Generative AI Examples GitHub repo. After cloning the repo, the main thing you need to do is edit  RAG/examples/basic_rag/langchain/docker-compose.yaml. Set the PLAYGROUND_MODE for the rag-playground service to speech and add the following environment variables to that service: 

services:
  ...
  rag-playground:
    ...
    environment: 
      ...
      RIVA_API_URI: <riva-ip-address>:50051
      TTS_SAMPLE_RATE: 48000

If you wish to set RIVA_API_URI with an overridable default value in the docker-compose file (in the format shown below), do not put quotation marks around the default value. If you do, the Python os module will include them in the string defining the Riva URI, which will cause problems.

RIVA_API_URI: ${RIVA_API_URI:-<riva-ip-address>:50051}

Even if you’re running the RAG example and the speech NIM microservices on the same machine, you need to specify the IP address or permanent hostname; localhost won’t work here.

See more detailed instructions on adding ASR and TTS capabilities to a RAG pipeline in the Generative AI Examples repo. 

Once the docker-compose file is edited appropriately, run the following command in the same directory to start the container network.

docker compose up -d --build

Also verify that each container within the network is running: 

docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Status}}"

Test the speech NIM and RAG integration

To test your setup, navigate to localhost:8090 in your browser. If you’re running the RAG app on a remote system without port forwarding, use <remote-IP-address>:8090 instead. The interface (Figures 1 and 2) enables you to query the large language model (LLM) by text or voice, and receive a spoken response. If the models are available, you can change the ASR and TTS languages using menu options. 

For document-based queries, click on the Knowledge Base tab near the top right of the page. Here, you can upload PDF, plain text, or markdown files. The contents are embedded as multidimensional vectors and indexed in a vector database, enabling the LLM to answer questions based on this new information. 

To provide an example, upload a PDF version of the recent press release, NVIDIA Blackwell Platform Arrives to Power a New Era of Computing. Despite being published over five months ago, the default LLM for this sample web app is not pretrained of this information.

Next, return to the Converse tab, click the microphone button, and ask the app, “How many transistors does the NVIDIA Blackwell GPU contain?” Without the knowledge base, the LLM is unable to provide the correct answer (Figure 1).

Now, with the knowledge base active, ask the same question again. This time, leveraging the full RAG pipeline, the LLM answers correctly based on the newly embedded information (Figure 2).

Get started adding multilingual speech AI to your apps

In this post, you’ve learned to set up NVIDIA speech and translation NIM microservices and test them directly through your browser by using the interactive speech and translation model interfaces. You have dived into the flexibility of deploying NIM speech and translation microservices and integrating them into a RAG pipeline for document-based knowledge retrieval with synthesized voice responses. 

Ready to add powerful multilingual speech AI to your own applications? Try speech NIM microservices to experience the ease of integrating ASR, NMT, and TTS into your pipelines. Explore the APIs and see how these NIM microservices can transform your applications to scalable, real-time voice services for worldwide users.