TABLE OF CONTENTS

• Introduction
• Prerequisites
  • Install Nvidia Docker
• Steps
  • Step 1: Install Jetson Runtime Containers
  • Step 2: Install and Run the Ollama Container
  • Step 3: Pre-load the Target LLM
  • Step 4: Install Lazydocker for Container Management
  • Step 5: Enhance Your Shell Environment
  • Step 6: Create a Quickstart Script
  • Step 7: Install Additional Tools
  • Step 8: Prepare Supporting Scripts and Dependencies
• Conclusion

Introduction

Large Language Models (LLMs) are revolutionizing natural language processing by enabling sophisticated text generation and understanding. This demonstration leverages the Cardshark platform to showcase how an LLM container can be deployed using Jetson runtime containers. The process involves installing essential container tools, deploying an LLM image (using a placeholder such as “llama3.2”), and integrating management utilities to streamline your workflow.

This recipe uses several key projects:

• Jetson Runtime Containers: A set of scripts and configurations for running containers on Nvidia Jetson platforms. GitHub Repository for jetson containers
• Ollama Container Image: An LLM container image deployed using the Jetson containers utility. GitHub Repository for Ollama
• Lazydocker: A terminal UI for Docker, which provides an easy-to-use interface to manage containers. GitHub Repository for LazyDocker
• Nvidia Container Toolkit: Facilitates the use of Nvidia GPUs within Docker containers. Nvidia Docker

Prerequisites

Before beginning, ensure that your Cardshark device is running a compatible version of the Jetson software stack and that you have network connectivity to download the necessary packages.

Install Nvidia Docker

curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update
sudo apt-get install -y nvidia-docker2
sudo pkill -SIGHUP dockerd

Steps

Step 1: Install Jetson Runtime Containers

Clone the Jetson containers repository, execute the installation script, and install the Nvidia container toolkit:

cd ~/
git clone https://github.com/dusty-nv/jetson-containers
bash jetson-containers/install.sh
sudo apt-get install -y nvidia-container-toolkit

Step 2: Install and Run the Ollama Container

Launch the Ollama container using the Jetson containers utility. The command uses an autotag to select the appropriate container image:

jetson-containers run --name ollama $(autotag ollama)

Step 3: Pre-load the Target LLM

Inside the Ollama container, pre-load the desired LLM. In this example, we use “llama3.2” as a placeholder:

ollama run llama3.2

Step 4: Install Lazydocker for Container Management

Lazydocker provides a simple graphical interface to manage Docker containers. Download and install Lazydocker as follows:

wget https://github.com/jesseduffield/lazydocker/releases/download/v0.24.1/lazydocker_0.24.1_Linux_arm64.tar.gz
tar -xvzf lazydocker_0.24.1_Linux_arm64.tar.gz
chmod +x lazydocker
sudo mv lazydocker /usr/local/bin

Step 5: Enhance Your Shell Environment

For a more colorful SSH experience, update your bash profile to source your .bashrc file. Open your bash profile:

nano ~/.bash_profile

Append the following lines to the bottom of the file, then save and reconnect:

if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

Step 6: Create a Quickstart Script

To simplify re-launching the Ollama container, add the command to a quickstart script:

echo "jetson-containers run --name ollama $(autotag ollama)" >> start.sh
chmod +x start.sh

Step 7: Install Additional Tools

Install tmux for terminal multiplexing:

sudo apt install tmux

Step 8: Prepare Supporting Scripts and Dependencies

Copy over the necessary Cardshark script (cardshark.py) once it is available in the repository. In addition, install the Nvidia Riva client and audio dependencies to support audio processing features that may complement your LLM demonstration:

sudo pip install nvidia-riva-client
sudo apt-get install portaudio19-dev python3-pyaudio
sudo apt install pulsemixer
sudo pip install pyaudio --upgrade

Conclusion

This setup recipe demonstrates how to deploy an LLM on the Cardshark platform, integrating several key open-source projects to create a robust and manageable containerized environment. As more details become available, additional enhancements and troubleshooting tips will be provided. For further information, please refer to the linked project pages.

Happy experimenting with Cardshark and your LLM deployments!