Lepton AI

Pod

Often in time, as an AI application developer, it is of great importance to have a handy development environment to test your AI applications. With Pod, you'll be able to have a AI development environment with:

  • Fast startup time: get your dev environment ready in seconds
  • Easy to use: no need to install anything, just use your browser or terminal to access the dev environment
  • Storage integration: mount your storage to the dev environment and access it

Craete a Pod

To create a Pod, you can use the Lepton CLI to do so or visit the pods page in the Dashboard to create a Pod.

During the pod creation, you can specify the following parameters:

  • Pod Name: the name of the Pod
  • Resource Shape: the resource shape of the Pod. For now, A10, A100, and H100 are supported.
  • Environment Variables: (Optional)the environment variables that will be injected into the Pod. You can use this to specify the environment variables that your AI application needs.
  • File System Mount: (Optiona)the file system that will be mounted to the Pod. You can access the file system in the Pod if specified.

Once the pod creation request is submitted, the Pod will be created in seconds. You will be able to see the access information of the Pod in the Pod detail page.

Access the pod via terminal SSH

You can use the SSH command to access the Pod. The SSH command is shown in both the Pod detail page and the Pods list. And the password is the workspace token.

Access the pod via browser

You can also access the Pod via Web terminal in the Pod detail page as shown below. Do notice that the Web terminal is more suitable for temporary access. If you want to have a long-term session to the Pod, please use the SSH command.

Access services hosted in a Pod

If you have a service running inside the Pod, you can access the service by using the internal port 18888 and access this service via the public ip and a randomly assigned port. The randomly assigned port is shown in the Pod detail page. Here is an example code of a Gradio application running in a Pod.

import gradio as gr

def greet(name, intensity):
    return "Hello " * intensity + name + "!"

demo = gr.Interface(
    fn=greet,
    inputs=["text", "slider"],
    outputs=["text"],
)

demo.launch(server_name="0.0.0.0", server_port=18888) # speciy the server name and port

You can save the code to a file named main.py and start this application with command python main.py. Then you can access the application via the public ip and the randomly assigned port as indicated below.

Preserve long running sessions

The network connection could get lost if the Pod is idle for a long time or the network is unstable.

To preserve the long running sessions, you can use the tmux command to create a new session and run your application in the session. The tmux command is already installed in the Pod. Here is an example of how to use the tmux command.

# Install the tmux command
apt install tmux
# Create a new session
tmux 
# list all the sessions
tmux ls
# attach to the session
tmux attach -t 0

To detach from the session, you can use the Ctrl+b d command. To learn more about the tmux command, you can visit the tmux documentation

Setup SSH for AI Pod

If you are using default image provided by Lepton AI, the SSH service is already configured. If you'd like to use ssh public key authentication, you can add your public key to the authorized_keys file in the Pod with following command.

export PUBLIC_KEY="YOUR_PUBLIC_KEY"
# Create the .ssh directory and authorized_keys file if they don't exist
if [ ! -d "$HOME/.ssh" ]; then
    mkdir -p "$HOME/.ssh"
    chmod 0700 "$HOME/.ssh"
    echo "Directory $HOME/.ssh created."
fi
if [ ! -f "$HOME/.ssh/authorized_keys" ]; then
    touch "$HOME/.ssh/authorized_keys"
    chmod 0600 "$HOME/.ssh/authorized_keys"
    echo "File $HOME/.ssh/authorized_keys created."
fi
# Check if the public key is not already present in authorized_keys
if ! grep -q "${PUBLIC_KEY}" "$HOME/.ssh/authorized_keys"; then
    # Append the public key to authorized_keys
    echo "$PUBLIC_KEY" >> "$HOME/.ssh/authorized_keys"
    echo "Public key from env variable added."
fi

Once the public key is added to the authorized_keys file, you can use the SSH command to access the Pod with the public key.

If you are using your own image, there are few environment variables that you can set to configure the SSH service.

  • PASSWORD: the password of the root user. Leave it empty to disable root password login.
  • PUBLIC_KEY: the public key that will be added to the authorized_keys file.

Replace the environment variables with your own values and update the following script accordingly.

#!/bin/bash

export PASSWORD="YOUR_PASSWORD_HERE" # Leave it empty to disable root password
export PUBLIC_KEY="YOUR_PUBLIC_KEY_HERE" # Leave it empty to disable public key authentication

# Check if OpenSSH server is already installed
if ! command -v sshd &> /dev/null; then
    echo "OpenSSH server is not installed. Installing..."
    
    apt update
    apt install -y openssh-server
    
    echo "OpenSSH server installation complete."
else
    echo "OpenSSH server is already installed."
fi

# Set root password if PASSWORD is provided
if [[ -n "$PASSWORD" ]]; then
    # Enable password authentication in SSH configuration
    sed -i '/^#PasswordAuthentication/c\PasswordAuthentication yes' /etc/ssh/sshd_config
    sed -i '/^PasswordAuthentication/c\PasswordAuthentication yes' /etc/ssh/sshd_config
    # Display a message indicating that user/password SSH access is enabled
    echo "User/password SSH access is enabled."
    # Set the root password
    echo "root:${PASSWORD}" | chpasswd
    echo "Root password has been set."
fi

# Check if PUBLIC_KEY variable is set and not empty
if [[ -n "$PUBLIC_KEY" ]]; then
    # Create the .ssh directory and authorized_keys file if they don't exist
    if [ ! -d "$HOME/.ssh" ]; then
        mkdir -p "$HOME/.ssh"
        chmod 0700 "$HOME/.ssh"
        echo "Directory $HOME/.ssh created."
    fi
    if [ ! -f "$HOME/.ssh/authorized_keys" ]; then
        touch "$HOME/.ssh/authorized_keys"
        chmod 0600 "$HOME/.ssh/authorized_keys"
        echo "File $HOME/.ssh/authorized_keys created."
    fi
    # Check if the public key is not already present in authorized_keys
    if ! grep -q "${PUBLIC_KEY}" "$HOME/.ssh/authorized_keys"; then
        # Append the public key to authorized_keys
        echo "$PUBLIC_KEY" >> "$HOME/.ssh/authorized_keys"
        echo "Public key from env variable added."
    fi
fi

# Enable root login in SSH configuration
sed -i '/^#PermitRootLogin/c\PermitRootLogin yes' /etc/ssh/sshd_config
sed -i '/^PermitRootLogin/c\PermitRootLogin yes' /etc/ssh/sshd_config
echo "Root login is enabled."

echo "Exposing ENV variables"
env > /etc/environment

mkdir /run/sshd
chmod 0755 /run/sshd

echo "Starting sshD"
exec /usr/sbin/sshd -D -p 2222

Use the web terminal to save the script to a file named sshd_init.sh, and run the following commands to start the SSH service in the Pod.

apt update && apt install curl vim -y # install the necessary tools
touch sshd_init.sh # create a new file named sshd_init.sh
vi sshd_init.sh # copy the content from the sshd_init.sh script
chmod +x sshd_init.sh 
nohup ./sshd_init.sh & # run the script in the background

Once the SSH service is started, you can use the SSH command to access the Pod. The password is the workspace token.