Pod
Often in time, as an AI application developer, it is of great importance to have a handy development environment to test your AI applications. With Pod, you'll be able to have a AI development environment with:
- Fast startup time: get your dev environment ready in seconds
- Easy to use: no need to install anything, just use your browser or terminal to access the dev environment
- Storage integration: mount your storage to the dev environment and access it
Craete a Pod
To create a Pod, you can use the Lepton CLI to do so or visit the pods page in the Dashboard to create a Pod.
During the pod creation, you can specify the following parameters:
- Resource Shape: the resource type of the Pod. A10, A100, and H100 are supported. The GPU Count will be 1 by default. If you need more GPUs, you can specify the GPU Count.
- Node Groups: (Optional) the node group that the Pod will be created in. If not specified, the Pod will be created using the on-demand resource.
- Nodes: (Optional) Only available once the Node Groups is specified. Specify the list of nodes that the Pod will be created in. If not specified, the Pod will be created in the first available node within the Node Group specified.
- Pod Name: the name of the Pod, only alphanumeric characters (a-z, 0-9) and '-' allowed.
- Container Image: (Optional) the container image that will be used to create the Pod. Choose from default image lists or pass in docker image urls like
ollama/ollama:latest.
If not specified, the default image will be used. - File System Mount: (Optional) the file system that will be mounted to the Pod. You can access the file system in the Pod if specified. If not specified, the storage will be ephemeral and will be deleted once the Pod is deleted or restarted.
- SSH Public Key: (Optional) the public key that will be added to the authorized_keys file in the Pod. You can use this to specify the public key that you want to use to access the Pod.
Advanced Settings
- Private Image Registry Auth: (Optional)the private image registry authentication that will be used to pull the container image. You can use this to specify the private image registry authentication if the container image is hosted in a private registry.
- Environment Variables: (Optional)the environment variables that will be injected into the Pod. You can use this to specify the environment variables that your AI application needs.
- Visibility: (Optional)the visibility of the Pod. You can use this to specify the visibility of the Pod. If the visibility is set to private, only the creator of the Pod can access the Pod. If the visibility is set to public, all the users in the workspace can access the Pod.
Once the pod creation request is submitted, the Pod will be created in seconds. You will be able to see the access information of the Pod in the Pod detail page.
Access the pod via terminal SSH
You can use the SSH command to access the Pod. The SSH command is shown in both the Pod detail page and the Pods list.
Access the pod via browser
You can also access the Pod via Web terminal in the Pod detail page as shown below. Do notice that the Web terminal is more suitable for temporary access. If you want to have a long-term session to the Pod, please use the SSH command.
Access services hosted in a Pod
If you have a service running inside the Pod, you can access the service by using the internal port 18888
and access this service via the public ip and a randomly assigned port. The randomly assigned port is shown in the Pod detail page. Here is an example code of a Gradio application running in a Pod.
import gradio as gr
def greet(name, intensity):
return "Hello " * intensity + name + "!"
demo = gr.Interface(
fn=greet,
inputs=["text", "slider"],
outputs=["text"],
)
demo.launch(server_name="0.0.0.0", server_port=18888) # speciy the server name and port
You can save the code to a file named main.py
and start this application with command python main.py
. Then you can access the application via the public ip and the randomly assigned port as indicated below.
Preserve long running sessions
To preserve the long running sessions, you can use the nohup
command to run the application in the background. For more information on how to use nohup, you can refer to the nohup documentation. Here is an example code of running the Gradio application in the background.
nohup python main.py --port 18888 --listen 0.0.0.0 > main.log 2>&1 &
After executing this command, the terminal will display the process ID of the Gradio application. You can now disconnect from the Pod, and the application will continue to run in the background. For more information on how to use nohup, you can refer to the nohup documentation
Connect to Pod via SSH
If you are using default image provided by Lepton AI, the SSH service is already configured. If you are using your own image, you can configure the SSH service by change the code below and run it in the Web terminal.
Edit line 4 to 8 below to be your SSH password, SSH public key and Jupyter Lab token. Leave it empty to skip installation.
#!/bin/bash
############ User Space ##############
# Please fill in your Jupyter Lab token here, leave empty to use env value
USER_JUPKEY=""
# Please fill in your SSH public key here (something like ssh-rsa AAAxxxxx lepton@sampleDomain.com), leave empty to use env value
USER_SSHPUB=""
# Please fill in your SSH root password here, leave empty to use env value
USER_SSHKEY=""
######################################
JUPKEY=$LEPTON_POD_JUPKEY
SSHPUB=$LEPTON_POD_SSHPUB
SSHKEY=$LEPTON_POD_SSHKEY
if [[ -n "$USER_JUPKEY" ]]; then
JUPKEY=$USER_JUPKEY
fi
if [[ -n "$USER_SSHPUB" ]]; then
SSHPUB=$USER_SSHPUB
fi
if [[ -n "$USER_SSHKEY" ]]; then
SSHKEY=$USER_SSHKEY
fi
export DEBIAN_FRONTEND=noninteractive
export TZ=Etc/UTC
function InstallSSH {
if service sshd status > /dev/null 2>&1; then
echo "OpenSSH server is already started."
return
fi
# Check if OpenSSH server is already installed
if ! command -v sshd &> /dev/null; then
echo "OpenSSH server is not installed. Installing..."
apt update
apt install -y openssh-server
echo "OpenSSH server installation complete."
else
echo "OpenSSH server is already installed."
fi
# Set root password if SSHKEY is provided
if [[ -n "$SSHKEY" ]]; then
# Enable password authentication in SSH configuration
sed -i '/^#PasswordAuthentication/c\PasswordAuthentication yes' /etc/ssh/sshd_config
sed -i '/^PasswordAuthentication/c\PasswordAuthentication yes' /etc/ssh/sshd_config
# Enable root login in SSH configuration
sed -i '/^#PermitRootLogin/c\PermitRootLogin yes' /etc/ssh/sshd_config
sed -i '/^PermitRootLogin/c\PermitRootLogin yes' /etc/ssh/sshd_config
echo "Root login is enabled."
# Display a message indicating that user/password SSH access is enabled
echo "User/password SSH access is enabled."
echo "root:${SSHKEY}" | chpasswd
echo "Root password has been set."
fi
# Check if LEPTON_PUBLIC_KEY variable is set and not empty
if [[ -n "$SSHPUB" ]]; then
# Create the .ssh directory and authorized_keys file if they don't exist
if [ ! -d "$HOME/.ssh" ]; then
mkdir -p "$HOME/.ssh"
chmod 0700 "$HOME/.ssh"
echo "Directory $HOME/.ssh created."
fi
if [ ! -f "$HOME/.ssh/authorized_keys" ]; then
touch "$HOME/.ssh/authorized_keys"
chmod 0600 "$HOME/.ssh/authorized_keys"
echo "File $HOME/.ssh/authorized_keys created."
fi
# Check if the public key is not already present in authorized_keys
if ! grep -q "${SSHPUB}" "$HOME/.ssh/authorized_keys"; then
# Append the public key to authorized_keys
echo "$SSHPUB" >> "$HOME/.ssh/authorized_keys"
echo "Public key from env variable added."
fi
fi
# turn off PAM to fix sshd login issue
sed -i 's/UsePAM yes/UsePAM no/' /etc/ssh/sshd_config
# set default port to 2222
sed -i 's/#Port 22/Port 2222/' /etc/ssh/sshd_config
echo "Exposing ENV variables"
env | sed 's/=/="/' | sed 's/$/"/' > /etc/environment
echo "set -a; source /etc/environment; set +a;" >> /root/.bashrc
mkdir /run/sshd
chmod 0755 /run/sshd
service ssh start
echo "sshd service started"
}
function InstallJupyter {
if pgrep jupyter-lab > /dev/null 2>&1; then
echo "jupyter already started"
return
fi
# Check if jupyter lab is already installed
if ! command -v jupyter-lab &> /dev/null; then
echo "jupyter lab is not installed. Installing..."
apt update
apt install python3 python3-pip -y
pip install -U virtualenv --break-system-packages
pip3 install jupyterlab --break-system-packages
echo "jupyter lab installation complete."
else
echo "jupyter lab is already installed."
fi
jupyter lab --generate-config
{
echo "c.ServerApp.ip = '0.0.0.0'"
echo "c.ServerApp.open_browser = False"
echo "c.ServerApp.port = 18889"
} >> ~/.jupyter/jupyter_lab_config.py
# Set root password if LEPTON_POD_ROOT_PASSWORD is provided
if [[ -n "$JUPKEY" ]]; then
echo "c.ServerApp.token = '${JUPKEY}'" >> ~/.jupyter/jupyter_lab_config.py
echo "Root password has been set."
fi
jupyter lab --allow-root > /var/log/jupyter.log 2>&1 &
}
if [[ -n $SSHKEY || -n $SSHPUB ]]; then
InstallSSH
fi
if [[ -n $JUPKEY ]]; then
InstallJupyter
fi
Once you have edited the code accordingly, you can copy the code and paste it into the Web terminal and press enter. The SSH service will be configured and you can access the Pod via SSH.