Photons

The anatomy of a photon

First of all, we will walk through the anatomy of a photon, including the python class of the same name, and the packaging and deployment process.

Lepton defines a basic python class called Photon. A photon class defines a set of handlers, which can be used as python methods in the python clients, and are usually also referred to as "endpoints" in web frameworks like FastAPI. The Photon class also defines a set of python dependencies, which are automatically installed when the photon is deployed. Those dependencies are defined as a list of strings, each string is a valid pip package name.

To write one's own photon, we can derive from the Photon class, and implement the following functions:

an init(self) function. This function is called by the photon runtime when the photon is first loaded. It is used to initialize the photon, and can be used to load models, initialize variables, etc.
several methods decorated by @photon.handler. This will be the handlers of the photon, and can be called by the python client. The @photon.handler decorator takes an optional string argument, which is the name of the handler. The handler takes arguments and returns values that are FastAPI compatible.

Here is a simple example implementing a counter photon, that accepts two handlers add and sub. They add or subtract the input number from the counter, and return the new counter value.

from leptonai.photon import Photon

class Counter(Photon):
    # The init method implements any custom initialization logic we need.
    def init(self):
        self.counter = 0

    # When no name is specified, the handler name is the method name.
    @Photon.handler
    def add(self, x: int) -> int:
        self.counter += x
        return self.counter

    # Or, we can specify a name for the handler.
    @Photon.handler("sub")
    def sub(self, x: int) -> int:
        return self.add(-x)

Locally debug photons

When we debug the photon class, we can directly run it locally in the same python process, and the handlers will be available as class methods. For example, we can run the following code to test the Counter class:

c = Counter()
# we can call c.init() manually, but it is not necessary.
# The init() method will be called automatically when a handler is called for the first time.
c.add(x=3) # This will return 3
c.add(x=5) # This will return 8
c.sub(x=2) # This will return 6

All the calls are simple wrapped Python methods, and local pdb works as well for us to debug the code with ease.

You might have noticed that we use kwargs formats to call the function. This is because remote services over http prefers such a format for the request body, and we want to keep the interface consistent.

Create a photon

After learning the anatomy of Photon, we can start creating our own photons.

Create from scratch

We can create a photon from scratch with the Lepton CLI.

Packaging and deploying

lep photon create -n counter -m counter.py

Think of a photon as a container image, but much lightweighted: it records the absolute necessary information to run an algorithm, a python program, etcs. Unlike a container, it doesn't have to be a full-fledged operating system image, which usually takes hundreds of megabytes if not gigabytes. Instead, it only records the python class and the dependencies, which is usually only a few kilobytes.

This creates a photon with the name counter, and the model specs being the counter.py file. Lepton will automatically find the python class Counter in the file. If we have multiple classes defined in the same file, we can specify the class we want to run after the filename, like -m counter.py:Counter, but it's not necessary if we have only one class.

Once we have the photon packaged, we can run it locally with:

lep photon runlocal -n counter

The difference between a local run and directly creating a Counter class in a python program is that, a local run is in a separate process, and we can test it with the python client:

from leptonai.client import Client, local

c = Client(local())
print(f"Add 3, result: {c.add(x=3)}") # This will return 3
print(f"Add 5, result: {c.add(x=5)}") # This will return 8
print(f"Sub 2, result: {c.sub(x=2)}") # This will return 6

Now that it is running as a standalone service, you can also use the curl command to access it:

curl -X POST -H "Content-Type: application/json" -d '{"x": 3}' http://localhost:8080/add

Remote Deployment

When we are satisfied with the photon, we can deploy it to the remote platform with the following command, given that we have already logged in with lep login:

lep photon push -n counter

This will push the photon to the remote platform. Think of this as the equivalent of pushing an image to a container image, but much faster. We can then list the photons in the workspace with lep photon list, and we should see the photon we just pushed:

lep photon list

Let's launch the photon with lep photon run:

lep photon run -n counter

This will create a deployment for the photon. In default, if we don't specify a name, a default name will be chosen for the deployment, with the first choice being the same as the photon name. We can then inspect the status of the deployment with:

lep deployment status -n counter

Calling the deployment is the same as calling the local photon run with the client, except that we need to specify how we access the deployment: the workspace, the deployment name, and the token to access it:

from leptonai.client import Client

# You can obtain your current workspace id via CLI as `lep workspace id`
workspace_id = "xxxxxxxx"
deployment_name = "counter"
# You can obtain your current workspace's token via CLI as `lep workspace token`
token = "xxxxxxxx"

c = Client(workspace_id, deployment_name, token)
print(f"Add 3, result: {c.add(x=3)}") # This will return 3
print(f"Add 5, result: {c.add(x=5)}") # This will return 8
print(f"Sub 2, result: {c.sub(x=2)}") # This will return 6

Adding Dependencies

The counter class does not have any dependencies, but most of the time we will need to import other packages. Instead of going as far as creating a full Docker image, we provide a lightweighted way for everyday python and Linux packages to be installed. This is done via the requirement_dependency (for python pip) and the system_dependency (for system apt) fields in the photon class. Here's a minimal example to demonstrate how to use them:

import subprocess
from leptonai.photon import Photon

class My(Photon):
    # The dependencies you usually write in requirements.txt
    requirement_dependency = ["numpy", "torch"]

    # The dependencies you usually do `apt install` with
    system_dependency = ["wget"]

    @Photon.handler
    def run(self):
        """
        A example handler to verify that the dependencies are installed.
        """
        import numpy  # This should succeed
        import torch  # This should succeed
        p = subprocess.Popen(["wget", "--version"],
                             stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE)
        out, _ = p.communicate()
        return out

When we push and run the photon remote, the required dependencies are installed automatically.

Note that you can use most formats supported by pip to install packages. For example, to install the segment anything library, you can use the format suggested by the library's installation instruction:

requirement_dependency = ["git+https://github.com/facebookresearch/segment-anything.git"]

Adding extra files

Sometimes we need to add extra files to the photon, such as a model checkpoint, a configuration file, etc. We can do this by adding the extra_files field to the photon class. It takes two forms:

A dictionary of {"remote_path": "local_path"} pairs, where the remote_path is relative to the cwd of the photon at runtime, and local_path is the path pointing to the file in the local file system. If local_path is relative, it is relative to the current working directory of the local environment.

A list of paths (relative to the cwd of the local directory) to be included. The remote path will be the same as the local path.

Here's an example to demonstrate how to use them:

.
├── counter.py
└── sample.txt

Using self-defined docker images

Sometimes we need to use a self-defined docker image for flexibility. We can do this by adding the docker_image field to the photon class. It takes a string, which is the name of the docker image. The image should be available in the docker registry of the platform. For example, if we have a docker image with name and tag like ssusb/anaconda3:latest, we can use it with the following code:

from leptonai.photon import Photon

class Counter(Photon):
    # The init method implements any custom initialization logic we need.
    image = "ssusb/anaconda3:latest"

    def init(self):
        pass

To use image from private registry, add your registry via Settings-Registries in the web console to pass the authentication. When creating deployment with private image, choose the registry you want to use in Advanced Settings.

Once the docker image is specified, the dependencies and extra files will be installed in the docker image. The photon will be run as the user lepton in the docker image, and the working directory will be /home/lepton.

Create from GitHub repository

We can also create a photon from a Github repository. This is useful when we want to create a photon from a repository without cloning it to local machine.

Generate a GitHub token

Go to Github Personal Access Tokens Page, click Generate new token. Make the following changes based on your requirements:

Resource Owner: Change it to the owner of the repo from which you will be creating the photon.
Repository access: For security purposes, only select the repo you'd like to be used.
Permissions:
- Contents: Give read-only permission.

Then click Generate token.

Set up the environment variable for CLI to pull the repo

In the terminal, type in the following commands:

export GITHUB_USER={YOUR_GITHUB_USERNAME}
export GITHUB_TOKEN={THE_TOKEN_GENERATED_FROM_STEP_1}

Create photon via lepton cli

lep photon create -n {PHOTON_NAME} -m py:{GIT_REPO_URL}:{PATH_TO_SCRIPT}:{CLASS_NAME}

Key	Description	Example
PHOTON_NAME	The name of the photon	my-fs-counter
GIT_REPO_URL	The URL for the repo	github.com/leptonai/examples.git
PATH_TO_SCRIPT	The file extends the runner class	getting-started/counter/counter.py
CLASS_NAME	The class extends the runner class inside the script	Counter

Manage remote photons

Photons are held in your Lepton AI workspace. You can think of a workspace as a project, and photons as the building blocks of the project.

Create and push a photon to a workspace

This will upload the photon to the cloud and make it available for endpoint creation.

lep photon create -n myphoton -m hf:gpt2
lep photon push -n myphoton

List all photons

You can list all photons in your workspace with lep photon list:

lep photon list

Optionally, use --pattern to filter out the photons you want to see.

Photons created via lep photon create are always stored locally. To view the local photons, use lep photon list --local.

Remove a photon

Use lep photon remove to remove a photon from your workspace. Note that photons pushed to the workspace are versioned, and you can remove a specific version of a photon by specifying its version id. If only name is specified, the most recently pushed version is removed. If you want to remove all versions of a photon, use --all flag.

lep photon remove -n myphoton --all

Also, if you are removing a local photon, use lep photon remove --local.

If you are following along the commands in this section, before moving to the next, run lep photon push -n myphoton again to push the photon back to the workspace.

Run a photon

Use lep photon run to run a photon and create a deployment. In the easiest case, you can simply specify the name of the photon to run. This will create a deployment with the same name (if not existing) as the photon:

lep photon run -n myphoton

In default, created deployments are protected by the access token of the workspace. You can specify a different token with --token flag. If you want to create a public deployment, use --public flag. Public deployments are useful when you want to create a public API or a web app.

You can also specify a lot of other configurations for the deployment, such as the number of replicas, the resource shape, etc. For a full list of options, use lep photon run --help, or refer to the API reference.

Prebuilt photons

Lepton also supports running commonly used state-of-the-art models out of the box. To ease the creation of such model deployments, we have created a set of prebuilt Photon templates that can be used to run such models in one single command.

HuggingFace

When you create a Photon, you can refer to a prebuilt Huggingface model as hf:{model_id}, where model_id is the HuggingFace model id of a pretrained model, e.g.: meta-llama/Llama-2-7b-hf, stabilityai/stable-diffusion-2-1 etc.:

lep photon create -n my-llm -m hf:meta-llama/Llama-2-7b-hf

Currently Lepton have prebuilt HuggingFace models of the following pipeline tags:

audio-classification (browse models)
automatic-speech-recognition (browse models)
depth-estimation (browse models)
feature-extraction (browse models)
image-to-text (browse models)
sentence-similarity (browse models)
summarization (browse models)
text-classification (browse models)
text2text-generation (browse models)
text-generation (browse models)
text-to-image (browse models)

There are two ways to find out the an HuggingFace model's pipline tag:

Go to the corresponding model page on HuggingFace, the model's pipeline tag is shown under the model id. e.g.:

On the HuggingFace model page of llama2-7b-hf model, it shows the pipeline tag is "Text Generation"

Use the huggingface_hub Python to query from Hugging Face Hub:

from huggingface_hub import model_info

mi = model_info("meta-llama/Llama-2-7b-hf")
print(mi.pipeline_tag)  # 'text-generation'

which shows this model's pipeline tag is "text-generation".

LLMs

vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs. It seamlessly supports many LLM models on HuggingFace. Lepton supports running vLLM out of box:

When running locally:

lep ph run \
    -n mistral \
    -m vllm:mistralai/Mistral-7B-Instruct-v0.1 \
    --local

When running on Lepton Cloud Platform:

lep ph run \
    -n mistral \
    -m vllm:mistralai/Mistral-7B-Instruct-v0.1 \
    --resource-shape gpu.a10

Once the deployment is ready, you can find the deployment url on Dashboard. You can use either the Lepton Client (leptonai.client.Client in Python) or OpenAI SDK to send requests:

Using leptonai.client.Client:

from leptonai.client import Client, local, current

client = Client(current(), "mistral")
# Or use `local()` if service was run locally
# client = Client(local())

completion = client.api.v1.chat.completions(
    model="mistralai/Mistral-7B-Instruct-v0.1",
    messages=[
        {"role": "user", "content": "Give me a 3 days travel plan for Hawaii"},
    ],
    max_tokens=512,
)

print(completion)

Using OpenAI Python SDK

import openai
from leptonai.api.workspace import WorkspaceInfoLocalRecord

openai.api_base = DEPLOYMENT_URL + "/api/v1"
# Or use http://localhost:8080 if service was run locally
# openai.api_base = "http://localhost:8080"
openai.api_key = WorkspaceInfoLocalRecord.get_current_workspace_token()

completion = openai.ChatCompletion.create(
    model="mistralai/Mistral-7B-Instruct-v0.1",
    messages=[
        {"role": "user", "content": "Give me a 3 days travel plan for Hawaii"},
    ],
    max_tokens=512,
)
print(completion)

Output:

 Hawaii is a fantastic destination with plenty of activities to enjoy. Here's a 3-day travel plan for Honolulu, the capital city of Hawaii.
#### Day 1:
1. Morning: Start your day at Waikiki Beach, where you can soak up the sun, swim in the ocean, and explore the famous Waikiki Surfing Lessons.
2. Afternoon: Visit the Honolulu Museum of Art, which features a vast collection of Asian, European, and American art. Afterward, head to the nearby Honolulu Zoo, where you can see a variety of animals, including exotic birds, monkeys, and endangered species.
3. Evening: Enjoy dinner at Alan Wong's Honolulu, a renowned restaurant known for its innovative Hawaiian cuisine. Then, take a leisurely stroll through the historic Chinatown, exploring its vibrant culture, shops, and nightlife.
#### Day 2:
1. Morning: Visit the Diamond Head State Monument, a iconic landmark offering panoramic views of Honolulu and the ocean. Then, head to the nearby Waikiki Aquarium, where you can see a variety of marine life, including sharks, sea turtles, and colorful fish.
2. Afternoon: Take a scenic drive to the North Shore, stopping at famous surf spots like Pipeline and Sunset Beach. Along the way, enjoy the stunning ocean views, lush landscapes, and picturesque towns.
3. Evening: Relax at one of the luxurious resorts on the North Shore, such as the Four Seasons Resort Maui at Wailea or the Montage Kapalua Bay. Enjoy a spa treatment, a gourmet dinner, or simply soak up the tranquility.
#### Day 3:
1. Morning: Go on a catamaran cruise, where you can explore the beautiful waters of Hawaii, snorkel with dolphins and sea turtles, and enjoy a delicious onboard lunch.
2. Afternoon: Visit the Polynesian Cultural Center, a unique museum that showcases the rich cultures of the Pacific Islands. Learn about traditional music, dance, art, and cuisine, and enjoy a hands-on experience at various cultural stations.
3. Evening: End your trip with a breathtaking sun