Lepton AI

Prebuilt photons

Lepton supports running commonly used state-of-the-art models out of the box. To ease the creation of such model deployments, we have created a set of prebuilt Photon templates that can be used to run such models in one single command.

HuggingFace

When you create a Photon, you can refer to a prebuilt Huggingface model as hf:{model_id}, where model_id is the HuggingFace model id of a pretrained model, e.g.: meta-llama/Llama-2-7b-hf, stabilityai/stable-diffusion-2-1 etc.:

lep photon create -n my-llm -m hf:meta-llama/Llama-2-7b-hf

Currently Lepton have prebuilt HuggingFace models of the following pipeline tags:

There are two ways to find out the an HuggingFace model's pipline tag:

  1. Go to the corresponding model page on HuggingFace, the model's pipeline tag is shown under the model id. e.g.:

    On the HuggingFace model page of llama2-7b-hf model, it shows the pipeline tag is "Text Generation"

  2. Use the huggingface_hub Python to query from Hugging Face Hub:

    from huggingface_hub import model_info

mi = model_info("meta-llama/Llama-2-7b-hf")
print(mi.pipeline_tag)  # 'text-generation'

    which shows this model's pipeline tag is "text-generation".

LLMs

vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs. It seamlessly supports many LLM models on HuggingFace. Lepton supports running vLLM out of box:

When running locally:

lep ph run \
    -n mistral \
    -m vllm:mistralai/Mistral-7B-Instruct-v0.1 \
    --local

When running on Lepton Cloud Platform:

lep ph run \
    -n mistral \
    -m vllm:mistralai/Mistral-7B-Instruct-v0.1 \
    --resource-shape gpu.a10

Once the deployment is ready, you can find the deployment url on Dashboard:

You can use either the Lepton Client (leptonai.client.Client in Python) or OpenAI SDK to send requests:

Using leptonai.client.Client:

from leptonai.client import Client, local, current

client = Client(current(), "mistral")
# Or use `local()` if service was run locally
# client = Client(local())

completion = client.api.v1.chat.completions(
    model="mistralai/Mistral-7B-Instruct-v0.1",
    messages=[
        {"role": "user", "content": "Give me a 3 days travel plan for Hawaii"},
    ],
    max_tokens=512,
)

print(completion)

Using OpenAI Python SDK

import openai
from leptonai.api.workspace import WorkspaceInfoLocalRecord

openai.api_base = DEPLOYMENT_URL + "/api/v1"
# Or use http://localhost:8080 if service was run locally
# openai.api_base = "http://localhost:8080"
openai.api_key = WorkspaceInfoLocalRecord.get_current_workspace_token()

completion = openai.ChatCompletion.create(
    model="mistralai/Mistral-7B-Instruct-v0.1",
    messages=[
        {"role": "user", "content": "Give me a 3 days travel plan for Hawaii"},
    ],
    max_tokens=512,
)
print(completion)

Output:

 Hawaii is a fantastic destination with plenty of activities to enjoy. Here's a 3-day travel plan for Honolulu, the capital city of Hawaii.
#### Day 1:
1. Morning: Start your day at Waikiki Beach, where you can soak up the sun, swim in the ocean, and explore the famous Waikiki Surfing Lessons.
2. Afternoon: Visit the Honolulu Museum of Art, which features a vast collection of Asian, European, and American art. Afterward, head to the nearby Honolulu Zoo, where you can see a variety of animals, including exotic birds, monkeys, and endangered species.
3. Evening: Enjoy dinner at Alan Wong's Honolulu, a renowned restaurant known for its innovative Hawaiian cuisine. Then, take a leisurely stroll through the historic Chinatown, exploring its vibrant culture, shops, and nightlife.
#### Day 2:
1. Morning: Visit the Diamond Head State Monument, a iconic landmark offering panoramic views of Honolulu and the ocean. Then, head to the nearby Waikiki Aquarium, where you can see a variety of marine life, including sharks, sea turtles, and colorful fish.
2. Afternoon: Take a scenic drive to the North Shore, stopping at famous surf spots like Pipeline and Sunset Beach. Along the way, enjoy the stunning ocean views, lush landscapes, and picturesque towns.
3. Evening: Relax at one of the luxurious resorts on the North Shore, such as the Four Seasons Resort Maui at Wailea or the Montage Kapalua Bay. Enjoy a spa treatment, a gourmet dinner, or simply soak up the tranquility.
#### Day 3:
1. Morning: Go on a catamaran cruise, where you can explore the beautiful waters of Hawaii, snorkel with dolphins and sea turtles, and enjoy a delicious onboard lunch.
2. Afternoon: Visit the Polynesian Cultural Center, a unique museum that showcases the rich cultures of the Pacific Islands. Learn about traditional music, dance, art, and cuisine, and enjoy a hands-on experience at various cultural stations.
3. Evening: End your trip with a breathtaking sun