Lepton AI

File System

File System is a persistent remote storage service that offers POSIX filesystem semantics. It can be mounted as a POSIX filesystem to Deployments, Pods, and Jobs. This allows you to manage files and folders within the file storage using either the dashboard or CLI tools.

Types of File System

Lepton provides different types of file systems to cater to different use cases. The following are the types of file systems available in Lepton:

Default File System

The default file system is a shared file system that is accessible across all types of workloads in a workspace.

Node group specific file system

For enterprise users who have reserved resources on Lepton, we provide the option to create a file system that is more performant to a specific node group.

Local Storage

Local Storage refers to the storage that utilizes the local SSDs attached to each node within a cluster. This is only available to workload with node groups specified. This type of storage provides high-performance, low-latency access to data, making it suitable for workloads that require fast I/O operations. However, local storage cannot be shared across nodes. To share data across workloads, you need to make sure the workload is scheduled on the same node.

Usage and Management

Mounting the File System

When creating workload, you can specify the file system to be mounted to the workload. A workload can have multiple file system mounts. Here are few parameters you need to specify when mounting a file system:

  • File System: The file system to be mounted. You can choose from the list of available file systems. To mount local storage, you need to make sure the node group option is selected when creating the workload. Otherwise, the list of local storage options will not be visible in the file system dropdown.

  • Mount From: The path within the file system to be mounted. For example, if you want to mount the root of the file system, you can specify /. Or if you want to mount a specific folder, you can specify the path to the folder within the file system. For local storage, you will not be able to specify the mount from path as it is always mounted the whole local storage block.

  • Mount As: The path within the workload where the file system will be mounted. For example, if you want to mount the file system to /mnt/data, you can specify /mnt/data as the mount as path.

File and Folder Management

Using the Dashboard

To manage files and folders within the defualt or node group specific file storage, you can use the dashboard. You can upload, download, delete, and create files and folders within the file storage using the dashboard directly from your local machine.

You can also upload files from cloud storage services such as AWS S3 and Cloudflare R2. On the File System page, click Upload File then choose From Cloud will open a dialog for you to select the cloud storage service and fill in the required information.

For AWS S3, you need to provide the following information:

  • Bucket Name : The name of the bucket you want to upload from.
  • Access Key ID : The access key of the AWS account.
  • Secret Access Key : The secret access key of the AWS account.
  • Destination Path : The path in the file system you want to upload to.

For Cloudflare R2, you need to provide the following information:

  • Endpoint URL : The S3 API URL of the Cloudflare R2 bucket. It can be found in the bucket's settings page uder Bucket Details. Do not include the bucket name in the URL. It should look like https://xxxxxxxxx.r2.cloudflarestorage.com.
  • Bucket Name : The name of the bucket you want to upload from.
  • Access Key ID : The access key of the Cloudflare R2 API Token. You can manage and create R2 tokens by clickin the Manage R2 API Tokens button in the bucket's settings page.
  • Secret Access Key : The secret access key of the Cloudflare R2 API Token.
  • Destination Path : The path in the file system you want to upload to.

The local storage is not accessible via the dashboard. You can only manage files within the local storage with a workload. For example, you can create a pod with local storage mounted and then use the pod to manage files within the local storage.

Using CLI tools

You can also manage files and folders within the file storage using the CLI tools. You can upload, download, delete, and create files and folders within the file storage using the CLI tools. Here are a few examples of how you can manage files and folders using the CLI tools:

# Upload a file to the file storage
$ lep storage upload /local/path/to/a.txt /remote/path/to/a.txt
# Upload a folder to the file storage, rsync is only available for the standard and enterprise plan
$ lep storage upload -r -p --rsync /local/path/to/folder /remote/path/to/folder
# Download a file from the file storage
$ lep storage download /remote/path/to/a.txt /local/path/to/a.txt
# Remove a file from the file storage
$ lep storage rm /remote/path/to/a.txt

For more information on how to manage files and folders using the CLI tools, refer to the CLI documentation.

Lepton AI

© 2024