Observability

Lepton provides a comprehensive observability platform to help you monitor and troubleshoot your applications, including logs, audit logs, and health checks.

Logs

Learn how to use real-time logs and advanced logs.

What are logs?

Logs are generated by the replica in endpoint, job and pod, including all standard output and error output (print or echo).

With real-time logs:

  • Logs are real-time and can be viewed as soon as the replica in the deployment, job and pod are running. Real-time logs will no longer be available once they are deleted or terminated.
  • Real-time logs can only be read from the last 10,000 rows

With advanced logs:

  • Support search and filter
  • View log context
  • Query at the deployment, job, pod and Workspace levels
  • Advanced logs can keep logs generated within 7 days at most

Real-time Logs

To view real-time logs:

  1. From the dashboard, navigate to the deployment, job or pod detail page.
  2. Select the replicas tab.
  3. Click the Logs button to view the real-time logs.
Navigating to Real-time Logs

You can also click on the Live button on the right top corner to switch the live mode.

Advanced Logs

To view advanced logs:

  • From the dashboard, select the Observability tab and select Logs.
  • Or, select the Logs tab from the deployment, job or pod detail page.
Advanced Logs

Filters

You can use the following filters from the left sidebar to get a refined search experience.

FilterDescription
TimelineQuery logs by specific time, including the maximum (7 days) or a custom time range, default is most recent 24 hours.
ScopeSelect or enter a deployment, job, or pod name to query the logs of all replicas within that scope.
replicaSelect or enter a specific replica name to query the logs of that replica.

Log Context

It is helpful to see the log context when the filter matches multiple time periods or scopes.

Mouse over the line whose context you want to view, and click the show context button to view the log context. In the log context dialog, the current line will also be highlighted, along with its context

log-context

Audit Logs

Learn how to view audit logs in Lepton.

What are audit logs?

Audit log records actions and API calls performed by members of your workspace. It helps you to track the changes made by members in your workspace.

View audit logs

From the dashboard, select the Observability tab and select Audit Logs.

Audit Logs

Health Checks

Learn how to create health checks in Lepton AI.

What is health checks?

Health checks involve observing and checking the status of your resources and services. It helps you understand their performance and health.

You can also configure alerts to get notified when a specific condition is met.

Create health checks

From the Dashboard, select the Observability tab and select Health Checks. Then click the Create button to create health checks.

Create Health Checks
  • Name: The name of the monitor.
  • Alert Destination: The alert destination to send the alert to.
  • URL: The URL and method to monitor, uou can select the API URL from deployment.
  • Body: The body of the request.
  • Headers: The headers of the request (use JSON format), you may also to add Authorization header if needed.
  • Expected Status: The HTTP status codes to expect in the response. default is 200 and you can add multiple status codes.
  • Expected Regex Match: The regex pattern to match in the response body, e.g. .*success":true.*.
  • Check Interval: How often to check the health of the target, in seconds.
  • Maximum Latency: The maximum latency allowed for the target, in milliseconds.
  • Consecutive Success Count: The number of consecutive successful checks required to mark the target as healthy.
  • Consecutive Failure Count: The number of consecutive failed checks required to mark the target as unhealthy.

Configure alerts

From the Dashboard, select the Observability tab and select Health Checks. Then click the Alert Configuration button to create alerts.

  • Name: The name of the alert.
  • Webhook: The URL to send the alert to, we send webhooks using an Slack webhooks format.

Mute alerts

You can also mute the alerts by click the mute button on the right side of the list or on the details page.

Telemetry

Lepton also supports to export prometheus metrics that are used to collect and forward metrics to third-party metric providers. Navigate to the Telemetry tab under the Observability page, and you can configure for exporting Prometheus metric here.

Now we only support integration with Google Cloud, press the configure button and paste your Google Cloud Accees Control then the telemetry will be set up in a minute.

Lepton AI

© 2024