Upgrading SkyPilot API Server#

This page provides an overview of the steps you should follow to upgrade a remote SkyPilot API server:

Upgrade API server deployed with Helm#

With Helm deployement, it is possible to upgrade the SkyPilot API server gracefully without causing client-side error with the steps below.

Step 1: Prepare an upgrade#

  1. Find the version to use in SkyPilot nightly build.

  2. Update SkyPilot helm repository to the latest version:

helm repo update skypilot

  1. Prepare versioning environment variables. NAMESPACE and RELEASE_NAME should be set to the currently installed namespace and release:

NAMESPACE=skypilot # TODO: change to your installed namespace
RELEASE_NAME=skypilot # TODO: change to your installed release name
VERSION=1.0.0-dev20250410 # TODO: change to the version you want to upgrade to
IMAGE_REPO=berkeleyskypilot/skypilot-nightly

Step 2: Upgrade the API server and clients#

Upgrade the clients:

pip install -U skypilot-nightly==${VERSION}

Upgrade the API server:

# --reuse-values is critical to keep the values set in the previous installation steps.
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot-nightly --devel --reuse-values \
  --set apiService.image=${IMAGE_REPO}:${VERSION}

When the API server is being upgraded, the SkyPilot CLI and Python SDK will automatically retry requests until the new version of the API server is started. So the upgrade process is graceful if the new version of the API server does not break API compatbility. For more details, refer to Graceful upgrade.

Optionally, you can watch the upgrade progress with:

$ kubectl get pod --namespace $NAMESPACE -l app=${RELEASE_NAME}-api --watch
NAME                                       READY   STATUS            RESTARTS   AGE
skypilot-demo-api-server-cf4896bdf-62c96   0/1     Init:0/2          0          7s
skypilot-demo-api-server-cf4896bdf-62c96   0/1     Init:1/2          0          24s
skypilot-demo-api-server-cf4896bdf-62c96   0/1     PodInitializing   0          26s
skypilot-demo-api-server-cf4896bdf-62c96   0/1     Running           0          27s
skypilot-demo-api-server-cf4896bdf-62c96   1/1     Running           0          50s

The upgraded API server is ready to serve requests after the pod becomes running and the READY column shows 1/1.

Note

apiService.config will be IGNORED during an upgrade. To update your SkyPilot config, see here.

Step 3: Verify the upgrade#

Verify the API server is able to serve requests and the version is consistent with the version you upgraded to:

$ sky api info
Using SkyPilot API server: <ENDPOINT>
├── Status: healthy, commit: 022a5c3ffe258f365764b03cb20fac70934f5a60, version: 1.0.0.dev20250410
└── User: aclice (abcd1234)

If possible, you can also trigger your pipelines that depend on the API server to verify there is no compatibility issue after the upgrade.

Upgrade the API server deployed on VM#

Note

VM deployment does not offer graceful upgrade. We recommend the Helm deployment Deploying SkyPilot API Server in production environments. The following is a workaround for upgrading SkyPilot API server in VM deployments.

Suppose the cluster name of the API server is api-server (which is used in the Alternative: Deploy on cloud VMs guide), you can upgrade the API server with the following steps:

  1. Get the version to upgrade to from SkyPilot nightly build.

  2. Switch to the original API server endpoint used to launch the cloud VM for API server. It is usually locally started when you ran sky launch -c api-server skypilot-api-server.yaml in Alternative: Deploy on cloud VMs guide:

# Replace http://localhost:46580 with the real API server endpoint if you were not using the local API server to launch the API server VM instance.
sky api login -e http://localhost:46580

  1. Check the API server VM instance is UP:

$ sky status api-server
Clusters
NAME        LAUNCHED     RESOURCES                                                                  STATUS  AUTOSTOP  COMMAND
api-server  41 mins ago  1x AWS(c6i.2xlarge, image_id={'us-east-1': 'docker:berkeleyskypilot/sk...  UP      -         sky exec api-server pip i...

  1. Upgrade the clients:

pip install -U skypilot-nightly==${VERSION}

Note

After upgrading the clients, they should not be used until the API server is upgraded to the new version.

  1. Upgrade the SkyPilot on the VM and restart the API server:

Note

Upgrading and restarting the API server will interrupt all pending and running requests.

sky exec api-server "pip install -U skypilot-nightly[all] && sky api stop && sky api start --deploy"
# Alternatively, you can also upgrade to a specific version with:
sky exec api-server "pip install -U skypilot-nightly[all]==${VERSION} && sky api stop && sky api start --deploy"

  1. Switch back to the remote API server:

ENDPOINT=$(sky status --endpoint api-server)
sky api login -e $ENDPOINT

  1. Verify the API server is running and the version is consistent with the version you upgraded to:

$ sky api info
Using SkyPilot API server: <ENDPOINT>
├── Status: healthy, commit: 022a5c3ffe258f365764b03cb20fac70934f5a60, version: 1.0.0.dev20250410
└── User: aclice (abcd1234)

Graceful upgrade#

A server can be gracefully upgraded when the following conditions are met:

Behavior when the API server is being upgraded:

  • For critical ongoing requests (e.g., launching a cluster), it waits for them to finish with a timeout.

  • For non-critical ongoing requests (e.g., log tailing), it cancels them and returns an error to ask the client to retry.

  • For new requests, it returns an error to ask the client to retry. New requests will be served when the new version of the API server is ready.

To further reduce the waiting time during upgrade, you can use rolling update for the API server.

SkyPilot Python SDK and CLI will automatically retry until the new version of API server starts, and ongoing requests (e.g., log tailing) will automatically resume:

GIF for graceful upgrade

To ensure that all the regular critical requests can complete within the timeout, you can adjust the timeout by setting apiService.terminationGracePeriodSeconds in helm values based on your workload, e.g.:

helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot-nightly --devel --reuse-values \
  --set apiService.terminationGracePeriodSeconds=300

Upgrade strategy#

By default, the API server is upgraded with the Recreate strategy, which introduces waiting time for new requests during upgrade. To eliminate the waiting time, you can upgrade the API server with the RollingUpdate strategy.

Note

RollingUpdate is an experimental feature. There is a known limitation that some running commands might fail when the old version of the API server gets removed from the ingress backend. It is recommended to schedule the upgrade during a maintenance window.

The following table compares the two upgrade strategies:

Upgrade Strategy Comparison#

Aspect

Recreate

RollingUpdate

Availability

Brief downtime during upgrade

Zero downtime

Request Handling

New requests wait until upgrade completes

New requests served continuously by available replicas

Database Requirements

Can use local storage (SQLite)

Must use external persistent database

Resource Usage During Upgrade

Terminates old API server pod, then starts new one

Starts new API server pod, then terminates old one

Use Cases

Development environments, simple setups

Production environments requiring high availability

To use the RollingUpdate strategy, you need to:

Here’s an example of deploying the API server with the RollingUpdate strategy:

helm install -n $NAMESPACE $RELEASE_NAME skypilot/skypilot-nightly --devel --reuse-values \
  --set apiService.upgradeStrategy=RollingUpdate \
  --set storage.enabled=false \
  --set apiService.dbConnectionSecretName=my-db-secret

Ingress config#

The SkyPilot helm chart automatically configures the ingress resource to achieve higher availability during upgrade. If you are managing the ingress resource outside of the SkyPilot helm chart, refer to the following snippet to improve the availability during upgrades:

Example ingress based on nginx-ingress-controller
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: your-ingress-name
  annotations:
    # Enable session affinity to route the requests of the same client to the same pod during upgrade.
    # Without session affinity, the chance that requests fail during upgrade would be higher.
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "SKYPILOT_ROUTEID"
    nginx.ingress.kubernetes.io/affinity-mode: "persistent"
    nginx.ingress.kubernetes.io/session-cookie-change-on-failure: "true"

API compatbility#

SkyPilot maintain an internal API version which will be bumped when an incompatible API change is introduced. Client and server can only communicate when they run on the same API version.

The version strategy of SkyPilot follows the following API compatbility guarantees:

  • The API version will not be bumped within a minor version, i.e. upgrading patch version is guaranteed to be compatible;

  • The API version might be bumped between minior versions, i.e. upgrading minior version should be treated as operation that breaks API compatibility;

  • There is no guarantee about the API version in the nightly build;