SkyPilot API Server Helm Chart Values

The SkyPilot API server helm chart provides typical helm values as configuration entries. Configuration values can be passed in two ways when installing the chart:

• --values (or -f): Specify a YAML file with overrides.

  cat <<EOF > values.yaml
  apiService:
    image: berkeleyskypilot/skypilot:0.9.1
  EOF
  
  helm install $RELEASE_NAME skypilot/skypilot-nightly --devel --values values.yaml
  

• --set: Specify overrides on the command line.

  helm install $RELEASE_NAME skypilot/skypilot-nightly --set apiService.image="berkeleyskypilot/skypilot:0.9.1"
  

Values

Below is the available helm value keys and the default value of each key:

apiService:
  image: berkeleyskypilot/skypilot:0.9.1
  preDeployHook: |-
    # Run commands before deploying the API server, e.g. installing an admin
    # policy. Remember to set the admin policy in the config section below.

    echo "Pre-deploy hook"

    # Uncomment the following lines to install the admin policy

    # echo "Installing admin policy"
    # pip install git+https://github.com/michaelvll/admin-policy-examples
  config: null
  sshNodePools: null
  sshKeySecret: null
  skipResourceCheck: false
  resources:
    requests:
      cpu: "4"
      memory: "8Gi"
    limits:
      cpu: "4"
      memory: "8Gi"
  skypilotDev: false

storage:
  enabled: true
  storageClassName: ""
  size: 10Gi
  selector: {}
  volumeName: ""
  annotations: {}

ingress:
  enabled: true
  authSecret: null
  authCredentials: "username:$apr1$encrypted_password"
  path: '/'
  oauth2-proxy:
    enabled: false
    # Required when enabled:
    oidc-issuer-url: null
    client-id: ""
    client-secret: ""
    # Optional settings:
    image: "quay.io/oauth2-proxy/oauth2-proxy:v7.9.0"
    use-https: false
    email-domain: "*"
    session-store-type: "redis"
    redis-url: null
    cookie-refresh: null
    cookie-expire: null

ingress-nginx:
  enabled: true
  controller:
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
        cloud.google.com/l4-rbs: "enabled"
        service.beta.kubernetes.io/port_443_health-probe_protocol: "TCP"
        service.beta.kubernetes.io/port_80_health-probe_protocol: "TCP"
    config:
      http-snippet: |
        map $http_upgrade $connection_upgrade {
            default upgrade;
            ''      close;
        }

rbac:
  create: true
  serviceAccountName: ""
  namespaceRules:
    - apiGroups: [ "" ]
      resources: [ "pods", "pods/status", "pods/exec", "pods/portforward" ]
      verbs: [ "*" ]
    - apiGroups: [ "" ]
      resources: [ "services" ]
      verbs: [ "*" ]
    - apiGroups: [ "" ]
      resources: [ "secrets" ]
      verbs: [ "*" ]
    - apiGroups: [ "" ]
      resources: [ "events" ]
      verbs: [ "get", "list", "watch" ]
  clusterRules:
    - apiGroups: [ "" ]
      resources: [ "nodes" ]
      verbs: [ "get", "list", "watch" ]
    - apiGroups: [ "" ]
      resources: [ "pods" ]
      verbs: [ "get", "list", "watch" ]
    - apiGroups: [ "node.k8s.io" ]
      resources: [ "runtimeclasses" ]
      verbs: [ "get", "list", "watch" ]
    - apiGroups: [ "networking.k8s.io" ]
      resources: [ "ingressclasses" ]
      verbs: [ "get", "list", "watch" ]
    - apiGroups: [""]
      resources: ["services"]
      verbs: ["list", "get"]
  manageRbacPolicies: true
  manageSystemComponents: true

kubernetesCredentials:
  useApiServerCluster: true
  useKubeconfig: false
  kubeconfigSecretName: kube-credentials
  inclusterNamespace: null

awsCredentials:
  enabled: false
  awsSecretName: aws-credentials
  accessKeyIdKeyName: aws_access_key_id
  secretAccessKeyKeyName: aws_secret_access_key

gcpCredentials:
  enabled: false
  projectId: null
  gcpSecretName: gcp-credentials

podSecurityContext: {}

securityContext:
  capabilities:
    drop:
    - ALL
  allowPrivilegeEscalation: false

runtimeClassName: ""

Fields

apiService

Configuration for the SkyPilot API server deployment.

apiService.image

Docker image to use for the API server.

Default: "berkeleyskypilot/skypilot:0.9.1"

apiService:
  image: berkeleyskypilot/skypilot:0.9.1

To use a nightly build, find the desired nightly version on pypi and update the image value:

apiService:
  # Replace 1.0.0.devYYYYMMDD with the desired nightly version
  image: berkeleyskypilot/skypilot-nightly:1.0.0.devYYYYMMDD

apiService.preDeployHook

Commands to run before deploying the API server (e.g., install admin policy).

Default: see the yaml below.

apiService:
  preDeployHook: |-
    # Run commands before deploying the API server, e.g. installing an admin
    # policy. Remember to set the admin policy in the config section below.
    echo "Pre-deploy hook"

    # Uncomment the following lines to install the admin policy
    # echo "Installing admin policy"
    # pip install git+https://github.com/michaelvll/admin-policy-examples

apiService.config

Content of the SkyPilot config.yaml to set on the API server. Set to null to use an empty config. Refer to setting the SkyPilot config for more details.

Default: null

apiService:
  config: |-
    allowed_clouds:
      - aws
      - gcp

apiService.sshNodePools

Content of the ~/.sky/ssh_node_pools.yaml to set on the API server. Set to null to use an empty ssh node pools. Refer to Deploy SkyPilot on existing machines for more details.

Default: null

apiService:
  sshNodePools: |-
    my-cluster:
      hosts:
        - 1.2.3.4
        - 1.2.3.5

    my-box:
      hosts:
        - hostname_in_ssh_config

apiService.sshKeySecret

Optional secret that contains SSH identity files to the API server to use, all the entries in the secret will be mounted to ~/.ssh/ directory in the API server. Refer to Deploy SkyPilot on existing machines for more details.

Default: null

apiService:
  sshKeySecret: my-ssh-key-secret

The content of the secret should be like:

apiVersion: v1
kind: Secret
metadata:
  name: my-ssh-key-secret
data:
  id_rsa: <secret-content>

apiService.skipResourceCheck

Skip resource check for the API server (not recommended for production), refer to tuning API server resources for more details.

Default: false

apiService:
  skipResourceCheck: false

apiService.resources

Resource requests and limits for the API server container. Refer to tuning API server resources for how to tune the resources.

Default: see the yaml below.

apiService:
  resources:
    requests:
      cpu: "4"
      memory: "8Gi"
    limits:
      cpu: "4"
      memory: "8Gi"

apiService.skypilotDev

Enable developer mode for SkyPilot.

Default: false

apiService:
  skypilotDev: false

storage

storage.enabled

Enable persistent storage for the API server, setting this to false is prone to data loss and should only be used for testing.

Default: true

storage:
  enabled: true

storage.storageClassName

Storage class to use for the API server, leave empty to use the default storage class of the hosting Kubernetes cluster.

Default: ""

storage:
  storageClassName: gp2

storage.size

Size of the persistent storage volume for the API server.

Default: 10Gi

storage:
  size: 10Gi

storage.selector

Selector for matching specific PersistentVolumes. Usually left empty.

Default: {}

storage:
  selector: {}

storage.volumeName

Name of the PersistentVolume to bind to. Usually left empty to let Kubernetes select and bind the volume automatically.

Default: ""

storage:
  volumeName: ""

storage.annotations

Annotations to add to the PersistentVolumeClaim.

Default: {}

storage:
  annotations: {}

ingress

ingress.enabled

Enable ingress for the API server. Set to true to expose the API server via an ingress controller.

Default: true

ingress:
  enabled: true

ingress.authSecret

Name of the Kubernetes secret containing basic auth credentials for ingress. If not specified, a new secret will be created using authCredentials. This is ignored if ingress.oauth2-proxy.enabled is true.

One of ingress.authSecret or ingress.authCredentials must be set, unless ingress.oauth2-proxy.enabled is true.

Default: null

ingress:
  authSecret: null

ingress.authCredentials

Basic auth credentials in the format username:encrypted_password. Used only if authSecret is not set. This is ignored if ingress.oauth2-proxy.enabled is true.

One of ingress.authSecret or ingress.authCredentials must be set, unless ingress.oauth2-proxy.enabled is true.

Default: "username:$apr1$encrypted_password"

ingress:
  authCredentials: "username:$apr1$encrypted_password"

ingress.path

The base path of the API server. You may use different paths to expose multiple API servers through a unified ingress controller.

Default: '/'

ingress:
  path: '/'

ingress.oauth2-proxy

Configuration for the OAuth2 Proxy authentication for the API server. This enables SSO providers like Okta.

If enabled, ingress.authSecret and ingress.authCredentials are ignored.

Default: see the yaml below.

ingress:
  oauth2-proxy:
    enabled: false
    # Required when enabled:
    oidc-issuer-url: null
    client-id: ""
    client-secret: ""
    # Optional settings:
    image: "quay.io/oauth2-proxy/oauth2-proxy:v7.9.0"
    use-https: false
    email-domain: "*"
    session-store-type: "redis"
    redis-url: null
    cookie-refresh: null
    cookie-expire: null

ingress.oauth2-proxy.enabled

Enable OAuth2 Proxy for authentication. When enabled, this will deploy an OAuth2 Proxy component and configure the ingress to use it for authentication instead of basic auth.

Default: false

ingress:
  oauth2-proxy:
    enabled: true

ingress.oauth2-proxy.oidc-issuer-url

The URL of the OIDC issuer (e.g., your Okta domain). Required when oauth2-proxy is enabled.

Default: null

ingress:
  oauth2-proxy:
    oidc-issuer-url: "https://mycompany.okta.com"

ingress.oauth2-proxy.client-id

The OAuth client ID from your OIDC provider (e.g., Okta). Required when oauth2-proxy is enabled.

Default: ""

ingress:
  oauth2-proxy:
    client-id: "0abc123def456"

ingress.oauth2-proxy.client-secret

The OAuth client secret from your OIDC provider (e.g., Okta). Required when oauth2-proxy is enabled.

Default: ""

ingress:
  oauth2-proxy:
    client-secret: "abcdef123456"

ingress.oauth2-proxy.image

Docker image for the OAuth2 Proxy component.

Default: "quay.io/oauth2-proxy/oauth2-proxy:v7.9.0"

ingress:
  oauth2-proxy:
    image: "quay.io/oauth2-proxy/oauth2-proxy:v7.9.0"

ingress.oauth2-proxy.use-https

Set to true when using HTTPS for the API server endpoint. When set to false, secure cookies are disabled, which is required for HTTP endpoints.

Default: false

ingress:
  oauth2-proxy:
    use-https: true

ingress.oauth2-proxy.email-domain

Email domains to allow for authentication. Use "*" to allow all email domains.

Default: "*"

ingress:
  oauth2-proxy:
    email-domain: "mycompany.com"

ingress.oauth2-proxy.session-store-type

Session storage type for OAuth2 Proxy. Can be set to "cookie" or "redis". Using Redis as a session store results in smaller cookies and better performance for large-scale deployments.

Default: "redis"

ingress:
  oauth2-proxy:
    session-store-type: "redis"

ingress.oauth2-proxy.redis-url

URL to connect to an external Redis instance for session storage. If set to null and session-store-type is "redis", a Redis instance will be automatically deployed. Format: redis://host[:port][/db-number]

Default: null

ingress:
  oauth2-proxy:
    redis-url: "redis://redis-host:6379/0"

ingress.oauth2-proxy.cookie-refresh

Duration in seconds after which to refresh the access token. This should typically be set to the access token lifespan minus 1 minute. If not set, tokens will not be refreshed automatically.

Default: null

ingress:
  oauth2-proxy:
    cookie-refresh: 3540  # 59 minutes (for a 60-minute access token)

ingress.oauth2-proxy.cookie-expire

Expiration time for cookies in seconds. Should match the refresh token lifespan from your OIDC provider.

Default: null

ingress:
  oauth2-proxy:
    cookie-expire: 86400  # 24 hours

ingress-nginx

ingress-nginx.enabled

Enable the ingress-nginx controller for the API server. If you have an existing ingress-nginx controller, you have to set this to false to avoid conflict.

Default: true

ingress-nginx:
  enabled: true

ingress-nginx.controller

Fields under ingress-nginx.controller will be mapped to controller values for the ingress-nginx controller sub-chart. Refer to the ingress-nginx chart documentation for more details.

Default: see the yaml below.

ingress-nginx:
  controller:
    service:
      # Service type of the ingress controller.
      type: LoadBalancer
      # Annotations for the ingress controller service.
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
        cloud.google.com/l4-rbs: "enabled"
        service.beta.kubernetes.io/port_443_health-probe_protocol: "TCP"
        service.beta.kubernetes.io/port_80_health-probe_protocol: "TCP"
    config:
      # Custom HTTP snippet to inject into the ingress-nginx configuration.
      http-snippet: |
        map $http_upgrade $connection_upgrade {
            default upgrade;
            ''      close;
        }

rbac

rbac.create

Whether to create the service account and RBAC policies for the API server. If false, an external service account is expected.

Default: true

rbac:
  create: true

rbac.serviceAccountName

Name of the service account to use. Leave empty to let the chart generate one.

Default: ""

rbac:
  serviceAccountName: ""

rbac.namespaceRules

Namespace-scoped RBAC rules granted to the namespace where the SkyPilot tasks will be launched.

Note

Modifying the rules may break functionalities of SkyPilot API server. Refer to setting minimum permissions in helm deployment for how to modify the rules based on your use case.

Default: see the yaml below.

rbac:
  namespaceRules:
    - apiGroups: [ "" ]
      resources: [ "pods", "pods/status", "pods/exec", "pods/portforward" ]
      verbs: [ "*" ]
    - apiGroups: [ "" ]
      resources: [ "services" ]
      verbs: [ "*" ]
    - apiGroups: [ "" ]
      resources: [ "secrets" ]
      verbs: [ "*" ]
    - apiGroups: [ "" ]
      resources: [ "events" ]
      verbs: [ "get", "list", "watch" ]

rbac.clusterRules

Cluster-scoped RBAC rules for the API server.

Note

Modifying the rules may break functionalities of SkyPilot API server. Refer to setting minimum permissions in helm deployment for how to modify the rules based on your use case.

Default: see the yaml below.

rbac:
  clusterRules:
    - apiGroups: [ "" ]
      resources: [ "nodes" ]
      verbs: [ "get", "list", "watch" ]
    - apiGroups: [ "" ]
      resources: [ "pods" ]
      verbs: [ "get", "list", "watch" ]
    - apiGroups: [ "node.k8s.io" ]
      resources: [ "runtimeclasses" ]
      verbs: [ "get", "list", "watch" ]
    - apiGroups: [ "networking.k8s.io" ]
      resources: [ "ingressclasses" ]
      verbs: [ "get", "list", "watch" ]
    - apiGroups: ["" ]
      resources: ["services"]
      verbs: ["list", "get"]

rbac.manageRbacPolicies

Allow the API server to grant permissions to SkyPilot Pods and system components. Refer to setting minimum permissions in helm deployment for more details.

Default: true

rbac:
  manageRbacPolicies: true

rbac.manageSystemComponents

Allow the API server to manage system components in the skypilot-system namespace. Required for object store mounting.

Default: true

rbac:
  manageSystemComponents: true

kubernetesCredentials

kubernetesCredentials.useApiServerCluster

Enable using the API server’s cluster for workloads.

Default: true

kubernetesCredentials:
  useApiServerCluster: true

kubernetesCredentials.useKubeconfig

Use the kube-credentials secret containing the kubeconfig to authenticate to Kubernetes.

Default: false

kubernetesCredentials:
  useKubeconfig: false

kubernetesCredentials.kubeconfigSecretName

Name of the secret containing the kubeconfig file. Only used if useKubeconfig is true.

Default: kube-credentials

kubernetesCredentials:
  kubeconfigSecretName: kube-credentials

kubernetesCredentials.inclusterNamespace

Namespace to use for in-cluster resources.

Default: null

kubernetesCredentials:
  inclusterNamespace: null

awsCredentials

awsCredentials.enabled

Enable AWS credentials for the API server.

Default: false

awsCredentials:
  enabled: false

awsCredentials.awsSecretName

Name of the secret containing the AWS credentials. Only used if enabled is true.

Default: aws-credentials

awsCredentials:
  awsSecretName: aws-credentials

awsCredentials.accessKeyIdKeyName

Key name used to set AWS_ACCESS_KEY_ID.

Default: aws_access_key_id

awsCredentials:
  accessKeyIdKeyName: aws_access_key_id

awsCredentials.secretAccessKeyKeyName

Key name used to set AWS_SECRET_ACCESS_KEY.

Default: aws_secret_access_key

awsCredentials:
  secretAccessKeyKeyName: aws_secret_access_key

gcpCredentials

gcpCredentials.enabled

Enable GCP credentials for the API server.

Default: false

gcpCredentials:
  enabled: false

gcpCredentials.projectId

GCP project ID. Only used if enabled is true.

Default: null

gcpCredentials:
  projectId: null

gcpCredentials.gcpSecretName

Name of the secret containing the GCP credentials. Only used if enabled is true.

Default: gcp-credentials

gcpCredentials:
  gcpSecretName: gcp-credentials

podSecurityContext

Security context for the API server pod. Usually left empty to use defaults. Refer to set the security context for Pod for more details.

Default: {}

podSecurityContext:
  runAsUser: 1000
  runAsGroup: 3000
  fsGroup: 2000

securityContext

securityContext.capabilities

Linux capabilities to drop for the API server container.

Default: drop all capabilities.

securityContext:
  capabilities:
    drop:
    - ALL

securityContext.allowPrivilegeEscalation

Whether to allow privilege escalation in the API server container.

Default: false

securityContext:
  allowPrivilegeEscalation: false

runtimeClassName

The runtime class to use for the API server pod. Usually left empty to use the default runtime class.

Default: (empty)

runtimeClassName: