Kubernetes#

When running outside your Kubernetes cluster, SkyPilot uses your local ~/.kube/config file for authentication and creating resources on your Kubernetes cluster.

When running inside your Kubernetes cluster (e.g., as a remote API server, Job controller or Serve controller), SkyPilot can operate using either of the following three authentication methods:

  1. Automatically create a service account: SkyPilot can automatically create the service account and roles for itself to manage resources in the Kubernetes cluster. This is the default method when running inside the cluster, and no additional configuration is required.

    For details on the permissions that are granted to the service account, refer to the Minimum Permissions Required for SkyPilot section below.

  2. Using a custom service account: If you have a custom service account with the necessary permissions, you can configure SkyPilot to use it by adding this to your ~/.sky/config.yaml file:

    kubernetes:
  remote_identity: your-service-account-name

  3. Using your local kubeconfig file: In this case, SkyPilot will copy your local ~/.kube/config file to the controller pod and use it for authentication. To use this method, set remote_identity: LOCAL_CREDENTIALS to your Kubernetes configuration in the ~/.sky/config.yaml file:

    kubernetes:
  remote_identity: LOCAL_CREDENTIALS

    Note

    If your cluster uses exec based authentication in your ~/.kube/config file (e.g., GKE uses exec auth by default), SkyPilot may not be able to authenticate using this method. In this case, consider using the service account methods below.

Note

Service account based authentication applies only when the remote SkyPilot cluster (including spot and serve controller) is launched inside the Kubernetes cluster. When running outside the cluster (e.g., on AWS), SkyPilot will use the local ~/.kube/config file for authentication.

Below are the permissions required by SkyPilot and an example service account YAML that you can use to create a service account with the necessary permissions.

Minimum permissions required for SkyPilot#

SkyPilot requires permissions equivalent to the following roles to be able to manage the resources in the Kubernetes cluster:

# Namespaced role for the service account
# Required for creating pods, services and other necessary resources in the namespace.
# Note these permissions only apply in the namespace where SkyPilot is deployed, and the namespace can be changed below.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sky-sa-role  # Can be changed if needed
  namespace: default  # Change to your namespace if using a different one.
rules:
  # Required for managing pods and their lifecycle
  - apiGroups: [ "" ]
    resources: [ "pods", "pods/status", "pods/exec", "pods/portforward" ]
    verbs: [ "*" ]
  # Required for managing services for SkyPilot Pods
  - apiGroups: [ "" ]
    resources: [ "services" ]
    verbs: [ "*" ]
  # Required for managing SSH keys
  - apiGroups: [ "" ]
    resources: [ "secrets" ]
    verbs: [ "*" ]
  # Required for retrieving reason when Pod scheduling fails.
  - apiGroups: [ "" ]
    resources: [ "events" ]
    verbs: [ "get", "list", "watch" ]
---
# ClusterRole for accessing cluster-wide resources. Details for each resource below:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sky-sa-cluster-role  # Can be changed if needed
  namespace: default  # Change to your namespace if using a different one.
  labels:
    parent: skypilot
rules:
  # Required for getting node resources.
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  # Required for autodetecting the runtime class of the nodes.
  - apiGroups: ["node.k8s.io"]
    resources: ["runtimeclasses"]
    verbs: ["get", "list", "watch"]
  # Required for accessing storage classes.
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]

Tip

If you are using a different namespace than default, make sure to change the namespace in the above manifests.

These roles must apply to both the user account configured in the kubeconfig file and the service account used by SkyPilot (if configured).

If you need to view real-time GPU availability with sky gpus list, your tasks use object store mounting or your tasks require access to ingress resources, you will need to grant additional permissions as described below.

Permissions for sky gpus list#

sky gpus list needs to list all pods across all namespaces to calculate GPU availability. To do this, SkyPilot needs the get and list permissions for pods in a ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
    name: sky-sa-cluster-role-pod-reader
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]

Tip

If this role is not granted to the service account, sky gpus list will still work but it will only show the total GPUs on the nodes, not the number of free GPUs.

Permissions for object store mounting#

If your tasks use object store mounting (e.g., S3, GCS, etc.), SkyPilot will need to run a DaemonSet to expose the FUSE device as a Kubernetes resource to SkyPilot pods.

To allow this, you will need to also create a skypilot-system namespace which will run the DaemonSet and grant the necessary permissions to the service account in that namespace.

# Required only if using object store mounting
# Create namespace for SkyPilot system
apiVersion: v1
kind: Namespace
metadata:
  name: skypilot-system  # Do not change this
  labels:
    parent: skypilot
---
# Role for the skypilot-system namespace to create fusermount-server and
# any other system components required by SkyPilot.
# This role must be bound in the skypilot-system namespace to the service account used for SkyPilot.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: skypilot-system-service-account-role  # Can be changed if needed
  namespace: skypilot-system  # Do not change this namespace
  labels:
    parent: skypilot
rules:
  - apiGroups: [ "*" ]
    resources: [ "apps" ]
    verbs: [ "daemonsets" ]

Permissions for using Ingress#

If your tasks use Ingress for exposing ports, you will need to grant the necessary permissions to the service account in the ingress-nginx namespace.

# Required only if using ingresses
# Role for accessing ingress service IP
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: ingress-nginx  # Do not change this
  name: sky-sa-role-ingress-nginx  # Can be changed if needed
rules:
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["list", "get"]

Example using custom service account#

To create a service account that has all necessary permissions for SkyPilot (including for accessing object stores), you can use the following YAML.

Tip

In this example, the service account is named sky-sa and is created in the default namespace. Change the namespace and service account name as needed.

  1 # create-sky-sa.yaml
  2 kind: ServiceAccount
  3 apiVersion: v1
  4 metadata:
  5   name: sky-sa  # Change to your service account name
  6   namespace: default  # Change to your namespace if using a different one.
  7   labels:
  8     parent: skypilot
  9 ---
 10 # Role for the service account
 11 kind: Role
 12 apiVersion: rbac.authorization.k8s.io/v1
 13 metadata:
 14   name: sky-sa-role  # Can be changed if needed
 15   namespace: default  # Change to your namespace if using a different one.
 16   labels:
 17     parent: skypilot
 18 rules:
 19   # Required for managing pods and their lifecycle
 20   - apiGroups: [ "" ]
 21     resources: [ "pods", "pods/status", "pods/exec", "pods/portforward" ]
 22     verbs: [ "*" ]
 23   # Required for managing services for SkyPilot Pods
 24   - apiGroups: [ "" ]
 25     resources: [ "services" ]
 26     verbs: [ "*" ]
 27   # Required for managing SSH keys
 28   - apiGroups: [ "" ]
 29     resources: [ "secrets" ]
 30     verbs: [ "*" ]
 31   # Required for retrieving reason when Pod scheduling fails.
 32   - apiGroups: [ "" ]
 33     resources: [ "events" ]
 34     verbs: [ "get", "list", "watch" ]
 35 ---
 36 # RoleBinding for the service account
 37 kind: RoleBinding
 38 apiVersion: rbac.authorization.k8s.io/v1
 39 metadata:
 40   name: sky-sa-rb  # Can be changed if needed
 41   namespace: default  # Change to your namespace if using a different one.
 42   labels:
 43     parent: skypilot
 44 subjects:
 45   - kind: ServiceAccount
 46     name: sky-sa  # Change to your service account name
 47 roleRef:
 48   kind: Role
 49   name: sky-sa-role  # Use the same name as the role at line 14
 50   apiGroup: rbac.authorization.k8s.io
 51 ---
 52 # ClusterRole for the service account
 53 kind: ClusterRole
 54 apiVersion: rbac.authorization.k8s.io/v1
 55 metadata:
 56   name: sky-sa-cluster-role  # Can be changed if needed
 57   labels:
 58     parent: skypilot
 59 rules:
 60   - apiGroups: [""]
 61     resources: ["nodes"]  # Required for getting node resources.
 62     verbs: ["get", "list", "watch"]
 63   - apiGroups: ["node.k8s.io"]
 64     resources: ["runtimeclasses"]   # Required for autodetecting the runtime class of the nodes.
 65     verbs: ["get", "list", "watch"]
 66   - apiGroups: ["networking.k8s.io"]   # Required for exposing services through ingresses
 67     resources: ["ingressclasses"]
 68     verbs: ["get", "list", "watch"]
 69   - apiGroups: [""]                 # Required for `sky gpus list` command
 70     resources: ["pods"]
 71     verbs: ["get", "list"]
 72   - apiGroups: ["storage.k8s.io"]   # Required for using volumes
 73     resources: ["storageclasses"]
 74     verbs: ["get", "list", "watch"]
 75 ---
 76 # ClusterRoleBinding for the service account
 77 apiVersion: rbac.authorization.k8s.io/v1
 78 kind: ClusterRoleBinding
 79 metadata:
 80   name: sky-sa-cluster-role-binding  # Can be changed if needed
 81   labels:
 82     parent: skypilot
 83 subjects:
 84   - kind: ServiceAccount
 85     name: sky-sa  # Change to your service account name
 86     namespace: default  # Change to your namespace if using a different one.
 87 roleRef:
 88   kind: ClusterRole
 89   name: sky-sa-cluster-role  # Use the same name as the cluster role at line 56
 90   apiGroup: rbac.authorization.k8s.io
 91 ---
 92 # Optional: If using object store mounting, create the skypilot-system namespace
 93 apiVersion: v1
 94 kind: Namespace
 95 metadata:
 96   name: skypilot-system  # Do not change this
 97   labels:
 98     parent: skypilot
 99 ---
100 # Optional: If using object store mounting, create role in the skypilot-system
101 # namespace to create fusermount-server.
102 kind: Role
103 apiVersion: rbac.authorization.k8s.io/v1
104 metadata:
105   name: skypilot-system-service-account-role  # Can be changed if needed
106   namespace: skypilot-system  # Do not change this namespace
107   labels:
108     parent: skypilot
109 rules:
110   - apiGroups: [ "apps" ]
111     resources: [ "daemonsets" ]
112     verbs: [ "*" ]
113 ---
114 # Optional: If using object store mounting, create rolebinding in the skypilot-system
115 # namespace to create fusermount-server.
116 apiVersion: rbac.authorization.k8s.io/v1
117 kind: RoleBinding
118 metadata:
119   name: sky-sa-skypilot-system-role-binding
120   namespace: skypilot-system  # Do not change this namespace
121   labels:
122     parent: skypilot
123 subjects:
124   - kind: ServiceAccount
125     name: sky-sa  # Change to your service account name
126     namespace: default  # Change this to the namespace where the service account is created
127 roleRef:
128   kind: Role
129   name: skypilot-system-service-account-role  # Use the same name as the role above
130   apiGroup: rbac.authorization.k8s.io
131 ---
132 # Optional: Role for accessing ingress resources
133 apiVersion: rbac.authorization.k8s.io/v1
134 kind: Role
135 metadata:
136   name: sky-sa-role-ingress-nginx  # Can be changed if needed
137   namespace: ingress-nginx  # Do not change this namespace
138   labels:
139     parent: skypilot
140 rules:
141   - apiGroups: [""]
142     resources: ["services"]
143     verbs: ["list", "get", "watch"]
144 ---
145 # Optional: RoleBinding for accessing ingress resources
146 apiVersion: rbac.authorization.k8s.io/v1
147 kind: RoleBinding
148 metadata:
149   name: sky-sa-rolebinding-ingress-nginx  # Can be changed if needed
150   namespace: ingress-nginx  # Do not change this namespace
151   labels:
152     parent: skypilot
153 subjects:
154   - kind: ServiceAccount
155     name: sky-sa  # Change to your service account name
156     namespace: default  # Change this to the namespace where the service account is created
157 roleRef:
158   kind: Role
159   name: sky-sa-role-ingress-nginx  # Use the same name as the role above
160   apiGroup: rbac.authorization.k8s.io

Create the service account using the following command:

$ kubectl apply -f create-sky-sa.yaml

After creating the service account, the cluster admin may distribute kubeconfigs with the sky-sa service account to users who need to access the cluster.

Users should also configure SkyPilot to use the sky-sa service account through ~/.sky/config.yaml:

# ~/.sky/config.yaml
kubernetes:
  remote_identity: sky-sa   # Or your service account name