Kubernetes#

When running outside your Kubernetes cluster, SkyPilot uses your local ~/.kube/config file for authentication and creating resources on your Kubernetes cluster.

When running inside your Kubernetes cluster (e.g., as a remote API server, Job controller or Serve controller), SkyPilot can operate using either of the following three authentication methods:

  1. Automatically create a service account: SkyPilot can automatically create the service account and roles for itself to manage resources in the Kubernetes cluster. This is the default method when running inside the cluster, and no additional configuration is required.

    For details on the permissions that are granted to the service account, refer to the Minimum Permissions Required for SkyPilot section below.

  2. Using a custom service account: If you have a custom service account with the necessary permissions, you can configure SkyPilot to use it by adding this to your ~/.sky/config.yaml file:

    kubernetes:
  remote_identity: your-service-account-name

  3. Using your local kubeconfig file: In this case, SkyPilot will copy your local ~/.kube/config file to the controller pod and use it for authentication. To use this method, set remote_identity: LOCAL_CREDENTIALS to your Kubernetes configuration in the ~/.sky/config.yaml file:

    kubernetes:
  remote_identity: LOCAL_CREDENTIALS

    Note

    If your cluster uses exec based authentication in your ~/.kube/config file (e.g., GKE uses exec auth by default), SkyPilot may not be able to authenticate using this method. In this case, consider using the service account methods below.

Note

Service account based authentication applies only when the remote SkyPilot cluster (including spot and serve controller) is launched inside the Kubernetes cluster. When running outside the cluster (e.g., on AWS), SkyPilot will use the local ~/.kube/config file for authentication.

Below are the permissions required by SkyPilot and an example service account YAML that you can use to create a service account with the necessary permissions.

Minimum permissions required for SkyPilot#

SkyPilot requires permissions equivalent to the following roles to be able to manage the resources in the Kubernetes cluster:

# Namespaced role for the service account
# Required for creating pods, services and other necessary resources in the namespace.
# Note these permissions only apply in the namespace where SkyPilot is deployed, and the namespace can be changed below.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sky-sa-role  # Can be changed if needed
  namespace: default  # Change to your namespace if using a different one.
rules:
  # Required for managing pods and their lifecycle
  - apiGroups: [ "" ]
    resources: [ "pods", "pods/status", "pods/exec", "pods/portforward" ]
    verbs: [ "*" ]
  # Required for managing services for SkyPilot Pods
  - apiGroups: [ "" ]
    resources: [ "services" ]
    verbs: [ "*" ]
  # Required for managing SSH keys
  - apiGroups: [ "" ]
    resources: [ "secrets" ]
    verbs: [ "*" ]
  # Required for retrieving reason when Pod scheduling fails.
  - apiGroups: [ "" ]
    resources: [ "events" ]
    verbs: [ "get", "list", "watch" ]
---
# ClusterRole for accessing cluster-wide resources. Details for each resource below:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sky-sa-cluster-role  # Can be changed if needed
  namespace: default  # Change to your namespace if using a different one.
  labels:
    parent: skypilot
rules:
  # Required for getting node resources.
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  # Required for autodetecting the runtime class of the nodes.
  - apiGroups: ["node.k8s.io"]
    resources: ["runtimeclasses"]
    verbs: ["get", "list", "watch"]

Tip

If you are using a different namespace than default, make sure to change the namespace in the above manifests.

These roles must apply to both the user account configured in the kubeconfig file and the service account used by SkyPilot (if configured).

If you need to view real-time GPU availability with sky show-gpus, your tasks use object store mounting or your tasks require access to ingress resources, you will need to grant additional permissions as described below.

Permissions for sky show-gpus#

sky show-gpus needs to list all pods across all namespaces to calculate GPU availability. To do this, SkyPilot needs the get and list permissions for pods in a ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
    name: sky-sa-cluster-role-pod-reader
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]

Tip

If this role is not granted to the service account, sky show-gpus will still work but it will only show the total GPUs on the nodes, not the number of free GPUs.

Permissions for object store mounting#

If your tasks use object store mounting (e.g., S3, GCS, etc.), SkyPilot will need to run a DaemonSet to expose the FUSE device as a Kubernetes resource to SkyPilot pods.

To allow this, you will need to also create a skypilot-system namespace which will run the DaemonSet and grant the necessary permissions to the service account in that namespace.

# Required only if using object store mounting
# Create namespace for SkyPilot system
apiVersion: v1
kind: Namespace
metadata:
  name: skypilot-system  # Do not change this
  labels:
    parent: skypilot
---
# Role for the skypilot-system namespace to create fusermount-server and
# any other system components required by SkyPilot.
# This role must be bound in the skypilot-system namespace to the service account used for SkyPilot.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: skypilot-system-service-account-role  # Can be changed if needed
  namespace: skypilot-system  # Do not change this namespace
  labels:
    parent: skypilot
rules:
  - apiGroups: [ "*" ]
    resources: [ "apps" ]
    verbs: [ "daemonsets" ]

Permissions for using Ingress#

If your tasks use Ingress for exposing ports, you will need to grant the necessary permissions to the service account in the ingress-nginx namespace.

# Required only if using ingresses
# Role for accessing ingress service IP
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: ingress-nginx  # Do not change this
  name: sky-sa-role-ingress-nginx  # Can be changed if needed
rules:
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["list", "get"]

Example using custom service account#

To create a service account that has all necessary permissions for SkyPilot (including for accessing object stores), you can use the following YAML.

Tip

In this example, the service account is named sky-sa and is created in the default namespace. Change the namespace and service account name as needed.

  1 # create-sky-sa.yaml
  2 kind: ServiceAccount
  3 apiVersion: v1
  4 metadata:
  5   name: sky-sa  # Change to your service account name
  6   namespace: default  # Change to your namespace if using a different one.
  7   labels:
  8     parent: skypilot
  9 ---
 10 # Role for the service account
 11 kind: Role
 12 apiVersion: rbac.authorization.k8s.io/v1
 13 metadata:
 14   name: sky-sa-role  # Can be changed if needed
 15   namespace: default  # Change to your namespace if using a different one.
 16   labels:
 17     parent: skypilot
 18 rules:
 19   # Required for managing pods and their lifecycle
 20   - apiGroups: [ "" ]
 21     resources: [ "pods", "pods/status", "pods/exec", "pods/portforward" ]
 22     verbs: [ "*" ]
 23   # Required for managing services for SkyPilot Pods
 24   - apiGroups: [ "" ]
 25     resources: [ "services" ]
 26     verbs: [ "*" ]
 27   # Required for managing SSH keys
 28   - apiGroups: [ "" ]
 29     resources: [ "secrets" ]
 30     verbs: [ "*" ]
 31   # Required for retrieving reason when Pod scheduling fails.
 32   - apiGroups: [ "" ]
 33     resources: [ "events" ]
 34     verbs: [ "get", "list", "watch" ]
 35 ---
 36 # RoleBinding for the service account
 37 kind: RoleBinding
 38 apiVersion: rbac.authorization.k8s.io/v1
 39 metadata:
 40   name: sky-sa-rb  # Can be changed if needed
 41   namespace: default  # Change to your namespace if using a different one.
 42   labels:
 43     parent: skypilot
 44 subjects:
 45   - kind: ServiceAccount
 46     name: sky-sa  # Change to your service account name
 47 roleRef:
 48   kind: Role
 49   name: sky-sa-role  # Use the same name as the role at line 14
 50   apiGroup: rbac.authorization.k8s.io
 51 ---
 52 # ClusterRole for the service account
 53 kind: ClusterRole
 54 apiVersion: rbac.authorization.k8s.io/v1
 55 metadata:
 56   name: sky-sa-cluster-role  # Can be changed if needed
 57   namespace: default  # Change to your namespace if using a different one.
 58   labels:
 59     parent: skypilot
 60 rules:
 61   - apiGroups: [""]
 62     resources: ["nodes"]  # Required for getting node resources.
 63     verbs: ["get", "list", "watch"]
 64   - apiGroups: ["node.k8s.io"]
 65     resources: ["runtimeclasses"]   # Required for autodetecting the runtime class of the nodes.
 66     verbs: ["get", "list", "watch"]
 67   - apiGroups: ["networking.k8s.io"]   # Required for exposing services through ingresses
 68     resources: ["ingressclasses"]
 69     verbs: ["get", "list", "watch"]
 70   - apiGroups: [""]                 # Required for `sky show-gpus` command
 71     resources: ["pods"]
 72     verbs: ["get", "list"]
 73 ---
 74 # ClusterRoleBinding for the service account
 75 apiVersion: rbac.authorization.k8s.io/v1
 76 kind: ClusterRoleBinding
 77 metadata:
 78   name: sky-sa-cluster-role-binding  # Can be changed if needed
 79   namespace: default  # Change to your namespace if using a different one.
 80   labels:
 81     parent: skypilot
 82 subjects:
 83   - kind: ServiceAccount
 84     name: sky-sa  # Change to your service account name
 85     namespace: default  # Change to your namespace if using a different one.
 86 roleRef:
 87   kind: ClusterRole
 88   name: sky-sa-cluster-role  # Use the same name as the cluster role at line 43
 89   apiGroup: rbac.authorization.k8s.io
 90 ---
 91 # Optional: If using object store mounting, create the skypilot-system namespace
 92 apiVersion: v1
 93 kind: Namespace
 94 metadata:
 95   name: skypilot-system  # Do not change this
 96   labels:
 97     parent: skypilot
 98 ---
 99 # Optional: If using object store mounting, create role in the skypilot-system
100 # namespace to create fusermount-server.
101 kind: Role
102 apiVersion: rbac.authorization.k8s.io/v1
103 metadata:
104   name: skypilot-system-service-account-role  # Can be changed if needed
105   namespace: skypilot-system  # Do not change this namespace
106   labels:
107     parent: skypilot
108 rules:
109   - apiGroups: [ "apps" ]
110     resources: [ "daemonsets" ]
111     verbs: [ "*" ]
112 ---
113 # Optional: If using object store mounting, create rolebinding in the skypilot-system
114 # namespace to create fusermount-server.
115 apiVersion: rbac.authorization.k8s.io/v1
116 kind: RoleBinding
117 metadata:
118   name: sky-sa-skypilot-system-role-binding
119   namespace: skypilot-system  # Do not change this namespace
120   labels:
121     parent: skypilot
122 subjects:
123   - kind: ServiceAccount
124     name: sky-sa  # Change to your service account name
125     namespace: default  # Change this to the namespace where the service account is created
126 roleRef:
127   kind: Role
128   name: skypilot-system-service-account-role  # Use the same name as the role above
129   apiGroup: rbac.authorization.k8s.io
130 ---
131 # Optional: Role for accessing ingress resources
132 apiVersion: rbac.authorization.k8s.io/v1
133 kind: Role
134 metadata:
135   name: sky-sa-role-ingress-nginx  # Can be changed if needed
136   namespace: ingress-nginx  # Do not change this namespace
137   labels:
138     parent: skypilot
139 rules:
140   - apiGroups: [""]
141     resources: ["services"]
142     verbs: ["list", "get", "watch"]
143 ---
144 # Optional: RoleBinding for accessing ingress resources
145 apiVersion: rbac.authorization.k8s.io/v1
146 kind: RoleBinding
147 metadata:
148   name: sky-sa-rolebinding-ingress-nginx  # Can be changed if needed
149   namespace: ingress-nginx  # Do not change this namespace
150   labels:
151     parent: skypilot
152 subjects:
153   - kind: ServiceAccount
154     name: sky-sa  # Change to your service account name
155     namespace: default  # Change this to the namespace where the service account is created
156 roleRef:
157   kind: Role
158   name: sky-sa-role-ingress-nginx  # Use the same name as the role above
159   apiGroup: rbac.authorization.k8s.io

Create the service account using the following command:

$ kubectl apply -f create-sky-sa.yaml

After creating the service account, the cluster admin may distribute kubeconfigs with the sky-sa service account to users who need to access the cluster.

Users should also configure SkyPilot to use the sky-sa service account through ~/.sky/config.yaml:

# ~/.sky/config.yaml
kubernetes:
  remote_identity: sky-sa   # Or your service account name