Source: examples/github_actions
Example: GitHub Actions + SkyPilot#
This example provides a GitHub CI pipeline that automatically starts a SkyPilot job when a PR is merged to main branch and notifies a slack channel. It is useful for automatically trigger a training job when there is a new commit for config changes, and send notification for the training status and logs.
NOTE: This example is adapted from Metta AI’s GitHub actions pipeline: Metta-AI/metta
Why use SkyPilot with GitHub Actions?#
Pairing SkyPilot with GitHub Actions can automate routine experiments to run without manual input, improving iteration speed. Using SkyPilot with GitHub Actions can:
Customize Workflow Triggers: GitHub Actions provides a breadth of triggers to automate workflows, including:
Code pushes to a branch
Changes to specific files
On a schedule
Orchestrate and Monitor Jobs across Clouds: SkyPilot allows the CI task to run across region, clouds and kubernetes clusters, and provides a single pane of glass to monitor the CI jobs.
Enable Custom Notifications: Send a notification whenever a CI job runs, with a link to monitor the job status and logs.
Prerequisites#
The following steps are required to use the example GitHub action in your repository.
SkyPilot: Deploy a centralized API server#
Follow the instructions to deploy a centralized SkyPilot API server.
[Optional] Obtain a SkyPilot service account key if using SSO#
This section is required only for SkyPilot API Server deployment using OAuth.
To create a service account key:
Navigate to the Users page: On the main page of the SkyPilot dashboard, click “Users”.
Access Service Account Settings: Click “Service Accounts” located at the top of the page.
Note: If “Service Accounts” section does not exist on the dashboard, the API server is not using SSO. This section can be skipped.
Create New Service Account: Click “+ Create Service Account” button located at the top of the page and follow the instructions to create a service account token.
GitHub: Define repository secrets#
The example GitHub action relies on a few repository secrets. Follow this tutorial to add repository secrets.
In this example, create the following repository secrets:
SKYPILOT_API_URL: URL to the SkyPilot API server, in format ofhttp(s)://url-or-ip. If using basic auth, the URL should also include the credentials in format ofhttp(s)://username:password@url-or-ip.SKYPILOT_SERVICE_ACCOUNT_TOKEN: Only required if using OAuth. Service account token for GitHub actions user generated above.SLACK_BOT_TOKEN: Optional, create a Slack App and get a slack “App-Level Token” withconnections:writepermission to send a summary message. If not provided, a slack message is not sent after a job is queued.SLACK_CHANNEL_ID: Optional, Slack Channel ID to send a summary message. If not provided, a slack message is not sent after a job is queued.
Repository Structure#
The example repository has the following directory tree:
.
├── .git
│ ...
├── .github
│ ├── actions
│ │ ├── launch-skypilot-job
│ │ │ └── action.yaml
│ │ └── setup-environment
│ │ └── action.yaml
│ └── workflows
│ └── sky-job.yaml
└── tasks
└── train.yaml
The sky-job.yaml defines the actual GitHub workflow. This GitHub action is configured to run in two modes:
workflow_dispatch: Triggered manually via the “Actions” page of the GitHub repo.push: Triggered when a commit is pushed to specific branches (in this example,main).
on:
workflow_dispatch:
inputs:
task_yaml_path:
description: "Path to the task YAML file"
required: true
type: string
commit_to_run:
description: "The full commit hash to run the job against (required for manual runs)."
required: true
type: string
push:
branches: [main]
The workflow checks out the GitHub repo to a specified commit, generates a unique job name, and launches a custom action located at .github/actions/launch-skypilot-job/action.yaml.
The Launch SkyPilot Job action in turn uses a custom action located at .github/actions/setup-environment/action.yaml to install necessary dependencies (including skypilot), and launches a SkyPilot job.
Once the job is successfully launched, sky-job.yaml then parses out the job ID of the submitted job.
The submitted job can be queried either by using sky jobs queue or by visiting the Jobs page of the SkyPilot API dashboard.
A slack message is then sent to the configured slack channel. An example message is provided below:
Frequently Asked Questions#
What if my target branch is named something other than main (e.g. master)?#
You can modify sky-jobs.yaml to specify a different target branch:
on:
...
push:
- branches: [main]
+ branches: [master]
How do I limit / isolate the resources available to the workflow?#
You can specify a specific cloud, region or kubernetes cluster for the workflow to use in the task YAML. Alternatively, you can define a separate workspace the workflow can use, isolating the infrastructure the workflow has access to.
Included files#
sample_repo/.github/actions/launch-skypilot-job/action.yaml
name: "Launch SkyPilot Job"
description: "Sets up and launches a SkyPilot job with necessary configurations."
inputs:
task_yaml_path:
description: "Path to the task YAML file"
required: true
job_name:
description: "The unique name for the SkyPilot job."
required: true
commit_sha:
description: "Git commit SHA for status reporting"
required: true
skypilot_api_url:
description: "SkyPilot API URL"
required: true
skypilot_service_account_token:
description: "SkyPilot service account token"
required: true
outputs:
launch_output:
description: "The output from the SkyPilot jobs launch command"
value: ${{ steps.launch.outputs.launch_output }}
runs:
using: "composite"
steps:
- name: Setup Environment
uses: ./.github/actions/setup-environment
- name: Log into SkyPilot API server
shell: bash
run: |
if [ -z "${{ inputs.skypilot_service_account_token }}" ]; then
sky api login -e ${{ inputs.skypilot_api_url }}
else
sky api login -e ${{ inputs.skypilot_api_url }} --token ${{ inputs.skypilot_service_account_token }}
fi
- name: Launch SkyPilot job
id: launch
shell: bash
run: |
# Prepare launch arguments from GitHub Actions inputs
# and launch the job asynchronously via CLI.
# Save the output which contains the request ID for later use.
set -x # Enable command printing for debugging
LAUNCH_ARGS=()
if [ -n "${{ inputs.job_name }}" ]; then
LAUNCH_ARGS+=( "--name=${{ inputs.job_name }}" )
fi
# Capture the output
LAUNCH_OUTPUT=$(sky jobs launch ${{ inputs.task_yaml_path }} -y --async "${LAUNCH_ARGS[@]}" 2>&1)
echo "$LAUNCH_OUTPUT"
# Save to file for fallback
echo "$LAUNCH_OUTPUT" > /tmp/skypilot_launch_output.txt
# Set as GitHub Actions output
{
echo 'launch_output<<EOF'
echo "$LAUNCH_OUTPUT"
echo 'EOF'
} >> $GITHUB_OUTPUT
set +x # Disable command printing
sample_repo/.github/actions/setup-environment/action.yaml
name: "Setup Environment"
description: "Set up the environment for CI"
runs:
using: "composite"
steps:
- name: Install the latest version of uv
uses: astral-sh/setup-uv@v6
with:
version: "latest"
python-version: "3.10"
- name: Install dependencies
shell: bash
run: |
# Export environment variables
echo "VIRTUAL_ENV=$(pwd)/.venv" >> $GITHUB_ENV
echo "$(pwd)/.venv/bin" >> $GITHUB_PATH
echo "UV_NO_SYNC=1" >> $GITHUB_ENV
echo "UV_HTTP_TIMEOUT=120" >> $GITHUB_ENV
uv venv --seed
uv pip install skypilot
sample_repo/.github/workflows/sky-job.yaml
name: "Launch SkyPilot Job"
on:
workflow_dispatch:
inputs:
task_yaml_path:
description: "Path to the task YAML file"
required: true
type: string
commit_to_run:
description: "The full commit hash to run the job against (required for manual runs)."
required: true
type: string
push:
branches: [main]
env:
RUN_NAME_PREFIX: "gh-actions"
DEFAULT_TASK_YAML_PATH: "tasks/train.yaml"
jobs:
launch-skypilot-job:
runs-on: ubuntu-latest
steps:
- name: Checkout specific commit for dispatch
if: github.event_name == 'workflow_dispatch'
uses: actions/checkout@v4
with:
ref: ${{ github.event.inputs.commit_to_run }}
fetch-depth: 1
- name: Checkout full history for push
if: github.event_name == 'push'
uses: actions/checkout@v4
with:
fetch-depth: 2
- name: Determine PR Number for Push Events
id: pr_info
if: github.event_name == 'push'
shell: bash
run: |
# Expects commit message to be in format "commit message (#123)"
PR_FROM_COMMIT=$(git log -1 --pretty=format:"%s" | sed -n 's/.*(#\([0-9][0-9]*\)).*/\1/p')
echo "Extracted PR number from commit message (if any): $PR_FROM_COMMIT"
echo "### Extracted PR number from commit message (if any): $PR_FROM_COMMIT" >> $GITHUB_STEP_SUMMARY
echo "pr_number_for_name_generation=${PR_FROM_COMMIT}" >> $GITHUB_OUTPUT
- name: Generate Job Name
id: generate_job_name
shell: bash
run: |
# Generate a unique name for the SkyPilot job.
# The name has format of github-actions.<pr_number>.<commit_hash_short>.<timestamp>
# If PR number is not found (e.g. for manual dispatch), the name is in the format of
# github-actions.<branch_name>.<commit_hash_short>.<timestamp>
set -eo pipefail
TIMESTAMP=$(date -u +"%Y%m%d_%H%M%S")
# SHORT_COMMIT_HASH is derived from the actual checked-out HEAD.
# This HEAD is set by the conditional 'Checkout code' steps:
# - For 'push', it's the trigger commit.
# - For 'workflow_dispatch', it's github.event.inputs.commit_to_run.
SHORT_COMMIT_HASH=$(git rev-parse --short HEAD)
if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then
# Manual dispatch: run name is github.sky.<short_hash>.<timestamp>
if [ -z "${{ github.event.inputs.commit_to_run }}" ]; then
echo "Error: commit_to_run is required for workflow_dispatch and was not provided."
exit 1
fi
# Sanity check that checked-out commit matches the input
INPUT_SHORT_HASH=$(echo "${{ github.event.inputs.commit_to_run }}" | cut -c1-7)
if [ "$SHORT_COMMIT_HASH" != "$INPUT_SHORT_HASH" ]; then
echo "Warning: Checked out HEAD's short hash ($SHORT_COMMIT_HASH) does not match input commit_to_run's short hash ($INPUT_SHORT_HASH). Using checked out HEAD."
fi
RUN_NAME_BASE="${{ env.RUN_NAME_PREFIX }}"
FINAL_RUN_NAME="$RUN_NAME_BASE.$SHORT_COMMIT_HASH.$TIMESTAMP"
echo "Manual dispatch: Using commit ($SHORT_COMMIT_HASH). Run name: $FINAL_RUN_NAME"
elif [ "${{ github.event_name }}" == "push" ]; then
# Push event: run name includes PR number (if found) or branch name.
PR_NUMBER_FROM_COMMIT="${{ steps.pr_info.outputs.pr_number_for_name_generation || '' }}"
if [ -n "$PR_NUMBER_FROM_COMMIT" ]; then
RUN_NAME_BASE="${{ env.RUN_NAME_PREFIX }}.pr${PR_NUMBER_FROM_COMMIT}"
echo "Push event: Using PR #$PR_NUMBER_FROM_COMMIT in run name base: $RUN_NAME_BASE"
else
BRANCH_NAME_RAW=$(git rev-parse --abbrev-ref HEAD)
BRANCH_NAME_SANITIZED=$(echo "$BRANCH_NAME_RAW" | sed 's/[^a-zA-Z0-9_-]/-/g')
RUN_NAME_BASE="${{ env.RUN_NAME_PREFIX }}.${BRANCH_NAME_SANITIZED}"
echo "Push event: Using branch name '$BRANCH_NAME_SANITIZED' in run name base: $RUN_NAME_BASE (PR number not found in commit)"
fi
FINAL_RUN_NAME="$RUN_NAME_BASE.$SHORT_COMMIT_HASH.$TIMESTAMP"
echo "Push event: Run name: $FINAL_RUN_NAME"
else
echo "Error: Unhandled event type ${{ github.event_name }}"
exit 1
fi
echo "Generated job name for SkyPilot: $FINAL_RUN_NAME"
echo "job_name=$FINAL_RUN_NAME" >> $GITHUB_OUTPUT
echo "### Generated job name: $FINAL_RUN_NAME" >> $GITHUB_STEP_SUMMARY
- name: Launch SkyPilot Job via Custom Action
id: skylaunch
uses: ./.github/actions/launch-skypilot-job
with:
task_yaml_path: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.task_yaml_path || env.DEFAULT_TASK_YAML_PATH }}
job_name: ${{ steps.generate_job_name.outputs.job_name }}
commit_sha: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.commit_to_run || github.sha }}
skypilot_api_url: ${{ secrets.SKYPILOT_API_URL }}
skypilot_service_account_token: ${{ secrets.SKYPILOT_SERVICE_ACCOUNT_TOKEN }}
- name: Extract SkyPilot Request ID
id: extract_request_id
shell: bash
run: |
# Extract the request ID from the launch output
# Looking for pattern like "Submitted sky.jobs.launch request: c01f1273-4610-4949-8f1c-e24365ad0330"
REQUEST_ID=$(echo "${{ steps.skylaunch.outputs.launch_output }}" | grep -oP 'Submitted sky.jobs.launch request: \K[a-f0-9-]+' || true)
if [ -z "$REQUEST_ID" ]; then
# Try alternate extraction method from stored output
REQUEST_ID=$(cat /tmp/skypilot_launch_output.txt 2>/dev/null | grep -oP 'Submitted sky.jobs.launch request: \K[a-f0-9-]+' || true)
fi
if [ -n "$REQUEST_ID" ]; then
echo "Extracted request ID: $REQUEST_ID"
echo "request_id=$REQUEST_ID" >> $GITHUB_OUTPUT
echo "request_id_short=$(echo $REQUEST_ID | cut -c1-8)" >> $GITHUB_OUTPUT
echo '### SkyPilot Request ID $REQUEST_ID extracted' >> $GITHUB_STEP_SUMMARY
else
echo "Could not extract request ID from launch output"
echo "request_id=" >> $GITHUB_OUTPUT
echo "request_id_short=" >> $GITHUB_OUTPUT
echo '### SkyPilot Request ID not found in logs' >> $GITHUB_STEP_SUMMARY
fi
- name: Get SkyPilot Job ID
id: get_job_id
if: steps.extract_request_id.outputs.request_id != ''
shell: bash
run: |
set +e # Don't exit on error
# Wait a bit for the job to be submitted
sleep 5
# Get the logs and extract the job ID
echo "Getting logs for request: ${{ steps.extract_request_id.outputs.request_id_short }}"
LOG_OUTPUT=$(sky api logs ${{ steps.extract_request_id.outputs.request_id_short }} 2>&1 | tail -20)
# Extract job ID from pattern like "Job submitted, ID: 4700"
JOB_ID=$(echo "$LOG_OUTPUT" | grep -oP 'Job submitted, ID: \K\d+' || true)
if [ -n "$JOB_ID" ]; then
echo "Extracted SkyPilot Job ID: $JOB_ID"
echo "job_id=$JOB_ID" >> $GITHUB_OUTPUT
echo '### SkyPilot Job ID $JOB_ID launched' >> $GITHUB_STEP_SUMMARY
else
echo "Could not extract Job ID from logs"
echo "Log output was:"
echo "$LOG_OUTPUT"
echo "job_id=" >> $GITHUB_OUTPUT
echo '### SkyPilot Job ID not found in logs' >> $GITHUB_STEP_SUMMARY
fi
set -e
- name: Check if slack secrets are set
id: check_slack_secrets
shell: bash
run: |
if [ -z "${{ secrets.SLACK_BOT_TOKEN }}" ] || [ -z "${{ secrets.SLACK_CHANNEL_ID }}" ]; then
echo "slack_secrets_set=False" >> $GITHUB_OUTPUT
else
echo "slack_secrets_set=True" >> $GITHUB_OUTPUT
fi
- name: Generate Run Information for Slack
id: generate_slack_message
if: steps.check_slack_secrets.outputs.slack_secrets_set == 'True'
shell: bash
run: |
MESSAGE_TEXT="Skypilot job ${{ steps.get_job_id.outputs.job_id }} triggered by: ${{ github.event_name }}"
MESSAGE_BLOCK="$MESSAGE_TEXT\nSkyPilot Job Name: ${{ steps.generate_job_name.outputs.job_name }}"
if [ -n "${{ steps.get_job_id.outputs.job_id }}" ]; then
MESSAGE_BLOCK="$MESSAGE_BLOCK\n✅ SkyPilot Job ID: ${{ steps.get_job_id.outputs.job_id }}"
MESSAGE_BLOCK="$MESSAGE_BLOCK\n📝 View logs: sky jobs logs ${{ steps.get_job_id.outputs.job_id }}"
MESSAGE_BLOCK="$MESSAGE_BLOCK\n🔗 View job: ${{ secrets.SKYPILOT_API_URL }}/dashboard/jobs/${{ steps.get_job_id.outputs.job_id }}"
elif [ -n "${{ steps.extract_request_id.outputs.request_id_short }}" ]; then
MESSAGE_BLOCK="$MESSAGE_BLOCK\n⏳ Request ID: ${{ steps.extract_request_id.outputs.request_id_short }}"
MESSAGE_BLOCK="$MESSAGE_BLOCK\n📝 Get job ID: sky api logs ${{ steps.extract_request_id.outputs.request_id_short }}"
else
MESSAGE_BLOCK="$MESSAGE_BLOCK\nJob ID / Request ID not found"
fi
echo "message_text=$MESSAGE_TEXT" >> $GITHUB_OUTPUT
echo "message_block=$MESSAGE_BLOCK" >> $GITHUB_OUTPUT
echo $MESSAGE_BLOCK
echo "$MESSAGE_BLOCK" >> $GITHUB_STEP_SUMMARY
- name: Notify Slack channel
uses: slackapi/slack-github-action@v2.0.0
if: steps.check_slack_secrets.outputs.slack_secrets_set == 'True'
with:
method: chat.postMessage
token: ${{ secrets.SLACK_BOT_TOKEN }}
payload: |
{
"channel": "${{ secrets.SLACK_CHANNEL_ID }}",
"text": "${{ steps.generate_slack_message.outputs.message_text }}",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "${{ steps.generate_slack_message.outputs.message_block }}"
}
}
]
}
sample_repo/tasks/train.yaml
# This YAML is taken directly from the PyTorch quickstart example.
# https://docs.skypilot.co/en/latest/getting-started/tutorial.html
resources:
cpus: 4+
accelerators: L4:4
setup: |
git clone --depth 1 https://github.com/pytorch/examples || true
cd examples
git filter-branch --prune-empty --subdirectory-filter distributed/minGPT-ddp
pip install -r requirements.txt
run: |
cd examples/mingpt
export LOGLEVEL=INFO
echo "Starting minGPT-ddp training"
torchrun \
--nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \
main.py