Source: examples/github_actions
Example: GitHub Actions + SkyPilot#
This example provides a GitHub CI pipeline that automatically starts a SkyPilot job when a PR is merged to main branch and notifies a slack channel. It is useful for automatically trigger a training job when there is a new commit for config changes, and send notification for the training status and logs.
NOTE: This example is adapted from Metta AI’s GitHub actions pipeline: Metta-AI/metta
Prerequisites#
The following steps are required to use the example GitHub action in your repository.
SkyPilot: Deploy a centralized API server#
Follow the instructions to deploy a centralized SkyPilot API server.
[Optional] Obtain a SkyPilot service account key if using SSO#
This section is required only for SkyPilot API Server deployment using OAuth.
To create a service account key:
Navigate to the Users page: On the main page of the SkyPilot dashboard, click “Users”.
Access Service Account Settings: Click “Service Accounts” located at the top of the page.
Create New Service Account: Click “+ Create Service Account” button located at the top of the page and follow the instructions to create a service account token.
GitHub: Define repository secrets#
The example GitHub action relies on a few repository secrets. Follow this tutorial to add repository secrets.
In this example, create the following repository secrets:
SKYPILOT_API_URL: URL to the SkyPilot API server, in format ofhttp(s)://url-or-ip. If using basic auth, the URL should also include the credentials in format ofhttp(s)://username:password@url-or-ip.SKYPILOT_SERVICE_ACCOUNT_TOKEN: Only required if using OAuth. Service account token for GitHub actions user generated above.SLACK_BOT_TOKEN: Optional, create a Slack App and get a slack “App-Level Token” withconnections:writepermission to send a summary message. If not provided, a slack message is not sent after a job is queued.SLACK_CHANNEL_ID: Optional, Slack Channel ID to send a summary message. If not provided, a slack message is not sent after a job is queued.
Repository Structure#
The example repository has the following directory tree:
.
├── .git
│ ...
├── .github
│ ├── actions
│ │ ├── launch-skypilot-job
│ │ │ └── action.yaml
│ │ └── setup-environment
│ │ └── action.yaml
│ └── workflows
│ └── sky-job.yaml
└── tasks
└── train.yaml
The sky-job.yaml defines the actual GitHub workflow. This GitHub action is configured to run in two modes:
workflow_dispatch: Triggered manually via the “Actions” page of the GitHub repo.push: Triggered when a commit is pushed to specific branches (in this example,main).
on:
workflow_dispatch:
inputs:
task_yaml_path:
description: "Path to the task YAML file"
required: true
type: string
commit_to_run:
description: "The full commit hash to run the job against (required for manual runs)."
required: true
type: string
push:
branches: [main]
The workflow checks out the GitHub repo to a specified commit, generates a unique job name, and launches a custom action located at .github/actions/launch-skypilot-job/action.yaml.
The Launch SkyPilot Job action in turn uses a custom action located at .github/actions/setup-environment/action.yaml to install necessary dependencies (including skypilot), and launches a SkyPilot job.
Once the job is successfully launched, sky-job.yaml then parses out the job ID of the submitted job. A slack message is then sent to the configured slack channel. An example message is provided below:
Frequently Asked Questions#
What if my target branch is named something other than main (e.g. master)?#
You can modify sky-jobs.yaml to specify a different target branch:
on:
...
push:
- branches: [main]
+ branches: [master]
Included files#
sample_repo/.github/actions/launch-skypilot-job/action.yaml
name: "Launch SkyPilot Job"
description: "Sets up and launches a SkyPilot job with necessary configurations."
inputs:
task_yaml_path:
description: "Path to the task YAML file"
required: true
job_name:
description: "The unique name for the SkyPilot job."
required: true
commit_sha:
description: "Git commit SHA for status reporting"
required: true
skypilot_api_url:
description: "SkyPilot API URL"
required: true
skypilot_service_account_token:
description: "SkyPilot service account token"
required: true
outputs:
launch_output:
description: "The output from the SkyPilot jobs launch command"
value: ${{ steps.launch.outputs.launch_output }}
runs:
using: "composite"
steps:
- name: Setup Environment
uses: ./.github/actions/setup-environment
- name: Log into SkyPilot API server
shell: bash
run: |
if [ -z "${{ inputs.skypilot_service_account_token }}" ]; then
sky api login -e ${{ inputs.skypilot_api_url }}
else
sky api login -e ${{ inputs.skypilot_api_url }} --token ${{ inputs.skypilot_service_account_token }}
fi
- name: Launch SkyPilot job
id: launch
shell: bash
run: |
# Prepare launch arguments from GitHub Actions inputs
# and launch the job asynchronously via CLI.
# Save the output which contains the request ID for later use.
set -x # Enable command printing for debugging
LAUNCH_ARGS=()
if [ -n "${{ inputs.job_name }}" ]; then
LAUNCH_ARGS+=( "--name=${{ inputs.job_name }}" )
fi
# Capture the output
LAUNCH_OUTPUT=$(sky jobs launch ${{ inputs.task_yaml_path }} -y --async "${LAUNCH_ARGS[@]}" 2>&1)
echo "$LAUNCH_OUTPUT"
# Save to file for fallback
echo "$LAUNCH_OUTPUT" > /tmp/skypilot_launch_output.txt
# Set as GitHub Actions output
{
echo 'launch_output<<EOF'
echo "$LAUNCH_OUTPUT"
echo 'EOF'
} >> $GITHUB_OUTPUT
set +x # Disable command printing
sample_repo/.github/actions/setup-environment/action.yaml
name: "Setup Environment"
description: "Set up the environment for CI"
runs:
using: "composite"
steps:
- name: Install the latest version of uv
uses: astral-sh/setup-uv@v6
with:
version: "latest"
python-version: "3.10"
- name: Install dependencies
shell: bash
run: |
# Export environment variables
echo "VIRTUAL_ENV=$(pwd)/.venv" >> $GITHUB_ENV
echo "$(pwd)/.venv/bin" >> $GITHUB_PATH
echo "UV_NO_SYNC=1" >> $GITHUB_ENV
echo "UV_HTTP_TIMEOUT=120" >> $GITHUB_ENV
uv venv --seed
uv pip install skypilot
sample_repo/.github/workflows/sky-job.yaml
name: "Launch SkyPilot Job"
on:
workflow_dispatch:
inputs:
task_yaml_path:
description: "Path to the task YAML file"
required: true
type: string
commit_to_run:
description: "The full commit hash to run the job against (required for manual runs)."
required: true
type: string
push:
branches: [main]
env:
RUN_NAME_PREFIX: "gh-actions"
DEFAULT_TASK_YAML_PATH: "tasks/train.yaml"
jobs:
launch-skypilot-job:
runs-on: ubuntu-latest
steps:
- name: Checkout specific commit for dispatch
if: github.event_name == 'workflow_dispatch'
uses: actions/checkout@v4
with:
ref: ${{ github.event.inputs.commit_to_run }}
fetch-depth: 1
- name: Checkout full history for push
if: github.event_name == 'push'
uses: actions/checkout@v4
with:
fetch-depth: 2
- name: Determine PR Number for Push Events
id: pr_info
if: github.event_name == 'push'
shell: bash
run: |
# Expects commit message to be in format "commit message (#123)"
PR_FROM_COMMIT=$(git log -1 --pretty=format:"%s" | sed -n 's/.*(#\([0-9][0-9]*\)).*/\1/p')
echo "Extracted PR number from commit message (if any): $PR_FROM_COMMIT"
echo "### Extracted PR number from commit message (if any): $PR_FROM_COMMIT" >> $GITHUB_STEP_SUMMARY
echo "pr_number_for_name_generation=${PR_FROM_COMMIT}" >> $GITHUB_OUTPUT
- name: Generate Job Name
id: generate_job_name
shell: bash
run: |
# Generate a unique name for the SkyPilot job.
# The name has format of github-actions.<pr_number>.<commit_hash_short>.<timestamp>
# If PR number is not found (e.g. for manual dispatch), the name is in the format of
# github-actions.<branch_name>.<commit_hash_short>.<timestamp>
set -eo pipefail
TIMESTAMP=$(date -u +"%Y%m%d_%H%M%S")
# SHORT_COMMIT_HASH is derived from the actual checked-out HEAD.
# This HEAD is set by the conditional 'Checkout code' steps:
# - For 'push', it's the trigger commit.
# - For 'workflow_dispatch', it's github.event.inputs.commit_to_run.
SHORT_COMMIT_HASH=$(git rev-parse --short HEAD)
if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then
# Manual dispatch: run name is github.sky.<short_hash>.<timestamp>
if [ -z "${{ github.event.inputs.commit_to_run }}" ]; then
echo "Error: commit_to_run is required for workflow_dispatch and was not provided."
exit 1
fi
# Sanity check that checked-out commit matches the input
INPUT_SHORT_HASH=$(echo "${{ github.event.inputs.commit_to_run }}" | cut -c1-7)
if [ "$SHORT_COMMIT_HASH" != "$INPUT_SHORT_HASH" ]; then
echo "Warning: Checked out HEAD's short hash ($SHORT_COMMIT_HASH) does not match input commit_to_run's short hash ($INPUT_SHORT_HASH). Using checked out HEAD."
fi
RUN_NAME_BASE="${{ env.RUN_NAME_PREFIX }}"
FINAL_RUN_NAME="$RUN_NAME_BASE.$SHORT_COMMIT_HASH.$TIMESTAMP"
echo "Manual dispatch: Using commit ($SHORT_COMMIT_HASH). Run name: $FINAL_RUN_NAME"
elif [ "${{ github.event_name }}" == "push" ]; then
# Push event: run name includes PR number (if found) or branch name.
PR_NUMBER_FROM_COMMIT="${{ steps.pr_info.outputs.pr_number_for_name_generation || '' }}"
if [ -n "$PR_NUMBER_FROM_COMMIT" ]; then
RUN_NAME_BASE="${{ env.RUN_NAME_PREFIX }}.pr${PR_NUMBER_FROM_COMMIT}"
echo "Push event: Using PR #$PR_NUMBER_FROM_COMMIT in run name base: $RUN_NAME_BASE"
else
BRANCH_NAME_RAW=$(git rev-parse --abbrev-ref HEAD)
BRANCH_NAME_SANITIZED=$(echo "$BRANCH_NAME_RAW" | sed 's/[^a-zA-Z0-9_-]/-/g')
RUN_NAME_BASE="${{ env.RUN_NAME_PREFIX }}.${BRANCH_NAME_SANITIZED}"
echo "Push event: Using branch name '$BRANCH_NAME_SANITIZED' in run name base: $RUN_NAME_BASE (PR number not found in commit)"
fi
FINAL_RUN_NAME="$RUN_NAME_BASE.$SHORT_COMMIT_HASH.$TIMESTAMP"
echo "Push event: Run name: $FINAL_RUN_NAME"
else
echo "Error: Unhandled event type ${{ github.event_name }}"
exit 1
fi
echo "Generated job name for SkyPilot: $FINAL_RUN_NAME"
echo "job_name=$FINAL_RUN_NAME" >> $GITHUB_OUTPUT
echo "### Generated job name: $FINAL_RUN_NAME" >> $GITHUB_STEP_SUMMARY
- name: Launch SkyPilot Job via Custom Action
id: skylaunch
uses: ./.github/actions/launch-skypilot-job
with:
task_yaml_path: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.task_yaml_path || env.DEFAULT_TASK_YAML_PATH }}
job_name: ${{ steps.generate_job_name.outputs.job_name }}
commit_sha: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.commit_to_run || github.sha }}
skypilot_api_url: ${{ secrets.SKYPILOT_API_URL }}
skypilot_service_account_token: ${{ secrets.SKYPILOT_SERVICE_ACCOUNT_TOKEN }}
- name: Extract SkyPilot Request ID
id: extract_request_id
shell: bash
run: |
# Extract the request ID from the launch output
# Looking for pattern like "Submitted sky.jobs.launch request: c01f1273-4610-4949-8f1c-e24365ad0330"
REQUEST_ID=$(echo "${{ steps.skylaunch.outputs.launch_output }}" | grep -oP 'Submitted sky.jobs.launch request: \K[a-f0-9-]+' || true)
if [ -z "$REQUEST_ID" ]; then
# Try alternate extraction method from stored output
REQUEST_ID=$(cat /tmp/skypilot_launch_output.txt 2>/dev/null | grep -oP 'Submitted sky.jobs.launch request: \K[a-f0-9-]+' || true)
fi
if [ -n "$REQUEST_ID" ]; then
echo "Extracted request ID: $REQUEST_ID"
echo "request_id=$REQUEST_ID" >> $GITHUB_OUTPUT
echo "request_id_short=$(echo $REQUEST_ID | cut -c1-8)" >> $GITHUB_OUTPUT
echo '### SkyPilot Request ID $REQUEST_ID extracted' >> $GITHUB_STEP_SUMMARY
else
echo "Could not extract request ID from launch output"
echo "request_id=" >> $GITHUB_OUTPUT
echo "request_id_short=" >> $GITHUB_OUTPUT
echo '### SkyPilot Request ID not found in logs' >> $GITHUB_STEP_SUMMARY
fi
- name: Get SkyPilot Job ID
id: get_job_id
if: steps.extract_request_id.outputs.request_id != ''
shell: bash
run: |
set +e # Don't exit on error
# Wait a bit for the job to be submitted
sleep 5
# Get the logs and extract the job ID
echo "Getting logs for request: ${{ steps.extract_request_id.outputs.request_id_short }}"
LOG_OUTPUT=$(sky api logs ${{ steps.extract_request_id.outputs.request_id_short }} 2>&1 | tail -20)
# Extract job ID from pattern like "Job submitted, ID: 4700"
JOB_ID=$(echo "$LOG_OUTPUT" | grep -oP 'Job submitted, ID: \K\d+' || true)
if [ -n "$JOB_ID" ]; then
echo "Extracted SkyPilot Job ID: $JOB_ID"
echo "job_id=$JOB_ID" >> $GITHUB_OUTPUT
echo '### SkyPilot Job ID $JOB_ID launched' >> $GITHUB_STEP_SUMMARY
else
echo "Could not extract Job ID from logs"
echo "Log output was:"
echo "$LOG_OUTPUT"
echo "job_id=" >> $GITHUB_OUTPUT
echo '### SkyPilot Job ID not found in logs' >> $GITHUB_STEP_SUMMARY
fi
set -e
- name: Check if slack secrets are set
id: check_slack_secrets
shell: bash
run: |
if [ -z "${{ secrets.SLACK_BOT_TOKEN }}" ] || [ -z "${{ secrets.SLACK_CHANNEL_ID }}" ]; then
echo "slack_secrets_set=False" >> $GITHUB_OUTPUT
else
echo "slack_secrets_set=True" >> $GITHUB_OUTPUT
fi
- name: Generate Run Information for Slack
id: generate_slack_message
if: steps.check_slack_secrets.outputs.slack_secrets_set == 'True'
shell: bash
run: |
MESSAGE_TEXT="Skypilot job ${{ steps.get_job_id.outputs.job_id }} triggered by: ${{ github.event_name }}"
MESSAGE_BLOCK="$MESSAGE_TEXT\nSkyPilot Job Name: ${{ steps.generate_job_name.outputs.job_name }}"
if [ -n "${{ steps.get_job_id.outputs.job_id }}" ]; then
MESSAGE_BLOCK="$MESSAGE_BLOCK\n✅ SkyPilot Job ID: ${{ steps.get_job_id.outputs.job_id }}"
MESSAGE_BLOCK="$MESSAGE_BLOCK\n📝 View logs: sky jobs logs ${{ steps.get_job_id.outputs.job_id }}"
MESSAGE_BLOCK="$MESSAGE_BLOCK\n🔗 View job: ${{ secrets.SKYPILOT_API_URL }}/dashboard/jobs/${{ steps.get_job_id.outputs.job_id }}"
elif [ -n "${{ steps.extract_request_id.outputs.request_id_short }}" ]; then
MESSAGE_BLOCK="$MESSAGE_BLOCK\n⏳ Request ID: ${{ steps.extract_request_id.outputs.request_id_short }}"
MESSAGE_BLOCK="$MESSAGE_BLOCK\n📝 Get job ID: sky api logs ${{ steps.extract_request_id.outputs.request_id_short }}"
else
MESSAGE_BLOCK="$MESSAGE_BLOCK\nJob ID / Request ID not found"
fi
echo "message_text=$MESSAGE_TEXT" >> $GITHUB_OUTPUT
echo "message_block=$MESSAGE_BLOCK" >> $GITHUB_OUTPUT
echo $MESSAGE_BLOCK
echo "$MESSAGE_BLOCK" >> $GITHUB_STEP_SUMMARY
- name: Notify Slack channel
uses: slackapi/slack-github-action@v2.0.0
if: steps.check_slack_secrets.outputs.slack_secrets_set == 'True'
with:
method: chat.postMessage
token: ${{ secrets.SLACK_BOT_TOKEN }}
payload: |
{
"channel": "${{ secrets.SLACK_CHANNEL_ID }}",
"text": "${{ steps.generate_slack_message.outputs.message_text }}",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "${{ steps.generate_slack_message.outputs.message_block }}"
}
}
]
}
sample_repo/tasks/train.yaml
# This YAML is taken directly from the PyTorch quickstart example.
# https://docs.skypilot.co/en/latest/getting-started/tutorial.html
resources:
cpus: 4+
accelerators: L4:4
setup: |
git clone --depth 1 https://github.com/pytorch/examples || true
cd examples
git filter-branch --prune-empty --subdirectory-filter distributed/minGPT-ddp
pip install -r requirements.txt
run: |
cd examples/mingpt
export LOGLEVEL=INFO
echo "Starting minGPT-ddp training"
torchrun \
--nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \
main.py