Tabby: Self-hosted AI Coding Assistant#
This post shows how to use SkyPilot to host an ai coding assistant with just one CLI command.
Background#
Tabby is a self-hosted AI coding assistant, offering an open-source and on-premises alternative to GitHub Copilot. It boasts several key features:
Self-contained, with no need for a DBMS or cloud service.
OpenAPI interface, easy to integrate with existing infrastructure (e.g Cloud IDE).
Supports consumer-grade GPUs.
SkyPilot is an open-source framework from UC Berkeley for seamlessly running machine learning on any cloud. With a simple CLI, users can easily launch many clusters and jobs, while substantially lowering their cloud bills. Currently, AWS, GCP, Azure, Lambda Cloud, IBM, Oracle Cloud Infrastructure (OCI), Cloudflare R2 and Samsung Cloud Platform (SCP) are supported. See docs to learn more.
Steps#
All YAML files used below live in the SkyPilot repo.
Install SkyPilot and check that cloud credentials exist:
# pip install skypilot pip install "skypilot[aws,gcp,azure,lambda]" # pick your clouds sky check
Get the Tabby SkyPilot config:
git clone https://github.com/skypilot-org/skypilot.git cd skypilot/llm/tabby
launch a Tabby cluster: You can select the model to be used by
--model
in thecommand
field of the docker-compose.cuda.yaml(GPU) or docker-compose.yaml(CPU).TabbyML/SantaCoder-1B
is used by default, for more model options, see the tabby docs to learn more.sky launch -c tabby tabby.yaml
After seeing
tabby server is ready, enjoy!
, you can use the following command to obtain the public IP:sky status --ip tabby 20.92.236.53
You can also use the
ssh
command directly to map the port to localhost.ssh -L 8080:localhost:8080 tabby
Install the Tabby extension in your IDE. For example, in VS Code, you can install the Tabby extension from the marketplace.
Configure your IDE to use
$(sky status --ip tabby):8080
as the Tabby endpoint.
Open a Python file and start coding! Tabby will start suggesting code snippets as you type.
Cleaning up#
When you are done, you can stop or tear down the cluster:
To stop the cluster, run
sky stop tabby # or pass your custom name if you used "-c <other name>"
You can restart a stopped cluster and relaunch the tabby (the
run
section in YAML) withsky launch tabby.yaml -c tabby --no-setup
Note the
--no-setup
flag: a stopped cluster preserves its disk contents so we can skip redoing the setup.To tear down the cluster (non-restartable), run
sky down tabby # or pass your custom name if you used "-c <other name>"
To see your clusters, run sky status
, which is a single pane of glass for all your clusters across regions/clouds.
To learn more about various SkyPilot commands, see Quickstart.