Tabby: Self-hosted AI Coding Assistant#

This post shows how to use SkyPilot to host an ai coding assistant with just one CLI command.

Tabby

Background#

Tabby is a self-hosted AI coding assistant, offering an open-source and on-premises alternative to GitHub Copilot. It boasts several key features:

  • Self-contained, with no need for a DBMS or cloud service.

  • OpenAPI interface, easy to integrate with existing infrastructure (e.g Cloud IDE).

  • Supports consumer-grade GPUs.

SkyPilot is an open-source framework from UC Berkeley for seamlessly running machine learning on any cloud. With a simple CLI, users can easily launch many clusters and jobs, while substantially lowering their cloud bills. Currently, AWS, GCP, Azure, Lambda Cloud, IBM, Oracle Cloud Infrastructure (OCI), Cloudflare R2 and Samsung Cloud Platform (SCP) are supported. See docs to learn more.

Steps#

All YAML files used below live in the SkyPilot repo.

  1. Install SkyPilot and check that cloud credentials exist:

    # pip install skypilot
    pip install "skypilot[aws,gcp,azure,lambda]"  # pick your clouds
    sky check
    
  2. Get the Tabby SkyPilot config:

    git clone https://github.com/skypilot-org/skypilot.git
    cd skypilot/llm/tabby
    
  3. launch a Tabby cluster: You can select the model to be used by --model in the command field of the docker-compose.cuda.yaml(GPU) or docker-compose.yaml(CPU). TabbyML/SantaCoder-1B is used by default, for more model options, see the tabby docs to learn more.

     sky launch -c tabby tabby.yaml
    

Connect the extension to your Tabby server

  1. After seeing tabby server is ready, enjoy!, you can use the following command to obtain the public IP:

    sky status --ip tabby
    20.92.236.53
    

    You can also use the ssh command directly to map the port to localhost.

    ssh -L 8080:localhost:8080 tabby
    
  2. Install the Tabby extension in your IDE. For example, in VS Code, you can install the Tabby extension from the marketplace.

  3. Configure your IDE to use $(sky status --ip tabby):8080 as the Tabby endpoint.

Connect the extension to your Tabby server

  1. Open a Python file and start coding! Tabby will start suggesting code snippets as you type.

Tabby in VS Code

Cleaning up#

When you are done, you can stop or tear down the cluster:

  • To stop the cluster, run

    sky stop tabby  # or pass your custom name if you used "-c <other name>"
    

    You can restart a stopped cluster and relaunch the tabby (the run section in YAML) with

    sky launch tabby.yaml -c tabby --no-setup
    

    Note the --no-setup flag: a stopped cluster preserves its disk contents so we can skip redoing the setup.

  • To tear down the cluster (non-restartable), run

    sky down tabby  # or pass your custom name if you used "-c <other name>"
    

To see your clusters, run sky status, which is a single pane of glass for all your clusters across regions/clouds.

To learn more about various SkyPilot commands, see Quickstart.