Source: llm/localgpt

Chat with your documents using LocalGPT and SkyPilot#

Logo

LocalGPT allows you to chat with your documents (txt, pdf, csv, and xlsx), ask questions and summarize content. The models run on your hardware and your data remains 100% private. SkyPilot can run localGPT on any cloud (AWS, Azure, GCP, Lambda Cloud, IBM, Samsung, OCI) with a single command, taking care of minimizing cost and finding availability.

Prerequisites#

Install SkyPilot and check your setup of cloud credentials:

pip install git+https://github.com/skypilot-org/skypilot.git
sky check

See docs for more.

Once you are done, we will use SkyPilot YAML for localGPT to define our task and run it.

Launching localGPT on your cloud with SkyPilot#

  1. Run localGPT on your cloud.

Option 1: Run with CLI

Use sky launch to run localGPT on your cloud.

sky launch -c localgpt localgpt.yaml

Once you see INFO:werkzeug:Press CTRL+C to quit, you can safely Ctrl+C from the sky launch command.

Option 2: Run with SDK

Use launch_localgpt.py to run localGPT on your cloud.

python3 launch_localgpt.py

SkyPilot will show the estimated cost and chosen cloud before provisioning. For reference, running on T4 instances on AWS would cost about $0.53 per hour.

  1. Run ssh -L 5111:localhost:5111 localgpt in a new terminal window to forward the port 5111 to your local machine. Keep this terminal running.

  2. Open http://localhost:5111 in your browser. Click on upload file to upload a document. Once the document has been ingested, you can chat with it, ask questions, and summarize it. For example, in the gif below, we use the SkyPilot NSDI 2023 paper to ask questions about how SkyPilot works.

LocalGPT demo

  1. Once you are done, you can terminate the instance with sky down localgpt.

Optional: To make the demo publicly accessible, configure your cloud to open port 5111 for the VPC used by your instance (see instructions for AWS and GCP). Then, you can access the demo at http://<your-instance-public-ip>:5111. You can get the IP for your instance by running:

host_name="localgpt" && sky status $host_name --endpoint 5111

Included files#

launch_localgpt.py

import sky

cluster_name = 'localgpt'
task = sky.Task.from_yaml('localgpt.yaml')
request_id = sky.launch(task,
                        cluster_name=cluster_name,
                        _need_confirmation=True)
job_id, _ = sky.stream_and_get(request_id)

logs = sky.tail_logs(cluster_name, job_id, follow=True, preload_content=False)
found_logline = False
for log in logs:
    print(log, end='')
    if "INFO:werkzeug:" in log and "Press CTRL+C to quit" in log:
        found_logline = True
        break
logs.close()

if found_logline:
    print("Cluster launched. In a new terminal run:\n\n"
          f"ssh -L 5111:localhost:5111 {cluster_name}\n\n"
          "while keeping this terminal running, open \n\n"
          "http://localhost:5111\n\n"
          "in your browser to start using localGPT")
else:
    print("Cluster failed to launch")

localgpt.yaml

# Run localGPT on your documents on any cloud using SkyPilot
# Usage:
#   sky launch -c localgpt localgpt.yaml
#
#   * Wait to see `INFO:werkzeug:Press CTRL+C to quit` before proceeding
#   * In a new terminal, run `ssh -L 5111:localhost:5111 localgpt`
#   * Open http://localhost:5111 in your browser

resources:
  accelerators: A100:1

setup: |
  conda activate py311
  if [ $? -eq 0 ]; then
    echo "conda env exists"
  else
    conda create -y -n py311 python=3.11
    conda activate py311
    git clone https://github.com/PromtEngineer/localGPT.git
    cd localGPT/
    git checkout 5ab5e1921adb45a2df8d61f189a3fb4e9b401e58
    pip install -r requirements.txt
    # bitsandbytes<0.42.0 fails on GCP, see issue:
    # https://github.com/TimDettmers/bitsandbytes/issues/620
    pip install -U bitsandbytes>=0.42.0
  fi

run: |
  conda activate py311
  cd localGPT/
  python run_localGPT_API.py & python localGPTUI/localGPTUI.py --port 5111 --host 0.0.0.0