Source: examples/cog

Example: Cog + SkyPilot

Use SkyPilot to self-host any Cog-packaged projects.

This is the “Blur” example from replicate/cog-examples

Serve using a single instance

sky launch -c cog ./sky.yaml

IP=$(sky status --ip cog)

curl http://$IP:5000/predictions -X POST \
  -H 'Content-Type: application/json' \
  -d '{"input": {"image": "https://blog.skypilot.co/introducing-sky-serve/images/sky-serve-thumbnail.png"}}' \
  | jq -r '.output | split(",")[1]' | base64 --decode > output.png

Scale up the deployment using SkyServe

We can use SkyServe (sky serve) to scale up the deployment to multiple instances, while enjoying load balancing, autoscaling, and other SkyServe features.

sky serve up -n cog ./sky.yaml

Notice the only change is from sky launch to sky serve up. The same YAML can be used without changes.

After the service is launched, access the deployment with the following:

ENDPOINT=$(sky serve status --endpoint cog)

curl http://$ENDPOINT/predictions -X POST \
  -H 'Content-Type: application/json' \
  -d '{"input": {"image": "https://blog.skypilot.co/introducing-sky-serve/images/sky-serve-thumbnail.png"}}' \
  | jq -r '.output | split(",")[1]' | base64 --decode > output.png

Included files

cog.yaml

build:
  python_version: "3.8"
  python_packages:
    - "pillow==8.2.0"
  system_packages:
    - "libpng-dev"
    - "libjpeg-dev"
predict: "predict.py:Predictor"

predict.py

import tempfile

import cog
from PIL import Image
from PIL import ImageFilter


class Predictor(cog.BasePredictor):

    def predict(
        self,
        image: cog.Path = cog.Input(description='Input image'),
        blur: float = cog.Input(description='Blur radius', default=5),
    ) -> cog.Path:
        if blur == 0:
            return input
        im = Image.open(str(image))
        im = im.filter(ImageFilter.BoxBlur(blur))
        out_path = cog.Path(tempfile.mkdtemp()) / 'out.png'
        im.save(str(out_path))
        return out_path

sky.yaml

# Example: Cog + SkyPilot.
#
# This is the "Blur" example from https://github.com/replicate/cog-examples/blob/main/blur/README.md
#
# Usage (1 serving instance):
#
#   sky launch -c cog ./sky.yaml
#
#   IP=$(sky status --ip cog)
#   curl http://$IP:5000/predictions -X POST \
#     -H 'Content-Type: application/json' \
#     -d '{"input": {"image": "https://blog.skypilot.co/introducing-sky-serve/images/sky-serve-thumbnail.png"}}' \
#   | jq -r '.output | split(",")[1]' | base64 --decode > output.png
#
# Usage (SkyServe): See README.md

service:
  readiness_probe:
    path: /predictions
    post_data:
      input: {"image": "https://blog.skypilot.co/introducing-sky-serve/images/sky-serve-thumbnail.png"}
  replicas: 2

resources:
  accelerators: {L4, T4, A10G}
  ports:
    - 5000

workdir: .

setup: |
  set -e
  sudo curl -o /usr/local/bin/cog -L "https://github.com/replicate/cog/releases/latest/download/cog_$(uname -s)_$(uname -m)"
  sudo chmod +x /usr/local/bin/cog

  cog build -t my-model

run: |
  docker run -d -p 5000:5000 --gpus all my-model