YAML Configuration

The resources to benchmark can be configured in the SkyPilot YAML interface. Below we provide an example:

# Only shows `resources` as other fields do not change.
resources:
    cloud: gcp  # Works as a default value for `cloud`.

    # Added only for SkyPilot Benchmark.
    candidates:
    - {accelerators: A100}
    - {accelerators: V100, instance_type: n1-highmem-16}
    - {accelerators: T4, cloud: aws}  # Overrides `cloud` to `aws`.

For SkyPilot Benchmark, candidates is newly added under the resources field. candidates is the list of dictionaries that configure the resources to benchmark. Any subfield of resources (accelerators, instance_type, etc.) can be re-defined in the dictionaries. Subfields defined outside candidates (e.g. cloud in this example) are used as default values and are overriden by those defined in the dictionaries. Thus, the above example can be interpreted as follows:

# Configuration of the first candidate.
resources:
    cloud: gcp
    accelerators: A100

# Configuration of the second candidate.
resources:
    cloud: gcp
    accelerators: V100
    instance_type: n1-highmem-16

# Configuration of the third candidate.
resources:
    cloud: aws
    accelerators: T4

Note

Currently, SkyPilot Benchmark does not support on-prem jobs and managed spot jobs. While you can set use_spot: True to benchmark spot VMs, automatic recovery will not be provided when preemption occurs.