CI Runnerscirunners.org
Methodology

Methodology: how we evaluate CI runners

Answer in brief

We list a product if it supplies the machines that run jobs for a CI system you already use. For each, we record documented facts (CI systems, operating systems, isolation, hardware, caching, pricing, deployment, open-source status) from the provider’s own documentation, and we date every entry. We do not rank providers against each other. We group them by kind and by the use cases their documented capabilities support, and list them alphabetically.

By the CI Runners editors · Updated

Inclusion criteria

A product is included if it supplies the machines (runners or agents) that execute jobs for a CI system the customer already uses: hosted by a vendor, launched in the customer’s own cloud account or on the customer’s hardware, or as open-source software to self-host. The first-party hosted runners of GitHub, GitLab, Buildkite and Azure Pipelines are included as baselines.

Complete CI platforms are out of scope unless they also sell runners for another CI system. This is a site about where jobs run, not about which pipeline syntax to write.

How facts are gathered

  • Primary sources only for product facts: official docs, pricing pages, changelogs and repositories. Each provider page lists the URLs used.
  • Speed and savings figures are vendor claims and are labelled as such. We have not benchmarked the runners.
  • When we cannot verify something, we say “not documented” rather than guessing. A “No” appears only where the provider says so or it follows from what the product is.
  • Prices are quoted as published on the verification date. The “Price” column is the provider’s headline rate: where one exists, the per-minute price of the smallest standard Linux x64 runner, usually 2 vCPU; otherwise the published unit, such as a licence fee or a price per parallel job.
  • Every entry carries a “verified” date. The whole reference was last reviewed on 2026-11-22.

Evaluation criteria

CriterionWhat we look at
CI systemsWhich systems the runner executes jobs for: GitHub Actions, GitLab CI, Buildkite, Azure Pipelines and others.
PlatformsOperating systems (Linux, macOS, Windows), architectures (x64, arm64) and GPU runners.
DeploymentVendor-hosted, in your own cloud account, on your own hardware; open-source licence.
IsolationThe documented boundary around a job (microVM, VM, container, bare metal) and whether every job gets a fresh environment.
Hardware and sizesDocumented CPU families, the range of runner sizes and the largest runner.
CachingDependency cache backend, Docker layer caching and persistent disks.
OperationsConcurrency and job limits, static IPs, SSH access for debugging, observability.
Pricing transparencyPublic per-minute rates, billing granularity, storage charges and free tier.
TrustCompliance attestations, regions and company headquarters.

How runners are organized

There is no single best runner for every pipeline, so we do not publish an overall ranking or a “#1 pick”. Instead:

  • Categories. Every runner belongs to one kind: managed runners for GitHub Actions, runners from your CI platform or cloud, runners in your own cloud or on your hardware, open-source autoscalers, and macOS runners. Each kind says when to choose it and what it costs you.
  • Use cases. For each use case we describe what the workload needs, using the criteria above, and list the runners whose documentation supports it, with a sentence on why each fits.
  • Order. Within every list, runners are grouped by kind and sorted alphabetically. Position carries no meaning.
  • Narrowing questions. The home page filters runners by documented facts (CI system, platform, deployment, isolation), computed directly from our data.

How the finder scores

The finder on the home page asks which CI system you use, what your jobs run on, where the runners must run and how isolated each job must be, plus optional must-haves (open source, Docker layer caching, GPUs, static IPs, SSH access, compliance, a free tier) and the cloud you already use.

  • A matrix, not a ranking. Every answer scores every runner from 0 to 3 using its documented facts: 3 means a documented fit, 2 capable with a caveat, 1 a stretch, and 0 rules the runner out for that answer. Each score carries the fact behind it, which is what the result cards show.
  • No hand-set scores. Every score is computed from the fields of the runner’s record, the same fields shown on its profile. There are no editorial weights and no hidden boosts.
  • Fit is the runner’s total divided by the maximum for the answers given: 90% and above is shown as a strong fit, 70% and above as good. Ties are listed alphabetically.
  • Nothing is hidden. Ruled-out runners are listed with the answer that excluded them, and no answer or preference favours a provider for commercial reasons; there are none.

Use it to build a shortlist of two or three, then run your own slowest workflow on each.

Limitations

  • We have not run independent benchmarks; performance claims come from vendors. A per-minute price says nothing about how many minutes a job takes on that hardware.
  • Pricing changes often. Always confirm on the provider’s pricing page.
  • Fit depends on your pipeline; use these pages to build a shortlist, then test.
  • This reference is general information, provided as is and without warranty. It is not professional advice. See the terms of use.

Updates

We aim to review the full reference monthly and update individual entries when providers ship notable changes. Every change is checked against the provider’s public documentation before an entry is updated.

Check before you rely on it. Products change quickly, and this page reflects public documentation as of November 22, 2026. Confirm anything that matters in the provider’s own documentation.

Cite this page

CI Runners. (2026). Methodology: how we evaluate CI runners. cirunners.org. Last reviewed November 22, 2026. https://cirunners.org/methodology/