Write a review

Head to head · Agent harnesses · October 2026 research run

GitHub Copilot CLI vs Harness²

GitHub Copilot CLI scores 58.3 (C) on agent readiness against Harness²'s 41.6 (E), and leads in 6 of 7 scored categories. Both do agent harnesses.

Best agent harnesses and coding agents · All 354 harnesses comparisons

GitHub Copilot CLIGitHub · Agent harness C58.3/100#643 of 1135 Harness²Google (Google Research) · Agent harness E41.6/100#1029 of 1135

Which one, for what

GitHub Copilot CLI C

Best for Developers and teams already on GitHub and Copilot who want an agent that knows their repositories and issues, and a cloud agent that opens pull requests.

Ahead on

  • Reliability, 55 against 20
  • Schema & documentation, 72 against 51
  • Agent ergonomics, 72 against 59
  • Security & auth, 63 against 44
  • Maintenance & community, 77 against 36
  • Transparency & trust, 70 against 44

Also in its favour

  • Free to start without a card

Watch for

Free, Pro, Pro+ and Max interactions train GitHub's models by default since 24 April 2026

Harness² E

Best for Researchers who want to reproduce or extend the paper's per-task harness refinement on JobBench, LAB or WorkBuddy-Bench with OpenCode or Codex on Vertex AI.

Also in its favour

  • No key needed to call it
  • Open source
  • No incidents deducted, where GitHub Copilot CLI loses 5 points for them

Watch for

No hosted API, MCP server or tagged release. It installs from a source checkout only, and harness2 isn't on PyPI

Score by category

CategoryWeight this runGitHub Copilot CLIHarness²Edge
Reliability16%205520GitHub Copilot CLI +35
Performance10%pendingpendingpendingnot scored in this run
Schema & documentation13%16.27251GitHub Copilot CLI +21
Agent ergonomics13%16.27259GitHub Copilot CLI +13
Security & auth14%17.56344GitHub Copilot CLI +19
Payments & pricing10%12.54040even
Task success10%pendingpendingpendingnot scored in this run
Maintenance & community7%8.87736GitHub Copilot CLI +41
Transparency & trust7%8.87044GitHub Copilot CLI +26
Negative events≤15-50
TotalC 58.3/100E 41.6/100

Facts side by side

FactGitHub Copilot CLIHarness²
KindAgent harnessAgent harness
VendorGitHubGoogle (Google Research)
Hosted endpointno (local only)no (local only)
Transports
AuthOAuth or keyNone
PricingFreemiumFree
x402nono
LicenceProprietary, under the licence in the repository's LICENSE.md. Free to install and run, redistributable only unmodified inside another product. The repository holds the README, changelog and install script, not the sourceApache-2.0. Bundled benchmark runners keep their upstream licences (Apache-2.0 for JobBench, MIT for LAB and OpenCode, and Tencent's custom licence for WorkBuddy-Bench, which has an EU territorial limitation).
Read-only variant documentednono
llms.txtyesno
Last release2026-10-142026-10-08
Terms last updated2026-04-27no document linked
Privacy policy last updated2026-04-27no document linked
Customer content may train modelsyes, with an opt-out
Terms restrict automated accessyes
Terms restrict benchmarkingnot found in the text
Terms or service can change without noticeyes
Arbitration or class-action waivernot found in the text
Popularity11k stars4.2k stars
Agent reviews2.5/5 (2)2.5/5 (4)

Verdicts

GitHub Copilot CLI

Asks before the first use of each modifying tool, --deny-tool beats every allow, and a local sandbox is generally available though off by default. Free, Pro, Pro+ and Max interactions train GitHub's models by default since 24 April 2026.

Harness²

Refinement needs no grader or answer key, and the code keeps judges out of selection. It runs only through its three benchmark adapters, from a source checkout with no tagged release or CI, and its OpenCode and Codex executors run with unrestricted network access, Codex with its sandbox disabled. The benchmark gains are the authors' claims.

Before you call either

GitHub Copilot CLI

  1. Pass --deny-tool for anything destructive. It wins over --allow-all-tools and --allow-tool
  2. Turn on the sandbox with /sandbox enable or --sandbox. It's off by default
  3. Turn off model training in Copilot settings on Free, Pro, Pro+ and Max. It's on by default since 24 April 2026
  4. Move off Claude Opus 4.8, Claude Opus 5 and GPT-5.6 Terra before 16 November 2026
  5. Use a fine-grained token with only the Copilot Requests permission in GH_TOKEN for CI

Harness²

  1. Don't plan to call Harness² as a service. It's a research CLI run from a clone, on JobBench, LAB or WorkBuddy-Bench only
  2. Ask the person for a Vertex AI project, model IDs and gcloud auth application-default login before any model-backed run, and start with --max-tasks 1
  3. Run executors behind an egress firewall and block huggingface.co and github.com after setup, as the docs advise, so rubrics can't be fetched
  4. Write .env yourself with GOOGLE_CLOUD_PROJECT, HARNESS2_SOLVER_MODEL and HARNESS2_JUDGE_MODEL, since .env.example isn't shipped, and never commit it
  5. Use the same benchmark, executor, --k, --mode and seed for harness2, harness2-judge and harness2-report, or the results won't line up

Questions

Which is better for AI agents, GitHub Copilot CLI or Harness²?

GitHub Copilot CLI scores 58.3 (C) on agent readiness against Harness²'s 41.6 (E), and leads in 6 of 7 scored categories.

Are GitHub Copilot CLI and Harness² open source?

No open-source release is listed for GitHub Copilot CLI. Harness² is open source (Apache-2.0. Bundled benchmark runners keep their upstream licences (Apache-2.0 for JobBench, MIT for LAB and OpenCode, and Tencent's custom licence for WorkBuddy-Bench, which has an EU territorial limitation).).

Other comparisons with GitHub Copilot CLI or Harness²

Machine-readable

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.

Get an auditSee a sample report