Tutorial

Submit a benchmark.

Anyone can publish on OpenChainBench. You write the harness, you host it, you keep your secrets. The project shares one Prometheus that scrapes your public /metrics endpoint. that is the only piece of common infrastructure.

Open an issue

Align on the metric.

Write the spec

One YAML file.

Build the harness

Expose /metrics.

Host it

Anywhere with HTTPS.

Wire the scrape

One block in prometheus.yml.

Open a PR

The page renders itself.

How it actually works

A federation, not a platform.

Every benchmark on this site is run by whoever wrote it. You host your harness wherever you like, expose /metrics over HTTPS, and the project's shared Prometheus scrapes that URL on a schedule. You keep your API keys, your wallet keys, your budget. Maintainers only see the metric values your harness chooses to publish.

The only piece of infrastructure shared by the project is one Prometheus instance. that is the URL every YAML spec points at. Adding a new harness is one extra scrape_configs block in infrastructure/prometheus/prometheus.yml. No new credentials, no new services, no privileged access to share.

For a concrete end-to-end example with a fictional contributor. the spec, the Go harness, deploying to Fly.io, opening the PR. see the walkthrough doc.

Realistic timeline

Day 0 · ~30 min. open issue, align on methodology with a maintainer.
Day 1-2 · ~2 h. write the spec + harness in your fork.
Day 2 · ~30 min. deploy your harness on Fly / Railway / your VPS, verify /metrics publicly reachable.
Day 2 · ~10 min. open the PR (spec + harness + scrape config).
Day 3 · ≤ 30 min. maintainer reviews, merges, reloads Prometheus. Site renders within 60 s.

~3-4 hours of focused work, spread across a few days.

Step 1. Open an issue

Use the Propose a benchmark template to describe what you want to measure, which providers, and where the harness will run. Maintainers respond with feedback before any code is written. Want to brainstorm first? Use Discussions → Ideas.

Step 2. Write the spec

Drop a file at benchmarks/<your-slug>.yml. It is the source of truth for the report. title, abstract, methodology, providers and the PromQL that fills in the numbers.

# benchmarks/wallet-portfolio-latency.yml

slug: wallet-portfolio-latency
number: "005"
title: Wallet Portfolio API. Read Latency
subtitle: How fast each wallet API returns a complete portfolio for a busy address.
category: Wallets
status: live
metric: Portfolio read
unit: ms

abstract: |
  We benchmark how long the major wallet APIs take to return a full
  portfolio (tokens, balances, USD values, NFTs) for a known busy address
  across 12 chains. The harness issues identical GETs from three regions
  and records p50, p90 and p99 wall-clock latency along with success rate.

methodology:
  - "Address set: 200 addresses with 50+ tokens across at least 5 chains."
  - "Cadence: 1 request / address / region every 60 s for 24 hours."
  - "Timeout: 5,000 ms. Failures excluded from latency aggregates."
  - "Regions: us-east-1, eu-west-1, ap-southeast-1."

findings: []

source: https://github.com/OpenChainBench/OpenChainBench/tree/main/harnesses/wallet-portfolio

prometheus:
  url: https://prom.openchainbench.xyz   # shared OpenChainBench Prometheus
  window: 24h

providers:
  - slug: provider-a
    name: Provider A
    tag: v3 endpoints
    secondary: { label: "Chains", value: "44" }
    queries:
      p50: histogram_quantile(0.5,  sum by (le) (rate(ocb_portfolio_ms_bucket{provider="provider-a"}[24h])))
      p90: histogram_quantile(0.9,  sum by (le) (rate(ocb_portfolio_ms_bucket{provider="provider-a"}[24h])))
      p99: histogram_quantile(0.99, sum by (le) (rate(ocb_portfolio_ms_bucket{provider="provider-a"}[24h])))
      success: sum(rate(ocb_portfolio_total{provider="provider-a", success="true"}[24h])) / sum(rate(ocb_portfolio_total{provider="provider-a"}[24h]))
      sample_size: sum(increase(ocb_portfolio_total{provider="provider-a"}[24h]))
      series: histogram_quantile(0.5, sum by (le) (rate(ocb_portfolio_ms_bucket{provider="provider-a"}[1h])))

That is the entire wire format. The Zod schema in src/lib/spec-schema.ts is the single source of truth; pnpm validate lints every spec in CI.

Step 3. Build the harness

The harness is a long-running data producer. Whatever fits the providers. Bun, Node, Python, Go, Rust. The contract is small:

·Run continuously, expose /metrics over HTTP on a documented port (e.g. :2112 or :9090).
·Use the metric and label names referenced by your YAML. That is how the site retrieves your numbers.
·Document inputs, regions, timeouts and the metrics port in harnesses/<slug>/README.md.
·Don't commit API keys. Read them from environment variables and document them in a .env.example.
·Don't bundle Prometheus, Grafana or Alertmanager. They live in infrastructure/ and are shared across every harness.

Step 4. Host it

OpenChainBench is a federation: each harness is hosted by whoever wrote it. Pick whatever fits. Railway, Fly, Cloud Run, a VPS, a home server with a static IP. The only requirement is that /metrics is reachable over HTTPS at a stable URL.

·You own the runtime, the secrets and the budget. Maintainers never see your API keys or wallet keys.
·If your harness needs API keys from the providers it benchmarks, you bring them. The data path treats every harness identically. Either Mobula-hosted or contributor-hosted.
·Plan for stability: if your URL changes you (or a maintainer) need to update the scrape config.

Step 5. Wire the scrape

Append a job to infrastructure/prometheus/prometheus.yml pointing at your public URL so the shared Prometheus picks up your harness:

- job_name: <your-slug>
  metrics_path: /metrics
  scheme: https
  static_configs:
    - targets:
        - your-harness.example.com   # or *.up.railway.app, *.fly.dev, …
      labels:
        benchmark: <your-slug>
        host: <you>                   # alice | acme-rpc | mobula …

Step 6. Dry-run + open the PR

Test the queries locally before opening the PR:

pnpm validate                                # schema lint
pnpm spec:dry-run wallet-portfolio-latency   # hit Prometheus, print resolved numbers
pnpm dev                                     # render the page locally

Open the PR. CI runs pnpm validate, pnpm typecheck and the build. Once merged, a maintainer redeploys the central Prometheus to apply the new scrape job and the site renders your benchmark on the next ISR cycle (within 60 seconds).

Reference

Ready?

Open a benchmark issue →Brainstorm in Discussions ↗GitHub repository ↗