node:bench

Node.js has had a built-in test runner for a while now. It does not have a built-in benchmark runner. PRs #65606 and #65631 hope to address that by adding an experimental node:bench module.

The API side is modeled closely after node:test:

import { bench, suite } from 'node:bench';
 
suite('URL', () => {
  const input = 'https://example.com/a?b=c';
 
  bench('construct', { samples: 30, params: { input: 'short' } }, (b) => {
    const operations = 10_000;
    let totalLength = 0;
 
    b.start();
    for (let i = 0; i < operations; i++) {
      totalLength += new URL(input).href.length;
    }
    b.end(operations);
 
    if (totalLength !== operations * input.length) {
      throw new Error('Unexpected URL result');
    }
  });
});

To run it:

node --bench benchmark.mjs

The --bench CLI arg takes explicit files or glob patterns, sorts them, and runs them in order.

node --bench benchmark.mjs
node --bench --bench-reporter=json 'benchmarks/**/*.js'

The default --bench-isolation=process runs each file in a fresh child process and emits one aggregate summary back to the runner.

--bench-isolation=none imports everything into one process. Faster startup, but module and heap state carries between files and user output shares destinations with reporters.

The remaining flags are:

• --bench-name-pattern
• --bench-samples
• --bench-warmup
• --bench-reporter
• --bench-reporter-destination

The available built-in reporters are spec and json.

Reporters can be repeated with matching destinations.

Events, not just output

It's also possible to run the benchmarks programattically. This intentionally works a lot like node:test's run().

Calling run() returns a BenchmarksStream. It's an object-mode stream.Readable with lifecycle records arriving as { type, data } emitted as named events.

import { bench, run } from 'node:bench';
 
const samples = 3;
bench('example', { samples }, (b) => {
  b.start();
  for (let n = 0; n < samples; n++) {
    doWork();
  }
  b.end(samples);
});
 
for await (const { type, data } of run()) {
  if (type === 'bench:complete' && data.error === undefined) {
    console.log(data.name, data.summary.mean);
  }
}

The record types are:

• bench:plan
• bench:start
• bench:sample
• bench:complete
• bench:diagnostic
• bench:summary.

The reporters process the stream an generate a friendly format. The json reporter emits newline delimited JSON with bigint values encoded as decimal strings and errors reduced to name, message, stack, code, cause, and errors.

Custom reporters are anything stream.compose() accepts that processes the records:

import { run } from 'node:bench';
import process from 'node:process';
 
async function* names(source) {
  for await (const { type, data } of source) {
    if (type === 'bench:complete') {
      yield `${data.name}\n`;
    }
  }
}
 
run().compose(names).pipe(process.stdout);

Three APIs exist specifically so that node:bench can be the basis for higher-level benchmark tools.

createRunner([options]) gives an isolated runner with its own declarations, hooks, filtering, and output. Unlike the module level functions, declaring a benchmark on an explicit runner does not schedule anything. A package can collect declarations now and start them later. Each runner runs once.

import { createRunner } from 'node:bench';
 
const runner = createRunner({ yieldBetweenSamples: false });
 
runner.bench('example', { samples: 100 }, (b) => {
  const operations = chooseOperationCount();
  b.start();
  runOperations(operations);
  const sample = b.end(operations);
 
  if (hasEnoughData(sample)) b.done();
});
 
for await (const record of runner.run()) {
  // Structured records.
}

runFile(path[, options]) runs exactly one benchmark module in a fresh child process and returns its stream. No globbing, no scheduling, no retry policy. Discovery, ordering, concurrency, and multi-file scheduling stay with the caller. A module load failure or abnormal exit produces an error diagnostic and a terminal bench:summary with success: false rather than erroring the stream, so a tool can keep going.

context.diagnostic(message[, options]) lets a benchmark emit info or warning records tied to the current benchmark, phase, and sample index.

meanCI in perf_hooks

One small additional Histogram addition. histogram.meanCI([options]) returns a two sided confidence interval for the mean using Student's t and the sample standard error.

const { createHistogram } = require('node:perf_hooks');
 
const h = createHistogram();
for (let i = 1; i <= 100; i++) h.record(i);
 
const { mean, lower, upper } = h.meanCI();
console.log(`mean=${mean}, 95% CI=[${lower}, ${upper}]`);

It rounds out the interval methods added in the earlier histogram PRs and it is what the spec reporter's 95% CI column is built on.

Node.js core's own benchmarks

The branch also ports a couple of core benchmarks to node:bench and adds parallel tools, benchmark/compare-node-bench.js and benchmark/scatter-node-bench.js as a proof point. These are really just intended to prove the point that node:bench works, but hopefully we'll be able to migrate the bespoke benchmarks over.

Where this goes

The module is marked Stability 1.0, Early Development, and the CLI flags are experimental.