Why I built smplmark.org

One of the application infrastructure products we built for smplkit is a simple HTTP job scheduler called Smpl Jobs. We wanted to promote how well it compares to other job schedulers by measuring how “on time” it fires. A benchmark. Who doesn’t love a benchmark, right?

But how?

It would need to be an even playing field: some publicly reachable endpoint we can hit from various cron job services to measure how “on time” it is. To keep it simple — and to stay under everybody’s “free tier” — we would configure each scheduler to execute at the top of the hour, every hour, and then measure how many milliseconds past the hour the request was actually received.

We knew we could aim the schedulers at something like webhook.site which would, indeed, show the date/time the requests were received, but it creates no lasting record.

Moreover, we wanted this to be a real, published benchmark. Something we can stand behind — and also invite others to publish their own.

What we wanted did not exist, so we built it.

What smplmark.org is

smplmark.org hosts benchmarks. Anyone can create an account and define a benchmark — the subjects, the metrics, the methodology, and the measurements. It’s API first so you can do everything via APIs or the UI. The site does the rest: live charts and tables, medians and percentiles, time-range filters, and embeddable images you can drop into an article and let update themselves. Every benchmark page shows its methodology and who published it. And it’s free!

Verifying publishers

Every benchmark is either published by a person or an organization. But if someone from Microsoft signs up, and wants to publish benchmarks as Microsoft, how would smplmark.org verify this person really is authorized to publish benchmarks on Microsoft’s behalf?

For that, we tapped into the usual trick: add a TXT record to your DNS that smplmark.org can then verify. If you can demonstrate control over microsoft.com, then you can publish benchmarks as Microsoft.

Otherwise, anyone can create an account and publish benchmarks under their own name (email address).

The first benchmark

The first benchmark on the site is the one I wanted for smplkit: job scheduler latency, which clocks how many milliseconds past the top of the hour each scheduler’s request actually lands. Because it’s API first, that meant that the API each scheduler was configured to hit is actually smplmark’s measurement API like this:

POST https://app.smplmark.org/api/v1/measurements

Header	Value
Content-Type	application/vnd.api+json
Authorization	Bearer {token}

{
  "data": {
    "type": "measurement",
    "attributes": {
      "run": "run-96135dcd",
      "subject": "smpl-jobs",
      "metrics": {}
    }
  }
}

Which job schedulers fire on time? We measured ten.

Open for business

If you have something worth measuring (your own product, your competitors’, the tools you depend on), smplmark.org will host it. Define the methodology, wire up the measurements, and put your name on it.