Why I built smplmark.org

One of the application infrastructure products we built for smplkit is an HTTP job scheduler called Smpl Jobs. We wanted to promote how well it compares to other job schedulers by measuring how “on time” it fires. A benchmark. Who doesn’t love a benchmark, right?

But how?

It would need to be an even playing field: some publicly reachable endpoint we can hit from various cron job services to measure how “on time” they are. To keep it simple (and to stay under everybody’s “free tier”), we would configure each scheduler to execute at the top of the hour, every hour, and then measure how many milliseconds past the hour the request was actually received.

We knew we could aim the schedulers at something like webhook.site, which would, indeed, show the date/time the requests were received, but it creates no lasting record.

Moreover, we wanted this to be a real, published benchmark. Something we can stand behind - and a place for others to publish their own.

What we wanted did not exist, so we built it.

What smplmark.org is

smplmark.org hosts benchmarks. Anyone can create an account and define a benchmark - the subjects, the metrics, the methodology, and the measurements. It’s API first, so you can do everything via APIs or the UI. The site does the rest: live charts and tables, medians and percentiles, time-range filters, and embeddable images you can share. Every benchmark page shows its methodology and who published it. And it’s free!

Verifying publishers

Every benchmark is either published by a person or an organization. But if someone from, say, Microsoft signs up and wants to publish benchmarks as Microsoft, how would smplmark.org verify this person really is authorized to publish benchmarks on Microsoft’s behalf?

For that, we tapped into the usual trick: add a TXT record to your DNS that smplmark.org can then verify. If you can demonstrate control over microsoft.com, then you can publish benchmarks as Microsoft.

Otherwise, anyone can create an account and publish benchmarks under their own name (email address).

The first benchmark

The first benchmark on the site is the one this whole thing was built for: measuring how “on time” Smpl Jobs is - job scheduler latency. Because it’s API first, that meant that the API each scheduler was configured to hit is actually smplmark’s measurement API with a request like this:

POST https://app.smplmark.org/api/v1/measurements
Content-Type: application/vnd.api+json
Authorization: Bearer {token}

…and a body like this:

{
  "data": {
    "type": "measurement",
    "attributes": {
      "run": "run-96135dcd",
      "subject": "smpl-jobs",
      "metrics": {}
    }
  }
}

So each scheduler is literally creating its own measurements - even as I write this. The payload POSTed identifies what “run” the measurement belongs to (and runs belong to benchmarks), and the ID of the subject the measurement is for. In my case, there is no other “metric”, per se. The existence of the measurement is the metric. The benchmark does the rest by defining a “derived” metric (a.k.a. “formula”) based on JsonLogic that calculates the number of milliseconds past the hour based on the timestamp of the measurement like this:

{
  "%": [
    {
      "var": "created_at"
    },
    3600000
  ]
}

And that’s it! Every hour, on the hour, we have 10 schedulers creating measurements by just POSTing to https://app.smplmark.org/api/v1/measurements. smplmark.org does the rest.

Which job schedulers fire on time? We tested ten.

Open for business

If you have something worth measuring (your own product, your competitors’, the tools you depend on), smplmark.org will host it. Define the methodology, upload the measurements, and put your name on it.