Start times: Make the programs do the benching too

[PEZ]

As I mentioned in #336 (comment) I think we can take inspiration from Dave Plummers Eratosthenes drag-race benchmark in order to control when we want start-times to matter or not.

Let's use levenshtein as an example. What we are insterested in is how long takes for the program, given a bunch of strings, to figure out the minimum Levenshtein distance. So then we should only be measuring this. Not the start time of the executable. Not the time it takes to parse the input arguments and prepare for figuring out the minimum distance.

In Dave Plummer's drag-race each program is given the same amount of time, and they measure how many times the program can finish the task. This controls the length of the whole benchmark run much better and we can allow slow programs to participate without worrying that they will take forever. We can make it so that even R finishes one run of Levenshtein in, say, 15s . Then we give each program 30s to run, and allow them to use the same time to warm up their JIT engines. Even with 100 languages, slow or fast, it would take a maximum of 2 hours to run a benchmark. Say we have 5 benchmarks and 100 languages, it would take 10 hours.

I think we could have some easy-to-parse config file that specifies the benchmark. How long benchmark time that is allowed, the name of the input file, maybe something more. Then each program has some infrastructure to read and parse the config file, read and parse the input, run the measured function, measure how many times it runs, measure how long time it took, output the result of the function as well as the measurements.

It would be some considerable work to update the contributions, but since it's crowd sourced work, I think we can do it. I can do it for Java, C, and Clojure for reference and learning before we go ahead updating the rest. (My C is not up-to-date, nor up-to-speed, but I think I can do it, haha.)

[igouy]

So then we should only be measuring this.

Without the other stuff the programs won't figure out the minimum Levenshtein distance.

[PEZ]

So the idea is some little “tooling” around the call to the thing being benchmarked. (As opposed to using tooling libraries or programs built for each platform.) It's going in-process to the extreme that hyperfine goes out-of-process.

For Clojure it may look like so:

(defn benchmark [run-ms f]
  (let [t-start (System/currentTimeMillis)
        t-stop (+ t-start (* run-ms))
        rs (reduce (fn [results i]
                     (let [result (f) ; <- call the benchmarked function
                           t (System/currentTimeMillis)]
                       (if (< t t-stop)
                         (conj results [i t result])
                         (reduced (conj results [i t result])))))
                   []
                   (range))
        [i t result] (last rs)
        elapsed-time (- t t-start)
        mean-time (/ elapsed-time i)]
    [result (+ i 1) mean-time]))

Where f is the function being benchmarked.

For fibonacci it would be used from the -main function like so:

(defn -main [& args]
  (let [run-ms (parse-long (first args))
        n (parse-long (second args))
        _ (benchmark run-ms #(fib-sum n))
        [result count mean-time] (benchmark run-ms #(fib-sum n))]
    (println (str result ";" count ";" (double mean-time)))))

Which first runs the benchmark without collecting result, allowing the JIT to warm up. Called with 10s run time and asked to calculate the sum up to fib(36):

$ code 10000 36
24157816;180;55.96648044692737

Which reports that the sum was calculated to 24157816, that it ran 180 times, and that the mean time was 55.97ms. The last run will always be “just beyond” 10s, how “just beyond“ depends on how fast the language is. For a really slow language it may be that it only runs 1 time and it takes 25s or whatever, making the mean time 25,000ms.

The code measuring the time and keeping tally will take some little time, but it should be insignificant compared to the function being benchmarked. (For this example I get the same mean times without the benchmark harness.)

I'll press on and implement this for at least C and Java as well, and we can play with it and see if we think we should go this way.

[PEZ]

Here's a PR with reference implementation of the tooling mentioned in my previous comment, and all benchmarks for Clojure, Java, and C:

• Programs do the benching #365

[PEZ]

We just merged:

• In-process benchmarking #370

Meaning we have started the move over to start-time-free benchmarking. Have a look at the first results visualized here: https://pez.github.io/languages-visualizations/#https://gist.github.com/PEZ/411e2da1af3bbe21c4ad1d626451ec1d

Closing this and we can open new, more targeted issues for the coming work.