Increase runtime of programs

[bddicken]

Something that has been brought up repeatedly is the fact that this benchmark includes interpreter/jit/whatever startup times, which makes it "unfair" relative to purely compiled languages. While this is partly true, it is not unreasonable to have a benchmark that does include these startup times, as it is real cost in some scenarios.

One thing that could be done to both (A) help mitigate this problem and (B) uncover other performance differences is make the program run longer. This can be done by giving each larger input values or more data to process.

At the time of writing this, there are four programs. Here's how we could change this for each:

• loops: update the implementations to take a variable amount of iterations. Then, pass in a value that causes longer runtime (perhaps 10 billion instead of 1, for example)
• levenshtein - Add more strings and longer strings to the passed-in input
• fibonacci - Call with 43 or 44 instead of 40
• hello-world nothing!

[PEZ]

I don't think increasing runtime of programs will do enough to compensate for start times to carry the weight it adds. E.g. Julia has a start time of around 200ms on the M1 machine I run benchmarks on. It will continue to be heavily penalized even with 10X longer runtimes.

Some languages already take a lot of time running some of the tests. R takes 4 minutes on my M1 machine to do Levenshtein. And I run each program 15 times. So that's an hour just for R, just for Levenshtein. If it would take 10 hours I would run out of patience. 😄

I am at where I think that the only way to control the start-times issue, without involving separate benchmark tooling per language/platform, is to require from the implementations that they do the measurements and report the timings as part of their output. This is how they do it in Dave Plummer's Eratosthenes drag-race. That way we can also avoid the problem with that parsing input gets to be part of the tests. (Separate parsing benchmarks can be designed to measure such things.)

[igouy]

as it is real cost in some scenarios

And there are many scenarios.

And what is done here will not apply to all scenarios.

Decide what scenario you are interested in. Do the right thing for that scenario.

[igouy]

without involving separate benchmark tooling per language/platform, is to require from the implementations that they do the measurements and report the timings

If "the implementations" must "do the measurements and report the timings" then would that not be "separate benchmark tooling per language/platform" ?

[zierf]

If "the implementations" must "do the measurements and report the timings" then would that not be "separate benchmark tooling per language/platform" ?

One simple thing would be taking the current time directly before and after invoking the algorithm in question.
End-Start time in e.g. milliseconds give you an insight how long the code ran you are interested in.
This method is one of the simplest forms and doesn't require any benchmark tooling per language.

The advantage is not measuring VM startup times, parsing cli arguments or pre-calculated variables (like random numbers with very different implementations) in some benchmarks. Some languages cleanup their memory before they end, in others you could "cheat" and just exit without cleaning up, because the OS will clean it anyway. But these programs reduce runtime by omitting work that some languages can't omit by design.

If you are interested in different parts, just enclose them with some timings too. The current benchmark script can still report the runtime overall, so you can see the whole time from starting a program until it's finished.

[igouy]

doesn't require any benchmark tooling per language

It seems to depend on the different ways "taking the current time" was implemented in each language implementation ?

End-Start time

Isn't "how long the code ran".

It's how long the code plus unrelated processes ran.

[zierf]

It seems to depend on the different ways "taking the current time" was implemented in each language implementation ?

We could also not measure anything relevant while we assume that not enough languages ​​can reliably produce a time indication. But what's the point of a benchmark for a specific algorithm if you measure something like 80% startup time. Increasing the workload to obtain more meaningful readings in fast languages ​​could also mean horrendous execution times for more niche languages or shells.

Isn't "how long the code ran".

It's how long the code plus unrelated processes ran.

Start and end here was the time before the algorithm startet and after it ended, not the runtime of the whole program.
You can define multiple starts and ends for everything you want to measure.
The current benchmark harness can still measure the whole time, from start to end of the program.

---

I also like the Issue with the proposal to give the benchmarks a timeframe and observe how many calculations they ran. This would work across all languages as long they count a successful execution for a single step. It also gives an upper bound for slow languages. So no need to wait 4 minutes for R (although it's a bit unfair due to missing vectorization for R).

Dave Plummer's drag-race was already a good suggestion. It's one way to produce comparable benchmarks. At least it should be considered for new benchmarks.

[igouy]

But what's the point of a benchmark for a specific algorithm if you measure something like 80% startup time.

If the scenario is boot the computer + start calculation then it would be important to know that 80% was whatever you mean by startup time.

Start and end here was the time before the algorithm startet and after it ended

Why might such measurements not be the same if re-run in an hour or a day or a week?

[PEZ]

I fleshed out the idea with the-programs-also-do-benching a bit here:

• Start times: Make the programs do the benching too #341 (comment)

[zierf]

Some languages like Java (Java Virtual Machine), JavaScript (V8/JavaScriptCore in NodeJS/Deno/Bun), Python, Go etc. have to start a VM/Runtime/Engine first. Their startup time doesn't matter that much if you are interested in a long running algorithm. If you want to deploy something on a "serverless" cloud environment where startup times are most important, then of course you’re interested in startup times.

But measuring one specific codepath doesn't exclude measuring others.

Here a very simple pseudo code example measuring two specific things of a program.

const start_params = time_ms();
const param_list = parse_input_params(env.args);
const end_params = time_ms();

const start_algorithm = time_ms();
const calculated_result = my_algorithm(4711);
const end_algorithm = time_ms();

print("Parameter Parsing took {end_params-start_params} ms.");
print("Calculating Result took {end_algorithm-start_algorithm} ms.");

Then let it run and measure itself plus it's complete execution time from the outside.

$> time my_program.out
Parameter Parsing took 12ms.
Calculating Result took 474ms.

my_program.out 0,48s user 0,00s system 99% cpu 0,486 total

That gives you 3 interesting timings about your program in a very simple setup.

From this point you can replace the time wrapper anytime with Hyperfine or any other benchmarking tool, like the run.sh in this repo does.

This is just one of many possible solutions, but it should show you at least one simple basic concept.

---

Another solution is to let it run for a specific time and let the algorithm calculate something big or infinite. Increase a counter for each successful iteration and print it as result at the end of the program. This way you can compare the outputs without any need for existing time functions across all languages and a simple addition is even less computing intensive (although retrieving a time is basically for free too).

---

It also looks like PEZ has already added another comment about how programs can benchmark themselves in Issue #341.

[igouy]

But measuring one specific codepath doesn't exclude measuring others.

The point is the need to be specific.

If the scenario is boot the computer + start calculation then say-so and use a measurement framework that supports that scenario.

For other scenarios, say what the scenario is and use a measurement framework that supports that different scenario.

startup time

[zierf]

I don't know how else to describe it.
Currently you only talk about imaginary scenarios and measurement frameworks.

The point is the need to be specific.

I gave you an example measuring two specific aspects of a program.

With hyperfine in addition you also get the runtime of the whole program.

If the scenario is boot the computer + start calculation then say-so and use a measurement framework that supports that scenario.

For other scenarios, say what the scenario is and use a measurement framework that supports that different scenario.

Then just suggest a test framework that supports whatever you want to test for all languages.
I don't even want to rule out that such a program exists.
It shouldn't be too complex, but if you know such a program, please just suggest it and someone can look into it.

The point of self-measuring programs is that they are independent of test frameworks, which often only exist for certain languages. I think the references to Dave Plummer's Drag Race were a good example of what was meant.

---

Since this issue was more about whether the runtime of the programs should be increased by more workload, I think it makes more sense to continue everything else in Issue #341. After all, that handles the question about whether and what the programs themselves should measure.