Programs do the benching

[PEZ]

I have:

• Read the project README, including the benchmark descriptions
• Read the PR template instructions before I deleted them
• Understood that if I have changed something that could impact performance of one or more contributions, I should provide results benchmark runs, using the run.sh script, from before and after the change.

Description of changes

This is the start of what I said in #341 that I should try to implement in-process benchmarking for Clojure, Java, and C. Starting with Clojure, to get it as right as possible before cloning the approach to any other languages.

I think the changes are best described in the New runner section in the updated README.md, plus:

• Loops README
• Fibonacci README
• Levenshtein README

Addressing:

• Start times: Make the programs do the benching too #341
• Increase runtime of programs #336

[PEZ]

Now there are implementations for C, Clojure and Java, and Babashka (which can use the Clojure benchmarking utility). Here's what a levenshtein run looks like:

[int2str]

I'll have a crack at the C++ version of the benchmark.

[PEZ]

Awesome, @int2str! I'm curious if you'll be able to use benchmark.c for it. Don't hesitate to make changes to it that would enable sharing.

[int2str]

I've drafted a C++ version -> int2str@c5f3f02

Haven't tried using the C version. I might try that as well.
Though it feels "icky" somehow :D lol

Also, if the return type for the benchmark ever is something more complex than an int, I'd prefer to have my template<> blanket to comfort me :D

[PEZ]

if the return type for the benchmark ever is something more complex than an int

It already is for Levenshtein where the return type is an array.

The reason for sharing is that it is less code to maintain. But if it makes things awkward for the CPP benchmarks that is a reason to implementing something more CPP friendly. Also, providing a separate CPP version could be good for reference, I think.

[int2str]

Updated my draft to print dots as requested.

Btw. Maybe the warm-up runs shouldn't be the same amount as the final run? As in, why not do a 1 or 2 second warm-up and then the 10s run. Seems a bit much to do 10s for both, no?

[PEZ]

I honestly don't know how much warmup is needed. But for JIT compilers it is different than for non-JIT. I can do a bunch of tests on Java and Clojure with different warmups and see if I can find a better guess than “same as benchmark”.

[PEZ]

Can you file a PR on my fork with your changes, @int2str?

[PEZ]

I'm closing this PR and will move the changes to a separate repository/project which will be focused on benchmarking with minimal impact from start/setup times: https://github.com/PEZ/languages-fun

Because the world simply don't have enough of microbenchmark projects! 😄

@int2str PR with your C++ benchmark runner welcome there!

[PEZ]

Alright @int2str. @bddicken and I decided to go for this runner in this repo, so I archived that other repo and now hope you'll PR your C++ implementation here. I'm sorry for the back-and-forth.

[int2str]

Will do some time over the weekend. No worries about the back-and-forth :) January 24, 2025 at 7:32 PM, "Peter Strömberg" ***@***.*** ***@***.***?to=%22Peter%20Str%C3%B6mberg%22%20%3Cnotifications%40github.com%3E > wrote:

…

Alright @int2str https://github.com/int2str . @bddicken https://github.com/bddicken and I decided to go for this runner in this repo, so I archived that other repo and now hope you'll PR your C++ implementation here. I'm sorry for the back-and-forth. — Reply to this email directly, view it on GitHub #365 (comment) , or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOWOV637T6WGN3OYVZQCM32MLLQRAVCNFSM6AAAAABVKZZY6SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJTGY2TSMBVHE . You are receiving this because you were mentioned.

[PEZ]

Cool! I added a separate warmup-ms arg to the programs, instead of just using run-ms for it. The run.ms script defaults to 2000 for warm-up. We can tweak that later if we see that more is needed for some environment. I also changed the benchmark.{clj,java,c} utilities to not run at all if the run-ms is 0. And the run.sh script sends 1, and 0, for run-ms and warmup-ns respectively in the correctness test run. I kept it simple in run.{clj,java,c} and just don't look at the return value of the warmup run. E.g. Executing something like ./c/run 0 0 levenshtein-words.txt will crash or not terminate or something. I think that's fine, running it with 0 run-ms doesn't make sense to me.