Closed
Conversation
Importantly, it puts all the possible commands into a single hyperfine
invocation, which can help you get a relative comparison in the final
report (see below). Also, use a normal number of runs and warmups to get
more statistically significant results.
The "runnable" test also improves: it checks that the program exists &&
that the runner program (which may be the same) exists and is
executable. That means if you don't have `miniruby` installed, for
example, it should be skipped.
The use of Zsh is for portability with associative arrays (macOS's
/bin/bash is still too old on most systems).
Other noteworthy changes:
- dropped Hare because it didn't appear to work with existing script
- dropped commented out programs
- "descriptions" lost, but could be re-added in a separate parallel
array
- kept Clojure program filename, though it doesn't look right to me
- common-lisp file check includes .lisp suffix (in run.sh, only the
command had the suffix)
- Clojure runner construction with -Spath might not work…
Example run with just Racket and C:
# ../run
Benchmark 1: racket/code aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...
Time (mean ± σ): 5.327 s ± 0.197 s [User: 4.991 s, System: 0.112 s]
Range (min … max): 5.118 s … 5.700 s 10 runs
Benchmark 2: c/code aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...
Time (mean ± σ): 836.5 ms ± 3.8 ms [User: 832.3 ms, System: 1.9 ms]
Range (min … max): 832.5 ms … 843.3 ms 10 runs
Summary
c/code aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa... ran
6.37 ± 0.24 times faster than racket/code aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...
Makes-progress-on: bddicken#320
Collaborator
|
Hi! With #370 we no longer use hyperfine for the main benchmarking. I do think some of your “Other noteworthy changes” may still apply. I understand if you don't want to do all this work again, and if so, I will use your commits as a TODO list for myself. 😄 |
Contributor
Author
@PEZ (& @bddicken)—indeed, I have other side projects that I wish to spend time on. I've done quite a bit of effort to repeatedly beat things into shape, and I need to move on. Best of luck. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I have:
Description of changes
convert run.sh to run
Importantly, it puts all the possible commands into a single hyperfine
invocation, which can help you get a relative comparison in the final
report (see below). Also, use a normal number of runs and warmups to get
more statistically significant results.
The "runnable" test also improves: it checks that the program exists &&
that the runner program (which may be the same) exists and is
executable. That means if you don't have
minirubyinstalled, forexample, it should be skipped.
The use of Zsh is for portability with associative arrays (macOS's
/bin/bash is still too old on most systems).
Other noteworthy changes:
array
command had the suffix)
Example run with just Racket and C:
Makes-progress-on: #320
This may still be too big as a single PR (and is not how I'd prefer to present the changes); if it's helpful, I can try to split out most of the "other noteworthy changes" into separate commits (one of them already is). NB that takes more work on my end ;)