Skip to content

Rework hyperfine invocation#350

Closed
benknoble wants to merge 2 commits intobddicken:mainfrom
benknoble:rework-hyperfine-invocation
Closed

Rework hyperfine invocation#350
benknoble wants to merge 2 commits intobddicken:mainfrom
benknoble:rework-hyperfine-invocation

Conversation

@benknoble
Copy link
Contributor

I have:

Description of changes

convert run.sh to run

Importantly, it puts all the possible commands into a single hyperfine
invocation, which can help you get a relative comparison in the final
report (see below). Also, use a normal number of runs and warmups to get
more statistically significant results.

The "runnable" test also improves: it checks that the program exists &&
that the runner program (which may be the same) exists and is
executable. That means if you don't have miniruby installed, for
example, it should be skipped.

The use of Zsh is for portability with associative arrays (macOS's
/bin/bash is still too old on most systems).

Other noteworthy changes:

  • dropped Hare because it didn't appear to work with existing script
  • dropped commented out programs
  • "descriptions" lost, but could be re-added in a separate parallel
    array
  • kept Clojure program filename, though it doesn't look right to me
  • common-lisp file check includes .lisp suffix (in run.sh, only the
    command had the suffix)
  • Clojure runner construction with -Spath might not work…
  • do not ignore failures when running benchmarks (8d3432e (run: do not ignore failures when running benchmarks, 2025-01-12))

Example run with just Racket and C:

# ../run
Benchmark 1: racket/code aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...
  Time (mean ± σ):      5.327 s ±  0.197 s    [User: 4.991 s, System: 0.112 s]
  Range (min … max):    5.118 s …  5.700 s    10 runs

Benchmark 2: c/code aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...
  Time (mean ± σ):     836.5 ms ±   3.8 ms    [User: 832.3 ms, System: 1.9 ms]
  Range (min … max):   832.5 ms … 843.3 ms    10 runs

Summary
  c/code aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa... ran
    6.37 ± 0.24 times faster than racket/code aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...

Makes-progress-on: #320


This may still be too big as a single PR (and is not how I'd prefer to present the changes); if it's helpful, I can try to split out most of the "other noteworthy changes" into separate commits (one of them already is). NB that takes more work on my end ;)

Importantly, it puts all the possible commands into a single hyperfine
invocation, which can help you get a relative comparison in the final
report (see below). Also, use a normal number of runs and warmups to get
more statistically significant results.

The "runnable" test also improves: it checks that the program exists &&
that the runner program (which may be the same) exists and is
executable. That means if you don't have `miniruby` installed, for
example, it should be skipped.

The use of Zsh is for portability with associative arrays (macOS's
/bin/bash is still too old on most systems).

Other noteworthy changes:
- dropped Hare because it didn't appear to work with existing script
- dropped commented out programs
- "descriptions" lost, but could be re-added in a separate parallel
  array
- kept Clojure program filename, though it doesn't look right to me
- common-lisp file check includes .lisp suffix (in run.sh, only the
  command had the suffix)
- Clojure runner construction with -Spath might not work…

Example run with just Racket and C:

    # ../run
    Benchmark 1: racket/code aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...
      Time (mean ± σ):      5.327 s ±  0.197 s    [User: 4.991 s, System: 0.112 s]
      Range (min … max):    5.118 s …  5.700 s    10 runs

    Benchmark 2: c/code aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...
      Time (mean ± σ):     836.5 ms ±   3.8 ms    [User: 832.3 ms, System: 1.9 ms]
      Range (min … max):   832.5 ms … 843.3 ms    10 runs

    Summary
      c/code aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa... ran
        6.37 ± 0.24 times faster than racket/code aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...

Makes-progress-on: bddicken#320
@PEZ
Copy link
Collaborator

PEZ commented Jan 25, 2025

Hi! With #370 we no longer use hyperfine for the main benchmarking.

I do think some of your “Other noteworthy changes” may still apply. I understand if you don't want to do all this work again, and if so, I will use your commits as a TODO list for myself. 😄

@benknoble
Copy link
Contributor Author

Hi! With #370 we no longer use hyperfine for the main benchmarking.

I do think some of your “Other noteworthy changes” may still apply. I understand if you don't want to do all this work again, and if so, I will use your commits as a TODO list for myself. 😄

@PEZ (& @bddicken)—indeed, I have other side projects that I wish to spend time on. I've done quite a bit of effort to repeatedly beat things into shape, and I need to move on. Best of luck.

@PEZ PEZ closed this Feb 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants