mmap input files rather than reading

[evmar]

[Consolidating some comments from #107 and #108]

Currently we read input files into a local buffer. I had thought this was better than mmap because:

1. we need a trailing nul bye on the file for parser reasons
2. performance -- I can't remember where I had read it, but I recall the argument that mmap can be costlier than reading when it's a large file you're reading sequentially, due to some kernel overhead related to setting up all the memory mapping data structures(?)

However, it turns out that you can just mmap a private page at the end of the file to write the nul, and @Colecf says "Judging based on n2's tracing output, just reading the file into memory takes over twice as long as the android fork's entire ninja invocation".

[evmar]

Relevant code in Android's Ninja fork. (Not sure how to get a permalink to that commit but the file is unlikely to change much in the future, but it's the function MapFile.)

[evmar]

I wrote a mini benchmark here.

On this Mac Air M2 for a random 1.4gb file:

without madvise:

  ./target/release/read-file mmap $F ran
    1.25 ± 0.01 times faster than ./target/release/read-file read $F

with madvise:

  ./target/release/read-file mmap $F ran
    1.36 ± 0.02 times faster than ./target/release/read-file read $F

So I conclude at least for this hw/OS that mmap is faster, and madvise helps too.

[Colecf]

Relevant code in Android's Ninja fork. (Not sure how to get a permalink to that commit but the file is unlikely to change much in the future, but it's the function MapFile.)

Sorry, this link should be permanent.

[Sarcasm]

Just a thought regarding "correctness" more than performances.

Is there any risk of the mmaped build.ninja changing during the build?
I had a bad experience trying to mmap user-controlled file, an update of the file at the wrong time could cause my process to crash with SIGBUS.
Edit: it could just be I didn't use the proper type of memory mapping.

The build.ninja file can be regenerated during build by the build system (I think cmake does this when a CMakeLists.txt changes), can this invalidate the memory mapping?

[evmar]

can this invalidate the memory mapping?

The Linux man pages says it's undefined behavior. The Mac ones don't say either way. It is true that we wouldn't want to preserve the mapping beyond the point where the file might be rewritten though.