block/file: be more defensive about the backing file

[iximeow]

in file.rs we'll just open the backing file for whatever permissions the disk should have:

propolis/lib/propolis/src/block/file.rs

Line 175 in 2dc6437

let fp = OpenOptions::new().read(true).write(!read_only).open(p)?;

but we don't get O_EXCL in here anywhere. I'd thought offhandedly that we could take advantage of that for a very weird one-off test configuration (opening a volume from multiple propolises concurrently), but this was appropriately worrying to everyone else with more calibrated fear-o-meter. we should probably open with O_EXCL, which I think we can plumb in with custom_flags()

if Propolis is told to open a file for writing by multiple VMs concurrently that's likely a serious operator error. I'm less opposed to concurrent read-only opens, but caution seems more appropriate here. I've definitely corrupted guests doing this nonsense locally before.

[jclulow]

O_EXCL doesn't actually stop two processes opening the same file, does it? For regular files that is.

[rmustacc]

O_EXCL doesn't actually stop two processes opening the same file, does it? For regular files that is.

No, but this first came up in the zvol case. For devices we should implement it where it is actually asking for exclusive access.