mprotect can fail on Linux when /proc/sys/vm/max_map_counts is exceeded

[pipcet]

(I'm working on the feature/igc branch of GNU Emacs which adds MPS as a garbage collector. You haven't heard much from us since things are going very well generally, and it's usually our fault when they don't)

However, we've now seen several reports which indicate the mprotect call in protix.c:ProtSet failed, which leads to unreachable code being reached and an Emacs crash:

  /* .assume.mprotect.base */
  result = mprotect((void *)base, (size_t)AddrOffset(base, limit), flags);
  if (MAYBE_HARDENED_RUNTIME && result != 0 && errno == EACCES
      && (flags & PROT_WRITE) && (flags & PROT_EXEC))
  {
    /* Apple Hardened Runtime is enabled, so that we cannot have
     * memory that is simultaneously writable and executable. Handle
     * this by dropping the executable part of the request. See
     * <design/prot#impl.xc.prot.exec> for details. */
    prot_all = PROT_READ | PROT_WRITE;
    result = mprotect((void *)base, (size_t)AddrOffset(base, limit), flags & prot_all);
  }
  if (result != 0)
    NOTREACHED;

protix.txt says this:

_`.fun.set.assume.mprotect`: We assume that the call to ``mprotect()``
always succeeds.  We should always call the function with valid
arguments (aligned, references to mapped pages, and with an access
that is compatible with the access of the underlying object).

Our current theory is that this assumption is violated on Linux, which keeps a maximum number of contiguous maps in /proc/sys/vm/max_map_count, with the default at 65530. The way it appears to work is that when a single page in a large contiguous map is mprotected to be different than its neighbors, that map turns into three maps for the purposes of the max_map_count limit, and mprotect eventually returns ENOMEM.

Experiments with lowering the max_map_count limit in a virtual machine appear to be consistent with that theory, because it causes Emacs crashes much like the ones that were reported by our users.

Here's the Emacs bug report:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=76705

[pipcet]

It seems to me like it's tricky, but we should be able to work around this: mprotect is usually used for memory barriers, and by triggering/removing those eagerly we should be able to reduce the number of active maps not to exceed a given threshold. That way, we should at least be able to save the current session and restart a new Emacs session after advising the user to increase ARENA_GRAIN_SIZE or the sysctl parameter to avoid the situation...

Ideally, we'd put the arena into the clamped or even parked state when this happens, and send a message to the client. I'll have a look at the code to see how we might be able to do so.