February 6, 2020

A possible mmap bug from Linux kernel

The theory

Before I discuss about today’s bug, here’s a little bit background: https://lwn.net/Articles/758594/

mmap() is not allowed (by standards like POSIX by many years of history) to return an error when given unknown flags… More serious is that there is no way for an application to know that the kernel it’s running on at the moment supports MAP_SYNC at all, since all kernels will return success with that flag set. Any application that is depending on MAP_SYNC to ensure the integrity of its data needs to know for sure that the feature is supported, but mmap() provides no way to obtain that knowledge. (MAP_SYNC is just one of the recently added flags)

People realized that applications should have some mechanisms to validate the flags passed to mmap, so kernel guys proposed a _VALIDATE postfix, specifying that any flags ended with _VALIDATE will return an error if encountered any unknown flags. (fact: there’s only one flag has this postfix, named MAP_SHARED_VALIDATE)

Ironically, this new flag (MAP_SHARED_VALIDATE) caused some new issues. Date back to kernel 4.15 when MAP_SHARED_VALIDATE was proposed, the implementation maintained a variable called LEGACY_MAP_MASK (this and this) to filter known flags.

When kernel version bumps to 4.17, however, mmap was extended to support a new feature called MAP_FIXED_NOREPLACE, but the kernel maintainers (seem to) forget to update the LEGACY_MAP_MASK variable, thus the new MAP_FIXED_NOREPLACE will be identified as unknown flag when combined with MAP_SHARED_VALIDATE.

The most ergonomic proposal for a new mmap flag should include two different versions, in this case, MAP_FIXED_NOREPLACE and MAP_FIXED_NOREPLACE_VALIDATE, the latter will validate other co-existing flags while the first one don’t. (fact: as you might have guessed, there’s a flag called MAP_FIXED.)

Get hands dirty

We know the story and the background, let’s validate it.

The idea is simple: MAP_SHARED_VALIDATE is incompatible with MAP_FIXED_NOREPLACE, while other combinations should all work.

Running the following code by

clang++ test_mmap_validate.cpp -o test_mmap_validate && ./test_mmap_validate

And the result below shows exactly what we guessed.

MAP_SHARED_VALIDATE | MAP_FIXED:                0x4f0000000000
MAP_SHARED_VALIDATE | MAP_FIXED_NOREPLACE:      unknown flags    0xffffffffffffffff
MAP_SHARED | MAP_FIXED:                         0x4f0000000000
MAP_SHARED | MAP_FIXED_NOREPLACE:               0x4f0000000000

Xiangpeng Hao 2020