Argument passing adventures

When I posted about the Emacs mouse bug in February, I looked into the patch set again, and of course spotted more mistakes. This patch simplifies and corrects some inconsistent logic in where CAP_SYS_ADMIN was required. The change does not make a difference for existing terminal mouse driver programs like GPM or Consolation, but it is noteworthy for the coding aspect:

Fantastic new ways of passing arguments

In the kernel code, the decision whether to apply CAP_SYS_ADMIN is made by looking at an “enum” value (sel_mode) which defines the operation to be run, in a struct which also carries arguments:

struct {
    char  subcode;
    short xs, ys, xe, ye;
    short sel_mode;
};

And in most cases, sel_mode was also used like an enum.

However, at some point someone added the operation TIOCL_SELMOUSEREPORT and found that it required an additional argument. But the existing argument struct did not have space for it! What to do?

Instead of implementing the operation with a different argument struct, the implementers then presumably figured that it would be easier to implement if they re-used the existing struct. So they made TIOCL_SELMOUSEREPORT the number 16 (in binary: 10000) and then used the lower 4 bits of the same enum as an additional argument. (It’s the 90’s. Go for it!)

None of this got documented either (Chaos is the price of freedom), so I also ended up submitting documentation for it to the TIOCLINUX man page, in the section about TIOCLINUX’s TIOCL_SETSEL subcode (highlight mine):

TIOCL_SELMOUSEREPORT

Make the terminal report (xs, ys) as the current mouse location using the xterm(1) mouse tracking protocol (see console_codes(4)). The lower 4 bits of sel_mode (TIOCL_SELBUTTONMASK) indicate the desired button press and modifier key information for the mouse event.

[…]

All of these features are luckily highly obscure and seldomly looked at – I really hope that no one will actually need this documentation again, but if they do, it’s documented now.

What’s the takeaway here?

Admittedly, I should have read the kernel code better and should not have tripped over this. But also, in my defense, people are trained to recognize conventional programming patterns. If code looks conventional on the surface but deviates in surprising ways, it becomes easy to miss. Don’t do this.

Along the way I also wrote up my understanding of Linux’s terminal mouse support. (I should note that I am not a terminal driver expert, and it’s possible that I am misunderstanding parts of it.)

The patch is now in all stable kernels (6.12.26+, 6.14.5+). Versions before 6.7 are unaffected.

Comments