mouseButton contains incorrect value after pressing both mouse buttons and releasing one of them

[servobot]

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build process
• Unit testing
• Internationalization
• Friendly errors
• Other (specify if possible)

p5.js version

1.9.0

Web browser and version

122.0.6261.95 (Official Build) (64-bit) (cohort: Stable)

Operating system

Windows 10 Home Edition

Steps to reproduce this

Steps:

1. Press and hold left mouse button
2. Press and hold right mouse button
3. Release left mouse button
   • Expected: console logs left
   • Actual: console logs right

Snippet:

function setup() {
  createCanvas(400, 400);
}

function mouseReleased() {
  console.log(mouseButton);
}

function draw() {
  background(220);
}

Additional Context

I discovered this issue when implementing the game of Minesweeper. In that game, pressing and releasing both mouse buttons at the same time is the way to uncover all unmarked neighbors of a cell, and I found that I could not detect this condition reliably using mouseButton. I believe this is due to mouseButton being updated in mousePressed but not mouseReleased.

https://github.com/processing/p5.js/blob/main/src/events/mouse.js#L657
https://github.com/processing/p5.js/blob/main/src/events/mouse.js#L733

I have a pull request (with associated unit test) that appears to resolve the issue. This bug was previously reported in #5733, but the submitter resolved that issue as they found a workaround.

[welcome]

Welcome! 👋 Thanks for opening your first issue here! And to ensure the community is able to respond to your issue, please make sure to fill out the inputs in the issue forms. Thank you!

[Renatolo]

Hey, I would like to work on this bug, please!

[limzykenneth]

This is more or less intended behavior documented and I'm not keen to change the behavior. There is probably room to have better support for simultaneous mouse button presses as currently having the context menu being brought up by the right mouse button can confuse things often.

There is a larger proposal here for better mouse event handling that I think can be explored in p5.js 2.0.

[diyaayay]

In the PR: #7378, setting mouseButton on _onmousedown() works well but I noticed pointer events work differently than mouse events meaning _setMouseButton() is only called when a single button is pressed as onpointerdown() is not invoked multiple times on pressing simultaneous mouse buttons. Instead _onpointermove() is triggered when multiple mouse buttons are clicked in pointer events: https://stackoverflow.com/questions/75030295/pointer-events-dont-support-simultaneous-mouse-buttons. So, I attempted to handle mouse buttons by checking which buttons are pressed currently, using the buttons property, which continuously changes as pointerdown, pointermove, and pointerup events occur. This approach clearly shows which buttons are currently pressed during simultaneous button presses. However, this would require updating the description of mouseButton accordingly. @limzykenneth @davepagurek

fn._setMouseButton = function(e) {
    // Using e.buttons bitmask to track which buttons are pressed
    this.mouseButton.left = (e.buttons & 1) !== 0;   // Left button
    this.mouseButton.center = (e.buttons & 4) !== 0;  // Middle button
    this.mouseButton.right = (e.buttons & 2) !== 0;   // Right button
};

[davepagurek]

I'm in favour of replacing the single variable with an object with left, right, and center properties like you're proposing, this seems like a useful breaking change to make.

Do those pointer move events that update the buttons also come attached to a specific pointer ID? I wonder if, in addition to a single global mouseButton property for use in single-pointer sketches, we could also store this same info per pointer in our touches array (or under whatever name we end up giving that.) So, touches[i].mouseButton.left would represent whether the left button is down for a specific pointer, while mouseButton.left would represent whether any pointer has its left button down.

[limzykenneth]

@diyaayay Would the proposed approach means that if there are only pointerdown event but not pointermove, pressing down on left mouse button followed by right mouse button without releasing the left will cause only left mouse button to be registered?

I haven't quite use this part of things directly very much before but perhaps the element.releasePointerCapture described here may point to a potential solution? If you have a test sketch, do share and I can test it out locally as well.

@davepagurek Pointer events do come with pointer ID so we can identify individual pointer.

[diyaayay]

Would the proposed approach means that if there are only pointerdown event but not pointermove, pressing down on left mouse button followed by right mouse button without releasing the left will cause only left mouse button to be registered?

@limzykenneth Yes, that is what I've been seeing. So, for multiple mouse clicks only mouseup and mousedown sets the button that we have been using. You can see the values on multiple button clicks here in an mdn playground here. But, the same thing does not work for pointerup and pointerdown as seen in the pointer example.
But subsequent mouse presses are recognised as pointermove event that match the mouse buttons Working pointer example. I've tested the same thing in sketches also.
I'm not too sure though. I may be missing something. But mouseButton being an object seemed like something that gives more info to the user about pressed and released state of each button.
Some pointer devices, such as mouse or pen, support multiple buttons. In the [UIEVENTS] Mouse Event model, each button press produces a mousedown and mouseup event. To better abstract this hardware difference and simplify cross-device input authoring, Pointer Events do not fire overlapping pointerdown and pointerup events for chorded button presses (depressing an additional button while another button on the pointer device is already depressed).

Instead, chorded button presses can be detected by inspecting changes to the button and buttons properties. The button and buttons properties are inherited from the MouseEvent interface, but with a change in semantics and values.
Ref:

[diyaayay]

Untitled.design.7.mp4

For the sketch:

function mousePressed() {
  console.log(mouseButton);
}

function setup() {
  createCanvas(400, 400);
}

function draw() {
  background(220);
}

This is how mouseButton is displayed, but we can change it to not keep it as an object in order to maintain the current behavior.
@davepagurek @limzykenneth