OSDev.org

The Place to Start for Operating System Developers
It is currently Thu Mar 28, 2024 10:42 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 10 posts ] 
Author Message
 Post subject: Pixel perfect click tracking
PostPosted: Fri Aug 26, 2022 7:18 am 
Offline
Member
Member
User avatar

Joined: Mon Jun 05, 2006 11:00 pm
Posts: 2293
Location: USA (and Australia)
Currently, everything in my UI toolkit is rectangular, and it's easy to figure out the UI element under the mouse pointer because I can just test if the mouse pointer falls within the bounding box of the widget, e.g.:

Code:
x >= minX && x <= maxX && y >= minY && y <= maxY


I'm interested in doing something less boxy, and potentially using a drawing library such as Cairo or Skia to draw my UI widgets, and giving my buttons curved corner, my tabs slanted edges, etc.

But testing if I'm clicking on an element becomes more complicated. How do people solve this?

Some ideas I've had:

1) Custom math for every widget. e.g. For a button with curved borders, I could do a basic bounding box test, but then if I fall within the 4 corners where it could be curved, do extra math (distance from the quarter-circle's center point to see that we fall within the curve radius.)

2) Don't care about pixel perfection and just use a rectangular bounding box around everything. Might not be noticable with slight curved edges, although round buttons would be obvious.

3) Have a secondary "click target" buffer, where each pixel isn't a color but a widget index (which could be a pointer), and anytime I draw a widget that is clickable, I make sure to draw the widget in the same shape (with anti-aliasing turned off) to the "click target" buffer in a solid color, where the color is the widget ID. This would give me super fast lookup of whatever is under any pixel on the screen (I'd support arbitrary polygons), but at the expense of more memory and slower drawing.

_________________
My OS is Perception.


Top
 Profile  
 
 Post subject: Re: Pixel perfect click tracking
PostPosted: Fri Aug 26, 2022 8:12 am 
Offline
Member
Member

Joined: Wed Aug 30, 2017 8:24 am
Posts: 1593
Most display libraries choose option 2. The only relatively quick solution I would have for pixel perfection is a quadtree, but with rounded corners that might become very deep at those corners. And all for the minuscule chance that the user clicked one pixel next to a rounded corner and did not actually want to click the button. I mean, if there is a different clickable widget in those rounded corners, the UI designer needs some reeducation.

_________________
Carpe diem!


Top
 Profile  
 
 Post subject: Re: Pixel perfect click tracking
PostPosted: Fri Aug 26, 2022 10:22 am 
Offline

Joined: Fri Nov 26, 2021 11:08 am
Posts: 8
Option 4)

Similar to option 3, but make the render routine respond to a flag that specifies if it’s supposed to render normally or to the hit buffer.


Top
 Profile  
 
 Post subject: Re: Pixel perfect click tracking
PostPosted: Fri Aug 26, 2022 11:13 am 
Offline
Member
Member
User avatar

Joined: Fri Jun 11, 2021 6:02 am
Posts: 96
Location: Belgium
I'd go with 2) but with pixel perfection.

You first do an AABB test to see what the cursor intersects with, which is O(whatever). Then you map the cursor to the widget's space and check if the pixel under the cursor is transparent or not, which is O(1).

_________________
My OS is Norost B (website, Github, sourcehut)
My filesystem is NRFS (Github, sourcehut)


Top
 Profile  
 
 Post subject: Re: Pixel perfect click tracking
PostPosted: Sun Aug 28, 2022 7:17 pm 
Offline
Member
Member
User avatar

Joined: Mon Jun 05, 2006 11:00 pm
Posts: 2293
Location: USA (and Australia)
Demindiro wrote:
I'd go with 2) but with pixel perfection.

You first do an AABB test to see what the cursor intersects with, which is O(whatever). Then you map the cursor to the widget's space and check if the pixel under the cursor is transparent or not, which is O(1).


The part about testing if a pixel is transparent would require a buffer per widget (or at least non-overlapping set of widgets) correct?

You've inspired me to do a combination. I'll do an AABB test, but specific widgets are allowed to override this and do a more specific test.

_________________
My OS is Perception.


Top
 Profile  
 
 Post subject: Re: Pixel perfect click tracking
PostPosted: Mon Aug 29, 2022 12:12 pm 
Offline
Member
Member
User avatar

Joined: Fri Jun 11, 2021 6:02 am
Posts: 96
Location: Belgium
AndrewAPrice wrote:
The part about testing if a pixel is transparent would require a buffer per widget (or at least non-overlapping set of widgets) correct?


Depends on the type of widget. For e.g. widgets that show a single icon you only need to keep one copy of that icon in memory.

_________________
My OS is Norost B (website, Github, sourcehut)
My filesystem is NRFS (Github, sourcehut)


Top
 Profile  
 
 Post subject: Re: Pixel perfect click tracking
PostPosted: Sun Sep 04, 2022 2:35 pm 
Offline
Member
Member
User avatar

Joined: Mon May 22, 2017 5:56 am
Posts: 812
Location: Hyperspace
I'm pretty sure X11's shaped-window extension tests against a series of connected lines, making it a variant of option 1. The lists seemed to me to be very long, even for something simple like xeyes which had only 2 ovals, but it was very quick even on a 486. I might be wrong about the connected lines, it might have been loads of rectangles instead. But all this was without alpha blending.

Alpha blending introduces a UI issue: at what alpha threshold do you pass the click through to the underlying widget? If you pick a value based on what you see on your screen, someone else might see it very differently if their screen has a different gamma. Without drivers supporting color profiles for different monitors, this will happen. I don't have a good technical answer to this. I think the best idea is to remember this issue at the art stage: don't make long gradual alpha fades. Or, if your fades are procedurally generated, maybe have an option to change the gamma value used in generating them.

_________________
Kaph — a modular OS intended to be easy and fun to administer and code for.
"May wisdom, fun, and the greater good shine forth in all your work." — Leo Brodie


Top
 Profile  
 
 Post subject: Re: Pixel perfect click tracking
PostPosted: Tue Sep 06, 2022 3:48 am 
Offline
Member
Member
User avatar

Joined: Mon Jun 05, 2006 11:00 pm
Posts: 2293
Location: USA (and Australia)
eekee wrote:
Alpha blending introduces a UI issue: at what alpha threshold do you pass the click through to the underlying widget? If you pick a value based on what you see on your screen, someone else might see it very differently if their screen has a different gamma. Without drivers supporting color profiles for different monitors, this will happen. I don't have a good technical answer to this. I think the best idea is to remember this issue at the art stage: don't make long gradual alpha fades. Or, if your fades are procedurally generated, maybe have an option to change the gamma value used in generating them.


I think the answer to this is a design question. Only use standard alpha blending for things like shadows where you can click behind the object. Clickable transparent surfaces should do something a little more advance than basic alpha blending, such as blurring the background. Windows calls this "acrylic":
Image

_________________
My OS is Perception.


Top
 Profile  
 
 Post subject: Re: Pixel perfect click tracking
PostPosted: Wed Sep 07, 2022 12:09 am 
Offline
Member
Member
User avatar

Joined: Mon May 22, 2017 5:56 am
Posts: 812
Location: Hyperspace
AndrewAPrice wrote:
I think the answer to this is a design question. Only use standard alpha blending for things like shadows where you can click behind the object. Clickable transparent surfaces should do something a little more advance than basic alpha blending, such as blurring the background. Windows calls this "acrylic":

I think you're right. I can't see any other answer than a design question, and this choice of when to blur works very well.

_________________
Kaph — a modular OS intended to be easy and fun to administer and code for.
"May wisdom, fun, and the greater good shine forth in all your work." — Leo Brodie


Top
 Profile  
 
 Post subject: Re: Pixel perfect click tracking
PostPosted: Wed Sep 07, 2022 1:28 am 
Offline
Member
Member

Joined: Wed Mar 30, 2011 12:31 am
Posts: 676
Note that there was some discussion about just that topic recently.

While I conflate window alpha transparency with clickability in my compositor, I offer applications the ability to set the threshold opacity value at which pixels become clickable. That same threshold is also used to determine whether blurring should be applied if it's enabled, so Andrew's suggested behavior holds true in Yutani at least for the cases where blurring is enabled at all. Unfortunately, blurring large areas correctly is a rather expensive task in a hobby OS without reasonable access to pixel shaders, so I made it opt-in and limited it to a handful of controlled situations by default...

_________________
toaruos on github | toaruos.org | gitlab | twitter | bim - a text editor


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 10 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 25 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group