Add mouse drag operations

[peterbrittain]

Currently, asciimatics only supports mouse click events. Assuming most *NIX terminals are like xterm, drag operations should be possible too. Windows can definitely do them.

The idea would be to add more flags to the mouse event type to clarify the mouse button up/down and movement, allowing users to detect drag an drop operations.

[peterbrittain]

This should also add support for moving Frames around the Screen when added.

[pearmini]

So does asciimatics support mouse drag operations now?

[peterbrittain]

Sorry, but no. That's why this enhancement is still open. The main issue here is finding the right settings for Linux terminals. Once I have that in place, the code shouldn't be too hard.

[pearmini]

It's very glad to hear that you still working on it, and I'm looking forward to your solution because I need it in my recent project while I have no idea how to solve it.

[EricBalingit]

The main issue here is finding the right settings for Linux terminals.

Can you point me to what you're referring to?

[peterbrittain]

@EricBalingit Turns out understanding Linux terminal mouse support is tricky... Best I could find was https://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h2-Mouse-Tracking

[EricBalingit]

Hmm, I see.

Maybe this helps? Not sure how inclusive it is, but seems to be on the right track.

https://github.com/prompt-toolkit/python-prompt-toolkit/blob/master/prompt_toolkit/output/vt100.py#L536

https://github.com/prompt-toolkit/python-prompt-toolkit/blob/master/prompt_toolkit/input/vt100_parser.py#L23

Still digging to find where the blob is normalized into a mouse event, so far I only see it being matched and sent along.

[peterbrittain]

Not sure what to make of that code. It looks like it relies on partial implementation of the xterm options on various different terminals and then xterm (and any code based off of it) will end up in extended SGR mode. IIUC, none of those options actually enables mouse drag messages.

[EricBalingit]

Ah, ok, so mouse move when button state is non-zero. I suppose reducing/eliminating throughput is expected given the design/purpose of such systems.

I'm not used to tty, coming from DOS/Windows where a key has an ascii code and screens have memory addresses. :-)

I did notice that mouse move seems to thrash the refresh rate in windows.

Might be worth handling mouse events in a thread and only reporting when row, col changes, but I suppose that's not the expected behavior.

[EricBalingit]

Btw, thanks for this lib! Really neat idea, just had a need for more layout options at the moment.

Also by going through your win32 code, I was able to get mouse working with prompt-toolkit.

Funny the docs didn't mention anything about disabling quick edit mode, but that did the trick for me.

[peterbrittain]

Thanks. If there's a feature you need, feel free to raise a new issue.

Yeah - quickedit is a pain for any console app that cares about the mouse because it stops sending events to your app.

You shouldn't see any change to the refresh rate on events unless you allow it by specifying allow_int=True on the call to play(). This was the compromise for people that want a super responsive UI as opposed to the predictable refresh provided by default. Typically that means I set it to False on animations and True for TUIs in the samples.

[EricBalingit]

:D

I understand, but I was suggesting that you can have the best of both by threading the event loop ( if it's not already ), but wait for events and only report mouse events e.g. by saving last row,col and then passing along new events when they differ from the saved row, col.

I see this on mouse move, and it's related to drag.

And when you get drag working, could be drastic on large layouts - depending on how you schedule redraw. Might be worth considering.

The event rate for move ( in windows at least ) is not throttled and can be upwards of 1 millis between events.

That's why it can pull down the ui refresh.

[EricBalingit]

but only for mouse move, not sure how button events interfere with that kind of process

[EricBalingit]

button changes would be passed, but only move if the row, col changes

I know it's hackish, but it's best I can think of, unless you're willing to discard mouse moves until the scheduled refresh, and only report the most recent mouse position at that time. ( this is a typical technique for throttling mouse move events in html/javascript )

But I can see the blinking cursor hesitates when I move the mouse, to me that spells bad news for layout refresh.

[EricBalingit]

but maybe it's because I'm not fully aware of the looping procedure here :-)

[peterbrittain]

Aaahh... I see what you're getting at now. Thanks. Yes - there should definitely be throttling of the event processing.

Note that there is nothing giving the user feedback for any input until the UI refreshes, so this is a question of total delay from user interaction to seeing it appear on the screen. The existing design handles this by allowing events to queue up between refreshes and then processing them in bulk before the screen refresh. This means that there is already implicit throttling in the existing design: if your scene is complex to render and takes (say) 100ms, the fastest refresh you can get is 10 times a second. However, it does allow you to run riot at the other end (where refresh cost trivial).

The way most systems handle this is using a frame rate cap. Should be trivial to do that here. Just enforce that limit on the delay if specified. The existing input processing will then handle all the queued mouse events without any need for any peeking/summarizing.

However, it does show the weakness in the design of relying on frame count for animation. This was a compromise when I first added the widget sub-package. The whole system was originally pre-determined by the 20Hz refresh cycle, but people expect faster interaction when typing. The slight difference between 10ms and 50ms is really noticeable to some people - hence the allow_int compromise. Changing this isn't going to be backwards compatible, so should be something to add to the v2 API ideas - detaching animation sequencing from frame counters. I'll add that to the list.

PS: I have deliberately avoided using separate threads in the design. It does all the work required (processing all input and re-drawing the screen) in a single frame. You then just need to schedule that frame refresh regularly, making it trivial to run inside any async framework.

[EricBalingit]

Also worth noting, some older mouse wheels are analog, and can report a range of ticks, not a constant value per event.

It's up to the OS to decide how often to report wheel events, and what the tick value is, but I don't know how the tick value is handled by terminals.

So this too can thrash the layout in some cases.

[djtorrey]

in the interim would it be fair to assume something like an XTERM=1002 or XTERM=1003 and based on https://tldp.org/HOWTO/NCURSES-Programming-HOWTO/mouse.html send a new int value for BUTTON_1_RELEASED or BUTTON_3_RELEASED?

Then at the very least you'd be able to manually track when the button is pressed and released. In my case, I am using asciimatics as the renderer but wrote my own update loop which i poll each frame and would easily be able to set an internal value to see if the released button has been signaled yet.

It sure would make my little custom terminal sprite-painter easier to work with having a click-drag fill capability :-)

[peterbrittain]

Yeah - the theory should be to enable the right mode and then just process the events on Linux. However, last time I tried, I got some weird results that confused button presses... windows would just be a matter of adding the extra events to the event processing.