Shortcuts API

[ocornut]

Writing down notes about implementing a shortcut system. I have started using menus more extensively in my own applications, and the lack of support for shortcuts has become a little annoying.

Without shortcuts the user is required to duplicate code to handle menu items and shortcuts for a same action. Consider that MenuItem() can takes both a "checked" and "enabled" parameter which may need to be fetch/computed from your application state, duplicating that code can be pretty annoying and ImGui strive to reduce redundancy so it's quite a flaw to have to do that.

In the old Menus API #126 thread I said we'd need;

Support local keyboard shortcuts, We can use Windows syntax of using & (e.g. "&Save") for convenience.

And

Support general "global" shortcuts (e.g. "CTRL+S"). As a design goal of ImGui we want to avoid code and state duplication, so I'd like the ImGui system to handle shortcuts for the user. It will be optional but likely available by default. So the program can have a single entry point for "Save" whether it is activated via clicking in the menu or via pressing the shortcut. The way it would work is that when a global shortcut scheme is activated, the menu functions always notify the user code to develop its content so ImGui can parse and execute the shortcuts as they are declared, but the actual menu is not layed out nor rendered. The shortcut scheme can be disabled on a per-menu/window/global basis. In particular, procedurally generated menus that may have infinite depth will need to be able to disable the global shortcut scheme. In its "closed" state, the system has to be as lightweight as if the user were testing a bunch of shortcuts themselves. The scope of shortcuts can be dependent on factor such as if the parent window is focused so they aren't always "global". The user should also be able to display the label for a shortcut in the menu without letting ImGui handle the shortcut itself.

So here is a rough list of what I think we need. Unfortunately some of those will need the user to update their ImGui integration to provide the necessary inputs, but there won't be any breakage.

PART A, for regular shortcuts (typically global shortcuts, but they can be local to a window)

EDIT Not needed!
- [ ] We are going to need translated characters, aka the "A" in "CTRL+A" or "ALT+A" is a translated character. So the end-user application needs to feed those inputs probably via io.AddInputCharacter() and this isn't really a problem for ImGui to solve. However, and that's very surprising, retrieve this information under Windows is NOT straightforward. The WM_CHAR message isn't sent when ALT or CTRL are pressed. WM_SYSCHAR is only sent when ALT is pressed. No CHAR messages are sent when CTRL is pressed. By adding this in a Windows message handler I seem to be able to retrieve those characters.

    case WM_KEYDOWN:
    case WM_SYSKEYDOWN:
            if (GetKeyState(VK_CONTROL) & 0x8000)
            {
                BYTE keys[256];
                memset(keys, 0, sizeof(keys));
                keys[VK_SHIFT] = (GetKeyState(VK_SHIFT) & 0x8000) ? 0x80 : 0;
                WCHAR buf[4];
                const int scancode = (lParam >> 16) & 0x1ff;
                const int buf_sz = ToUnicodeEx(wParam, scancode, keys, buf, 4, 0, NULL);
                for (int n = 0; n < buf_sz; n++)
                    io.AddInputCharacters(buf[n]);
            }
            return 1;

It is a little scary but appears to work. End-user would need a similar mechanism which is rather annoying. For Windows messages user can copy demo code if they are pointed to it. GLFW has been patched in the 3.1 branch to support the ALT key but not the CTRL key yet (would be nice if it does so), what that means is that support for those inputs won't be widespread in most applications soon and it is only likely to be widely available in GLFW when 3.2 ships (also means that support for CTRL key would better be pushed in GLFW before they release 3.2!). Or user can freely do their own cheap conversions if they don't care about funky localisation things.

I'd be curious to know whether GLFW 3.1 gives you character inputs in ImGui_ImplGlfw_CharCallback when ALT or CTRL are pressed on MacOS, Linux, etc. See following rely for instructions on how to perform the test. If you can do a test let me know which version of GLFW you are using. Thanks!

• Need to add all non-printable to the ImGuiKey_ enum: functions keys, insert, keypad stuff.. Unfortunately I didn't add those initially, my bad :( (Been thinking about adding a "Tester" to the demo code that helps you verify your ImGui integration, test things like clipping rectangle, texture changes, all sort of inputs. So this tester could list the keys and make sure you can input them.)
• We are going to need to parse shortcuts string efficiently. Encoding should fit in 32-bits (e.g. 16-bit value, 1 bit to select translated character or key, 3 bits modifiers, 2 bits scope: window, window-and-parent, global). Maybe just parsing on the spot (hardcoded stricmp for prefixes such as "CTRL+") is simpler and faster but we need some sort of fast handling of all the named keys that aren't character (e..g F5, HOME, etc.) so this may make the encoding slower. Also figure out a syntax to specify shortcut scope, or this may be not in the string but rather specified in the callee function. If parsing is slow, perhaps hash/FNV1a the string and cache parsed result as a single u32 stored in a ImGuiStorage which is a contiguous sorted map, touching O(N log N) 8-bytes entries. Either way both are rather easy to try.
• Function that checks if an encoded shortcut is pressed. May or not be merged with the parsing function. Trivial.
• Menus need to be able to run in a mode where nothing renders but the global shortcuts are still processed. As per my old comment above, this may be optional and not the default, and the user needs to be able to disable the feature to allow for recursive menu. Perhaps the feature not spreading to sub-menu automatically would make more sense?

PART B (for & ALT-style local shortcuts, lesser priority)

• Menus needs to process the & ALT-style local shortcuts. The behavior little different from the other shortcuts and may imply keyboard controllable menu, at least support for the Enter key.
• We are going to need be able to parse and render the "&Save" syntax for & ALT-style local shortcuts. The text size calculation and text renderer will need a flag to support this feature. For size calculation, we can treat & as zero-width. For rendering, we need to query the width of the next character and draw an underscore below the baseline. We might or not need to handle inhibition with the \ character to be able to render regular & in this mode. All that would be normally easy BUT one problem is that CalcTextSizeA() is often a major bottleneck in very large UI and thus we can't afford to make it slower so those feature would have to be designed accordingly and will need to alter the low-level text rendering API. (Unrelated to this we want to allow centered and right text alignment on a per-line basis and this would have an effect on the text rendering API as well).
• How about adding support for shortcuts to regular widgets like Buttons? Problem with & ALT-style is that it may have a small overhead with text-processing. But may be a nice default and thus we may design the feature expecting it to be always on. For CTRL-style shortcut, probably best to avoid encoding them in a label and just accept that the user can do their own shortcut checking aka
  if (ImGui::Button("Refresh") || ImGui::IsShortcutPressed("F5") { .. }.

That's it for now. Those are merely notes for myself. If there are

[emoon]

I tested this (in the opengl_example) on Mac and added a printf() in KeyCallback function and I got there everytime when pressing some key and holding ctrl, alt or command.

[ocornut]

Sorry my request wasn't clear, the problem is with the CharCallback not the KeyCallback.

To enable CharCallback with modifiers in GLFW 3.1 you need change
glfwSetCharCallback(window, ImGui_ImplGlfw_CharCallback);
to
glfwSetCharModsCallback(window, ImGui_ImplGlfw_CharCallback);
And add the extra integer to the ImGui_ImplGlfw_CharCallback function.
It isn't done in the default ImGui example because this function was introduced in 3.1 which is not in some package repos.

[emoon]

Did the changes and Alt seems to work just fine. Ctrl seems to work in some cases but not in others.

[ocornut]

Any specific pattern with CTRL working vs not working?
Do simple things like Ctrl+A, Ctrl+Shift+A, Ctrl+Alt+A works? Are the broken cases on specifc keys?

Thanks for looking, this is really helpful. This stupid app-side input problem has been the number one barrier for adding shortcuts so I am eager to solve it. I submitted a GLFW tentative patch for Windows but have no way to look into Mac right now.

For this specific use of shortcuts frankly I don't mind too much if some weird keys don't work seeing it is an application-side problem that can be fully solved later, I just would like basic support to work for most users.

[emoon]

So it seems Ctrl+A-Z doesn't work but 0-9 and other non A-Z keys (like /= etc) works fine.

Alt seems to work on all keys. Ctrl+Alt-.... behaves the same as Ctrl- only

[ocornut]

Thank you, very useful. Taking it to glfw.

[emoon]

np :)

[ocornut]

The first point in the list appears to be the most problematic. It looks like I may just need to redesign the IO API, which was desirable anyway, to switch to an event-registration system so e.g. have a AddKeyPress the same way we have a AddInputCharacter and the user may pass a optional stringified localized key name along with keypress.

May or not need to break compatibility with the user-glue code, very obviously if I can avoid it I'll try very hard to avoid it. I always liked how ImGui took this nice shortcut of taking a sample of input state because it makes initial integration with all sort of libraries easier, but we also need a way out of that.

[emoon]

Yeah that sounds. Good also something I have been thinking about for a while (sorry for being a bit of topic) is the versioning of the lib. Something I have started to come to like is "Semantic Versioning" http://semver.org I kinda like this approach as it's clear when breaking changes comes that may cause issue. This would of course break the current versioning and it's of course up to you to use or not but may be something to consider.

[zhiayang]

Sorry to jump on like this, but would it be possible to allow for more flexibility in the shortcut modifier? On OS X it's quite annoying whenever an application doesn't implement Cmd+A/S/V/N/X shortcuts, and chooses to go with Ctrl+whatever.

I already maintain a local patchjob of ImGui::InputTextEx that handles the OS X shortcut style and text editing behaviour (rather, I just #ifdef it away and include a copied version from a header file). It would be great if the new global shortcut system could allow a little leeway in not hardcoding for Ctrl so much?

Just a suggestion, of course.

[ocornut]

How do you think it would work? I don't know enough about Mac experience to design that. My idea is that the user would pass in strings to the api, e.g. "CTRL+S". Then it could be the responsibility of user code to pass in "CMD+S" there? Or an optional/default behavior would be to allow to remap CTRL to CMD by default? (both for key testing and display).

Is the Command key a wholly different key? Does typical third API feed it as an modifier? GLFW has GLFW_KEY_LEFT_SUPER/GLFW_KEY_RIGHT_SUPER keys for the Command key and a keyboard modifier GLFW_MOD_SUPER so that's all ok. Is Super a good terminology that can adapt to both Command and Windows key?

EDIT SDL2 has KMOD_GUI/KMOD_LGUI/KMOD_RGUI so I'll assume other API are supporting this as a modifiers keys as well.

Also - you could post your InputTextEx() mod for reference?

[emoon]

So the way I have done that in the past is like this:

https://github.com/emoon/rocket/blob/master/ogl_editor/src/Menu.c?ts=4#L26

EMGUI_KEY_COMMAND, EMGUI_KEY_CTRL

First part is the modifier on Mac and the other is on Windows/Linux. The thing is that while the Windows key isn't used that much on Windows actually for shortcuts inside programs that is certainly the case on Mac.

[zhiayang]

Ah, thanks for responding. I've changed a number of things actually:

1. Command key for shortcuts. In theory it should work on windows (if ENABLE_OSX_TEXT_EDITING is defined to be 0)
2. Double click selects a word instead of everything (this is the behaviour on OS X)
3. Movement by word in OS X is done with the ALT key instead of the control key, I put a couple of #if guards in there to handle it as well.

(3) is done a bit hackily, since I'm not too familiar with the code, I did a move left word, select word right.

I know that these changes work on OS X, but since I don't have a windows machine to test on (and I'm not sure the rest of my code would work), I can only guarantee my changes to work... "in theory". It's all compile time checks too, so no performance hit either.

I also added a KeyCmd member in the ImGuiIO struct, which is basically the same as KeyAlt and the others. It's updated in the backend input event function as well.

Here's the diff: https://gist.github.com/zhiayang/5764aacd621ebbed6f0b
(um... i have a habit of changing the whitespace, so in case the diff is too unreadable there's the whole function below. terribly sorry)

Here's my edited version: https://gist.github.com/zhiayang/27b3f032a054e503484a

[ocornut]

On InputTextEx( ) for Mac: regardless of the shortcut API above we should probably merge those three into master, add the KeyCmd member. They could be a runtime settings (for slightly better build coverage) and set the default differently based on __APPLE__ define. If someone wants to try making a clean version of that I'll merge else I'll give it a go sometimes.

For shortcuts, we could require the user to pass in different strings per os for apps caring about portability in this manner (not all imgui apps will do), but that would make user code more bulky. Or support a syntax like "CTRL|CMD+S", simple but a little weird. Or have a system where by default "CTRL+S" gets translated to "CMD+S" on Mac ("Mac" defined via a runtime flag again) BUT with a mechanism to enforce actual CTRL when desired, aka a different identifier for CTRL on Mac, e.g. "XCTRL". Or Control key stays "CTRL" and we have a different short identifier meaning "CTRL|CMD"., I don't know how to name that.

CTRL = Ctrl (Windows), Cmd (Mac)
XCTRL = Ctrl (Windows), Ctrl (Mac)
CMD = Cmd (Windows, nearly never used), Cmd (mac)

With R/L variants "LCTRL", "RCTRL".
Looking at Daniel's table it looks like the majority of occurrences of CTRL are to be CMD on Mac, but not all of them.

[zhiayang]

I'll have a go at a proper pull request for this, it shouldn't take long. A few things to clarify though:

1. Cmd/Ctrl should be a runtime setting defaulting based on __APPLE__. Does this bool go in ImGuiIO, or is somewhere else more appropriate?
2. Should the double click text behaviour change as well? Would it always select by word when double clicking, or would this also be OS dependent? If the latter, should it be governed by the same setting as (1) (eg. OSXAlikeBehaviour) or separate, eg. TextShortcutsUseCmdKey and DoubleClickSelectsWord?
3. I'm assuming KeyCmd would be present in the ImGuiIO struct regardless of OS? Should it then be named something more agnostic, like KeySuper or KeyGui? If the answer to the first question is "no", then ignore all this.
4. In a similar vein to (3), should there be new enum members for LSuper and RSuper in the ImGuiKey_ enum? And should the backends set this by default when handling events? Should these be set for Windows systems as well? (Actually idk if Windows passes down such keypresses to applications)
5. I've made a change to stb_textedit as well, where it calls different functions (is_word_boundary) based on whether it's next word forward or backward; this puts it in line with OS X behaviour, where the cursor sticks to the end of the word (before the space) if going forward, and sticks to the beginning of the word (after the space) if going backward. Linux does this too, only Windows (as usual) puts the cursor after the space, regardless of direction. Should this be merged as a local change (just a couple of lines), as a pull request to STB itself, or not at all?
6. Finally, is the current implementation of doubleclick word select (ie. move cursor to previous word, hold shift, move cursor to next word) satisfactory? I could look into properly moving the cursor, if needed.

EDIT:
7. Actually, I think multiples selection in lists or something that's done with Ctrl also needs to be handled. I'll look into it.

Thanks for sticking around I suppose, hope to hear your opinions on this. Should make for a more fluid and comfortable experience for OS X users, at least.

[ocornut]

Late answer and happy new year :)

I'll have a go at a proper pull request for this, it shouldn't take long. A few things to clarify though:

Cmd/Ctrl should be a runtime setting defaulting based on APPLE. Does this bool go in ImGuiIO, or is somewhere else more appropriate?

I'll have to think a little further about that but for now you can put it in ImGuiIO. Down the line I expect user controllable behaviors for variety of things (eg: slider/drag behavior). Probably just keeping things in IO would just be simpler, but for some behavior it might make sense to Push/Pop the value and then perhaps the ImGuiStyle or another structure may feel more suited. For now let's not worry about that.

Should the double click text behaviour change as well? Would it always select by word when double clicking, or would this also be OS dependent? If the latter, should it be governed by the same setting as (1) (eg. OSXAlikeBehaviour) or separate, eg. TextShortcutsUseCmdKey and DoubleClickSelectsWord?

I suggest to make a different bool. Eg: InputTextShortcutsUsesSuperKey, InputTextDoubleClickSelectsWord.

I'm assuming KeyCmd would be present in the ImGuiIO struct regardless of OS? Should it then be named something more agnostic, like KeySuper or KeyGui? If the answer to the first question is "no", then ignore all this.

Yes, easier to always have in the structure regardless of OS. Let's call it KeySuper because glfw is the cool active thing in town for (with a comment mentioning it maps to Cmd/Windows key). People don't really use the Windows key ar all because it is awkward to use.

In a similar vein to (3), should there be new enum members for LSuper and RSuper in the ImGuiKey_ enum? And should the backends set this by default when handling events?

Yes, yes.

Should these be set for Windows systems as well? (Actually idk if Windows passes down such keypresses to applications)

I don't know either. I wouldn't bother setting them unless it looks trivial. If you don't have access to a Windows machine I'll do some basic tests but not worry if there's a hurdle.

I've made a change to stb_textedit as well, where it calls different functions (is_word_boundary) based on whether it's next word forward or backward; this puts it in line with OS X behaviour, where the cursor sticks to the end of the word (before the space) if going forward, and sticks to the beginning of the word (after the space) if going backward. Linux does this too, only Windows (as usual) puts the cursor after the space, regardless of direction. Should this be merged as a local change (just a couple of lines), as a pull request to STB itself, or not at all?

Ideally merged in STB itself with an additional flag. Sean probably won't merge soon but we can apply the patch locally.

Finally, is the current implementation of doubleclick word select (ie. move cursor to previous word, hold shift, move cursor to next word) satisfactory? I could look into properly moving the cursor, if needed.

Haven't looked yet, sorry.
Thanks for sticking around I suppose, hope to hear your opinions on this. Should make for a more fluid and comfortable experience for OS X users, at least.

Thanks for the help!

[ratchetfreak]

Instead of using the "&Save" syntax you could use "Save", VK_S, aka let the user pass in which indices you have to monitor in the keysDown array.

It does require that the user will also need to pass the correct index of which letter to highlight. Because you can't rely on any mapping between them. Then there is no overhead with text processing.

[ocornut]

That would add an extra parameter to every function which would be practically a no-no.
It would be bearable if we only used it for MenuItem but ideally we want local shortcuts for keyboard activation for all widgets (e.g. Button)

[ratchetfreak]

I'm tempted to say to change the string param to a special struct that includes all the data for the & alt menu (plus the label/ID stuff ofcourse).

Which also has a constructor taking a char* so it won't break existing code. Which can then do the bit of parsing needed for the "###ID" stuff and you may as well add the & parsing as well. Also it may be an option to use && to specify a literal & and &&& for the version where the & itself is the shortcut.

Though the viability of that struct may depend on how the ImStr experiment #494 turns out.

[emoon]

On a slightly related note something that would be nice is keyboard navigation of menus (excluding the actual shortcuts) but perhaps that belongs in a separate issue.

[ocornut]

Agree, it would make sense to aim for that support early on (there's a keyboard navigation task and menus could be worked on earlier). Tho up/down without alt+letter may feel incomplete?

On 13 Mar 2016, at 19:39, Daniel Collin notifications@github.com wrote:

On a slightly related note something that would be nice is keyboard navigation of menus (excluding the actual shortcuts) but perhaps that belongs in a separate issue.

―
Reply to this email directly or view it on GitHub.

[emoon]

I was thinking about popups in this case (at least on Mac if you have a popup and press arrow down you get keyboard focus and can navigate it with arrows and then enter to select an item)

But for regular menus that would be nice yes.

[ocornut]

So the simple solution to my problem with obtaining translated characters when ALT/CTRL is pressed to use with shortcuts, is instead to include printables like A-Z 0-9 in the ImGuiKey_ enum and work with key events.

Pros:

• makes the remaining work of the shortcut API attainable :)
• makes it a little easier to share ImGui code using keyboard in a portable way
• don't need to break existing IO API (which would need to be reworked, but it can happen later)

Cons:

• initial setup a little more length/annoying but we'll provide tables with all bindings supported by default.

[xaxxon]

Is this the right thread to watch for things like cmd-a doing a select-all in a text box instead of ctrl-a on os x?

[Diegovsky]

Hey, just wanted to know how this is currently going and if I can help with anything

[ocornut]

Can shortcuts "flash" the parent menu a-la macOS, or show any visual indicator that they've been triggered? Non-modal confirmation for actions like "Save" is a nice thing to have.
[...]
You are right. I've added myself three tasks:
* [ ] Add visual feedback on top-most-visible menu in the blind-shortcut proof-of-concept (should be easy)
* [ ] Add API to trigger visual feedback on menu. [....]
* [ ] Investigate if the current design for Shortcut() is adequate in the sense that it may make it difficult to implement visual feedback. e.g. if (Button("Save") || Shortcut(...)) idiom works well but only until you want the visual feedback. Same thing if we want a tooltip!

FYI there has been another deluge of commits tangential to this, but I now have good proof-of-concept for all of those (e.g. Ctrl+S triggering menu items in closed menu, with visual feedback on said menu), so I may start reintroducing selected parts of the shortcut systems to public API.

[ocornut]

FYI a few weeks ago I have silently pushed a SetNextItemShortcut() api in imgui_internal.h:

ImGui::SetNextItemShortcut(ImGuiMod_Ctrl | ImGuiKey_S);
ImGui::Button("Save");

Which provides the visual feedback.
What's not yet exposed in this API is that we can later take advantage of shortcut routing flags in there, for now this is hardcoded/limited to using the equivalent of ImGuiInputFlags_RouteFocused which uses the focus stack (so a deeper-most Button will be the sole recipient of that shortcut if its window is focused even if a parent window in the focus route also claim the same shortcut, e.g.

// begin("a")
// use shortcut XXX
// beginchild("b")
//   use shortcut XXX
// endchild()
// end();

It does what you expect based on which window is focused, and in an order-independent well (so the call in "a" may be moved after the beginchild and results will be the same).