Ideas for better on/off toggle on mobile

[StarScape]

It's super convenient that 10ten supports iOS and Android. However, my most frequent frustration is that activating it on a webpage on mobile is a bit inconvenient. In contrast to desktop, where I can just hit Alt+R and then start mousing over stuff, on iOS Safari I have to first go to the page's top-left menu and click "10ten" to get the puck.

RPReplay_Final1711412911.mov

I'm wondering if there's a better way. I had two possible ideas:

Allow 10ten to be activated via the context menu

I.e. here:

This is the same number of taps, but somehow feels more convenient to me, maybe because it satisfies the most common use case.

Typically my flow with 10ten goes like: open web page in Japanese, start reading, come across some word I don't know, then activate the extension. If I could just activate it inline with the sentence by selecting the word and hitting "look up with 10ten" or something similar, that feels like quite a nice solution compared to searching through the Safari menu in the upper left.

Allow 10ten to be activated with some sort of phone motion

This one is more speculative, but could be a better solution overall. It's also not mutually exclusive with the above.

What if you could do a sort of 'flick of the wrist' motion (as an example) with the phone that would toggle 10ten for the page? Obviously we would need to find a motion that worked well, didn't interfere with any existing motions (a hard shake of the phone triggers 'undo typing' on iOS, for example), make sure the motion triggers consistently, etc. But I think this could be kinda cool...

I am not sure about the feasibility of either of the above ideas w/r/t what iOS Safari extensions allow. Maybe having to activate the extension from the menu is an inherit Apple limitation? @shirakaba or @birtles might know the answer to this.

I also haven't used 10ten on Android, so I'm not sure if the current experience is similar there.

I welcome thoughts on the above, and any ideas others might have :).

[SaltfishAmi]

I won't want to make 10ten require an additional gyroscope / accelerometer access permission...

[StarScape]

That’s a fair point, but it could always be made optional.

[shirakaba]

I agree the flow to turn on 10ten (well, any Web Extension that runs content scripts, really) is a pain, and these are really nice UX ideas. I imagine myself angrily shaking my phone upon not recognising a word, only for 10ten to come to the rescue.

Unfortunately, Safari Extensions (whether we're talking about the newer iOS-compatible Web Extensions, which 10ten is, or the older macOS-only App Extensions) are very restrictive and there's no way to turn on an extension by any API call, because nothing is exposed to us.

While we may be able to provide a button in the host app to improve the onboarding experience (by deeplinking you into the Settings… > Safari > Extensions screen to enable 10ten browser-wide for the first time), there is no API for the host app to turn on the extension for a given tab, so the user still has to press the jigsaw piece button (this thing: ) and turn on 10ten from the popover by hand.

Given that the host app is a dead-end, let's assess what we can do with the iOS Web Extension. In short, not a lot. We don't have any access to Safari itself. We can't add an item like "Look up with 10ten" into the context menu (the UIMenuController) because we don't have access to the responder chain of those text elements (in fact, you couldn't access them even if you made your own web browser around WKWebView – WebKit doesn't expose the responder chain even to embedders 🥲).

Even if we could access the context menu, or set up gesture handlers, or accelerometer monitors (all of which I think we can't, because the extension is conceptually more like a worker than a first-class corner of the app), there is no API to turn on/off a Web Extension, and if you think about it – if it hadn't been turned on yet, it'd be surprising if it could run code.

I do wonder though, whether Shortcuts might have a hacky way to do it, if it supports accessible navigation 🤔 I'm not very familiar with it, but it can do some surprising tricks. It would be pretty neat if we could have the Assistive Touch button on-screen system-wide at all times (a sacrifice you have to make) and for it to be able to launch a Shortcuts script or something to go through the workflow to turn on 10ten. But still seems clunky, and now you'd have two pucks on screen at once. 😅 Maybe we could activate a Shortcuts script via Siri..? Or ask Siri to do it directly? But still, I think Siri is slow enough that I'd rather just pick the item out of the menu myself.

[birtles]

Thanks for bringing this up!

In contrast to desktop, where I can just hit Alt+R and then start mousing over stuff

Just a side note here in case you're not aware, there's an option to show the popup when you hold a key like Ctrl and a lot of users find this even easier still. i.e. you leave 10ten enabled all the time and just hold Ctrl when your mouse is over a word you don't know.

Allow 10ten to be activated via the context menu

I agree this would be convenient. Implementation might be hard though. I think we already register a context menu item via the standard Web extension API:

Although you can disable/enable this from the options:

Perhaps on iOS we have that setting disabled? More likely, I suspect Web extension context menu items don't show in that particular selection menu.

Since Safari Web extensions include a native code component we could probably get an item added there but it would be a bit of work and wouldn't apply to Android. It would still be nice though so if someone's willing to dig into the iOS APIs and write the Swift code to implement this I'd be happy to assist.

Allow 10ten to be activated with some sort of phone motion

That's an interesting idea. I'm not sure how reliable gestures are but it might be a nice option.

I wonder, however, if the best thing might be to work on getting the Safari version to be able to be permanently enabled on all sites. On Android where we use a different permission model, once you activate the extension it remains activated for all pages. You can disable the puck by tapping it so that it becomes translucent. That way it's there for you to tap on all pages and you just enable it when you need it.

If that seems like a suitable workflow for iOS too we should work out how to get the Safari version to be able to do the same. It will be quite a bit of risky work but we need to do it eventually and we commonly get requests from Safari users about this.

[birtles]

Haha, I just saw @shirakaba's comment above and it looks like the native code path might not be possible after all. Sorry!

[shirakaba]

I wonder, however, if the best thing might be to work on getting the Safari version to be able to be permanently enabled on all sites. On Android where we use a different permission model, once you activate the extension it remains activated for all pages. You can disable the puck by tapping it so that it becomes translucent. That way it's there for you to tap on all pages and you just enable it when you need it.

This (being always-on) is subtly different to what I'd discussed (turning the extension on upon some interaction), and may be possible. Indeed, I think my retired LinguaBrowse macOS Safari App Extension may have been set up like that (always on, for all sites). It would inject a content script into every page (well, frame) with a flag basically saying "hide for now" until you then went into the Safari menubar, clicked onto the extension and toggled the "unhide" switch. So it would always effectively be there, injecting scripts, just dormant.

You might end up with a subtly different permissions flow (mainly just wording of messages) if you do switch to an always-on approach, but maybe it's rather a Web Extensions vs. App Extensions thing:

My tentative "it may be possible" point of view is because I am rusty on all this, and:

1. The LinguaBrowse extension is for macOS Safari, not iOS Safari
2. It's an App Extension, not a Web Extension

... so I don't know whether Web Extensions can be always-on, at least on iOS Safari. It might be perfectly possible!

The other thing is, as an end-user, I spend a lot of my time reading English stuff rather than Japanese, so I ideally wouldn't want to have even the translucent puck in the way, and wouldn't want any extra page startup delay or network usage.

But, happy to discuss and work out what's the best path forwards.

[birtles]

Right, on platforms other than Safari we are doing "Always Allow on Every Website". What I'd like to do is have all platforms start off in the same mode as the current Safari version (activeTab mode) and allow users to progressively expand permissions to particular sites or all sites if they wish. (Incidentally, allowing for all websites, even temporarily, would fix the issue with 10ten not working on cross-origin iframes on Safari.)

The other thing is, as an end-user, I spend a lot of my time reading English stuff rather than Japanese, so I ideally wouldn't want to have even the translucent puck in the way, and wouldn't want any extra page startup delay or network usage.

Yeah, I definitely understand that. I've been using 10ten on my Android phone recently keeping it always on and I've found I got used to the translucent puck very quickly and I don't even notice it anymore. That's just me though. There are probably things we can do to make it even less obtrusive, however.

Injecting the content script doesn't trigger any network usage and shouldn't introduce any measurable page startup delay but that would be worth checking and optimizing.

I wonder if we can find some clever way of detecting Japanese content that doesn't cause a performance penalty. Then we could hide the puck completely for pages that will never use it. The tricky part is responding to dynamic changes to the content, however.

[shirakaba]

Right, on platforms other than Safari we are doing "Always Allow on Every Website". What I'd like to do is have all platforms start off in the same mode as the current Safari version (activeTab mode) and allow users to progressively expand permissions to particular sites or all sites if they wish.

Progressive expansion for all platforms sounds sensible, yeah. Lets users choose how much they want to make use of it.

There are probably things we can do to make it even less obtrusive, however.

Make the puck shrink by 1% each time you navigate to a new page without using it 😄

But more seriously, I wonder if we could come up with a very specific yet generally non-interfering gesture (something like how right-click works on trackpads: place one finger down, then a second finger down alongside it) to either to bring up a dropdown menu with multiple options or toggle the puck between hidden and visible.

I wonder if we can find some clever way of detecting Japanese content that doesn't cause a performance penalty. Then we could hide the puck completely for pages that will never use it. The tricky part is responding to dynamic changes to the content, however.

Options with varying tradeoffs of cheapness to success rate:

• check the meta tags (is there a relevant tag/attribute)?
• register a MutationObserver at page load to:
  • check for lang="jp" attributes in all elements, or
  • search for CJK characters (hiragana and katakana alone might generally suffice) on page load
• check for usages of infamous Japanese fonts
• just do it for .jp sites

None particularly attractive.

[birtles]

But more seriously, I wonder if we could come up with a very specific yet generally non-interfering gesture (something like how right-click works on trackpads: place one finger down, then a second finger down alongside it) to either to bring up a dropdown menu with multiple options or toggle the puck between hidden and visible.

I'd be open to that. I think it wouldn't necessarily exclusive with some of the other ideas here too.

Options with varying tradeoffs of cheapness to success rate:

* check the meta tags (is there a relevant tag/attribute)?

* register a MutationObserver at page load to:
  
  * check for `lang="jp"` attributes in all elements, or
  * search for CJK characters (hiragana and katakana alone might generally suffice) on page load

* check for usages of infamous Japanese fonts

* just do it for `.jp` sites

None particularly attractive.

Yeah, the particular approach I had in mind was a combination of those:

1. On page navigation, we use the host / URL to do an initial check. We maintain an MRU cache of domains/sites where we've seen Japanese text or had the add-on explicitly enabled etc. We might also augment that with some speculative heuristics like looking for .jp sites or inspecting meta tags. If we got a match, we'd enable the puck immediately. That way when you're navigating between pages on the same site, the puck is always there.

2. If we failed to get a match in step 1, we'd register an idle callback that would run some time after page load, and do a one-time check of the page text contents for Japanese characters. That's assuming a call to document.documentElement.textContent isn't too expensive. We'd also probably need to check the text of form fields if that's not included in Element.textContent and possibly shadow DOM content too, I'm not sure. If we got a hit there we'd enable the puck, add the site to the MRU cache and be done.

3. If we failed to get a hit at step 2, we'd register a mutation observer and check for Japanese characters added to the document later. If we ever got a hit, we'd cancel the mutation observer, add the site to the MRU cache, and show the puck.

[StarScape]

Glad to see this has generated some good discussion! @birtles, @shirakaba, you both know the code and technical restraints much better than I do, so I don't have a lot to add there, but I have a few thoughts from a (mostly) user perspective.

Allow 10ten to be activated via the context menu

I agree this would be convenient. Implementation might be hard though. I think we already register a context menu item via the standard Web extension API:

Yeah, unfortunately @shirakaba confirmed what I already suspected when he said this wasn't possible on iOS 😢.

I wonder, however, if the best thing might be to work on getting the Safari version to be able to be permanently enabled on all sites.

I agree the always-on approach seems like the way forward. However, I also agree with @shirakaba that I don't want an omnipresent puck—at least, not without guarding it behind an option.

So I think there are two alternatives here, ranked by desirableness (as I see it, at least):

1. Move to an always-on permissions model, and pair it with a reliable gesture to quick activate the puck. The puck would also remain activateable via the same menu it is now.
2. If we can't figure out a suitable activation gesture, we can at least add an option to keep it always visible, similar to how @birtles is describing it already works on Android. Personally, I think this option should be off by default—even though you might get used to it after a while, I worry that installing 10ten and immediately having a little black dot on the screen for every website might deter potential users. However, I agree with the CONTRIBUTING.md that "options are a refuge for the indecisive," hence why I'm ranking this as less preferable than the above.

What I'd like to do is have all platforms start off in the same mode as the current Safari version (activeTab mode) and allow users to progressively expand permissions to particular sites or all sites if they wish. (Incidentally, allowing for all websites, even temporarily, would fix the issue with 10ten not working on cross-origin iframes on Safari.)

Forgive me for the rather dense question, but what's the motivation behind that change? Again, speaking mostly from a user perspective here, but there are there are other browser extensions I've used that have to be toggled on a per-website basis and it winds up being a PITA—particularly with something as broad as a dictionary, which I'm using on numerous sites, including ones I might only visit once. Having to reactivate it all the time gets to be annoying.

I do appreciate the extra layer of security with some extensions. But for 10ten, which is open source and I'm (fairly 😉) sure isn't yeeting all my data to the highest bidder, it's not such an issue. Again, just my perspective though.

I wonder if we could come up with a very specific yet generally non-interfering gesture (something like how right-click works on trackpads: place one finger down, then a second finger down alongside it) to either to bring up a dropdown menu with multiple options or toggle the puck between hidden and visible.

This is an interesting interaction design problem, I definitely think we could definitely come up with something for this. If you all are interested, I could see about prototyping out some ideas for activation gestures. I have a few ideas, also welcome any you might have.

None of the gesture code need find its way into the 10ten codebase yet—I could prototype it on a test webpage to prevent needing to muck with any of the current puck activation code, until we're happy with the gesture and ready to pull it into 10ten.

Regardless, the key would be a lot of testing. My main concern is coming up with something that's easy to activate once you know how, but hard to activate on accident. I'm not particularly concerned with discoverability. I view this as being analogous to a keyboard shortcut. A power user feature, not necessarily discoverable unless you're looking for it. It would certainly be worth adding a toggle in the settings though, with an accompanying sentence or two explaining the gesture, just like is currently done with keyboard shortcuts.

Yeah, the particular approach I had in mind was a combination of those:

I like what you've outlined here, and agree from a user perspective that having an auto-activation when there's Japanese on the page would be nice, as long as it doesn't cause lag.

That's assuming a call to document.documentElement.textContent isn't too expensive.

That's my concern as well. Also, iterating over all text to look for CJK characters could get costly depending on the size of the document.

Interestingly, I recently worked on a somewhat-adjacent problem, where I had a lot of text to process, more than JS could handle at once without causing noticeable UI lag. My solution was to chunk the text and break up the processing into a separate async task for each chunk. You then execute as many of these tasks as you can before reaching a certain millisecond threshold. After that you cede the remaining tasks to a setTimeout(..., 0) so the next frame(s) can run, rinse, and repeat until the queue is empty.

This approach is not without its downsides. To borrow a DS analogy, it's kind of like moving to an eventual consistency model: helps with latency, but introduces a lot more potential sources of error. However, once MutationObservers get involved, a lot of that complexity will be there anyway.

One additional thought on scanning for CJK characters—what about false positives on Chinese websites? It's not an issue I'm likely to have, but I imagine the overlap of people who study Japanese and Chinese to be pretty high.

[birtles]

This is an interesting interaction design problem, I definitely think we could definitely come up with something for this. If you all are interested, I could see about prototyping out some ideas for activation gestures. I have a few ideas, also welcome any you might have.

None of the gesture code need find its way into the 10ten codebase yet—I could prototype it on a test webpage to prevent needing to muck with any of the current puck activation code, until we're happy with the gesture and ready to pull it into 10ten.

Absolutely, please prototype away! I'd be most happy to incorporate that if we can find something that seems workable.

[birtles]

That's assuming a call to document.documentElement.textContent isn't too expensive.

That's my concern as well. Also, iterating over all text to look for CJK characters could get costly depending on the size of the document.

Interestingly, I recently worked on a somewhat-adjacent problem, where I had a lot of text to process, more than JS could handle at once without causing noticeable UI lag. My solution was to chunk the text and break up the processing into a separate async task for each chunk. You then execute as many of these tasks as you can before reaching a certain millisecond threshold. After that you cede the remaining tasks to a setTimeout(..., 0) so the next frame(s) can run, rinse, and repeat until the queue is empty.

Right. I think processing the text would be manageable since we have various options for optimizing that. If simply calling document.documentElement.innerText on a large document (e.g. the one-page version of the HTML spec) is expensive, however, there might not be a lot we can do about it other than trying to run it in an idle callback.

[StarScape]

If your extension has more than 10,000 users you will get offers from companies wanting to buy your extension so they can mine personal data. Obviously 10ten will never sell out to these companies but providing users with a way to gradually increase the level of privilege they grant the add-on gives an extra bit of piece of mind that at least 10ten is able to work without extensive privileges.

Wow, that is interesting—and frightening!

Yes, I agree that for most users always-on is preferable. At very least, we want to ensure that existing users are unaffected since they've already opted in to enabling it for all sites by installing it.

Totally agree, and thanks for laying out your reasoning. I'm definitely all for more better privacy guarantees. My only concern was that the always-on remains an option, for convenience reasons, but sounds like you're on the same page there 🙂.

Absolutely, please prototype away! I'd be most happy to incorporate that if we can find something that seems workable.

Will do! Will post any updates here.

[StarScape]

Right. I think processing the text would be manageable since we have various options for optimizing that. If simply calling document.documentElement.innerText on a large document (e.g. the one-page version of the HTML spec) is expensive, however, there might not be a lot we can do about it other than trying to run it in an idle callback.

Sorry, this is probably another dense question, but is there any reason you couldn't just walk the DOM, and call innerText on each element one at a time, rather than doing it on document.documentElement once? There could still be a case where there's a ginormous string entirely contained within one DOM element, but it's probably a lot more rare in practice.

[birtles]

Sorry, this is probably another dense question, but is there any reason you couldn't just walk the DOM, and call innerText on each element one at a time, rather than doing it on document.documentElement once? There could still be a case where there's a ginormous string entirely contained within one DOM element, but it's probably a lot more rare in practice.

Absolutely, that could work. It would introduce a race condition in that new content may be added to nodes we've already scanned while we're still scanning the rest of the document—but if we unconditionally set up a MutationObserver at the start of the scan we could handle that.