Fix iOS touch mapping

[Darkyenus]

iOS backend did not map touch coordinates to libGDX correctly. libGDX's GL View is stretched over whole screen, but libGDX draws only on a subset of that, because it excludes status bar, if present. Touch mapping was however completely oblivious to this fact - this worked well when the status bar was not visible, but was inaccurate when the status bar was indeed visible.

This PR rewrites rotation/resize handling and touch coordinate mapping to explicitly handle mentioned problems.

This is somewhat related to issues:
#2105 - although this seems to be the opposite issue, it is not present after this PR
#3278 - does not fix this, but allows it to be fixed in user code by overriding IOSApplication.getBounds and providing correct bounds

Note that I am not an iOS expert, but I have tested it and it works.

[Darkyenus]

Any thoughts?

[intrigus]

On which devices did you test it?

[Darkyenus]

iPhone 4S and a few simulators. Testing on other devices with and without the fix to make sure that the problem is indeed present everywhere and this fixes it, is welcome.

[badlogic]

Looks good, tested in iPad Mini as well.

[Darkyenus]

Thanks!

[lastbestmatt]

Hey, I'm noticing some issues that I think are related to this on an old iPod 4 -- iOS 6.1.6. It seems like touch coordinates remain in the portrait reference frame even after the display changes to landscape. (I've never had the status bar visible in this app.)

There was certainly flakey behavior before related to orientation changes, but it generally worked. After this, it seems like the display dimensions change more consistently/correctly on orientation change, but the touch coordinates are now consistently incorrect.

Any thoughts, or things I can check for?

Thanks.

[Darkyenus]

This will be hard to test for, I don't have any iOS 6 device and I am not sure if I can get a simulator. So if you decide to debug it yourself I'll provide any help you need.

I guess that the problem will be in one of iOS methods called in getBounds() - Apple likes to change what some of the methods do, just slightly but enough to break everything between versions.

[lastbestmatt]

Ok, I've looked into this a bit further. It appears that getBounds() is working correctly - the check for screenHeight > screenWidth there is catching the coordinate reversal. (And this probably explains why the graphics do rotate correctly with this version.)

However, in IOSInput.toTouchEvents(), this line to get the touch coordinates

CGPoint loc = touch.getLocationInView(touch.getWindow()); 

seems to return the un-flipped version. Or at least, if I touch the same part of the physical screen in portrait and landscape orientations, I get about the same x and y coordinates in loc.

It looks like this is maybe something that changed between iOS 6/7 and 8?
http://stackoverflow.com/questions/24150359/is-uiscreen-mainscreen-bounds-size-becoming-orientation-dependent-in-ios8

Any thoughts on the best way to resolve this? I know it's probably not worth too much effort to keep supporting the older versions, but this is the only problem I've run across, and everything else is working well.

Thanks for the help.

[Darkyenus]

Thanks for the investigation! Please try it with PR #3709 applied, if I am not mistaken it should fix it.

I can't get the simulator (only iOS 8 and 9) but I believe this issue applies to the iOS 6 and 7 (maybe later) so it is worth fixing. Too bad Apple tries so hard to break backwards compatibility.

[lastbestmatt]

Great, thanks for looking. I'd like to try this out, but I'm not set up to build libgdx from source yet. I should get that going...

These online docs seem a bit old, are they still accurate?
https://github.com/libgdx/libgdx/wiki/Building-libgdx-from-source
https://github.com/libgdx/libgdx/wiki/Running-Tests

In particular, is it still eclipse-only? And is it known to be working on Mac? (I know this shouldn't matter but it always seems to...)

[Darkyenus]

Running (iOS) from source is somewhat complicated, I find it easier to:

• Create your libGDX game project normally, with core and backend projects
• Clone libGDX sources & apply PR
• From game's iOS project, remove dependency on iOS backend (not natives!) and add transient dependencies of iOS backend explicitly (= robovm runtime)
• Copy sources of RoboVM backend from libGDX repo to your game and compile it as part of your game, not as part of libGDX

I have found this easier than figuring out gradle, which I am not 100% familiar with, but you can try it if you are - if I remember correctly it should be possible to build libGDX via gradle only (or with maven and ant? that is what confuses me).

[lastbestmatt]

All right, I was finally able to test your commits in #3709. It looks like this is on the right track, but it's not quite so simple. The conversion isn't so easy as checking axesFlipped -- it depends on the actual orientation. LandscapeLeft is different than LandscapeRight, and same with Portrait and PortraitUpsideDown.

To test I wanted to just get LandscapeLeft (home button on left) working. I changed your code in toTouchEvents to the following:

{
    CGPoint loc = touch.getLocationInView(touch.getWindow());
    double x = loc.getX();
    double y = loc.getY();
    if (screenBounds.axesFlipped) {
        // note: this works for LandscapeLeft only!
        locX = (int)(screenBounds.x + screenBounds.width - y);
        locY = (int)(screenBounds.y + x);

        // LandscapeRight would be something like this:
        // locX = (int)(screenBounds.x + y);
        // locY = (int)(screenBounds.y + screenBounds.height - x);
    } else {
        locX = (int)(x - screenBounds.x);
        locY = (int)(y - screenBounds.y);
    }
}

With this code, I can confirm the touch events seem correct in both Portrait and LandscapeLeft orientations (although I don't fully understand the status bar stuff -- I'm not using it). But they are still incorrect for LandscapeRight (because the transformation would be different as noted).

Let me know what you think!

[MaxGyver83]

I'm sorry to tell you but on my iPhone 3GS with iOS 6.1.6 everything was fine up to libGDX 1.8.0. When I upgrade to 1.9.x, x and y are flipped regarding touch (in my game with landscape orientation). Graphics look as before.

My code looks like this:
Gdx.input.setInputProcessor(new InputAdapter() {
public boolean touchDown(int x, int y, int pointer, int button) {
Vector3 touchPos = new Vector3();
touchPos.set(x, y, 0);
camera.unproject(touchPos);
x = (int) touchPos.x;
y = (int) touchPos.y;
...

[Darkyenus]

@MaxGyver83 I believe this is the same problem as reported by @RungKutta. I have been looking at the code and I understand how can it be broken now, but I don't understand why it wasn't broken before, the initialization of GLKView is pretty much the same and the touch coordinates are unmapped as before, only offsets are added, which does not explain the flipping.

Does anybody have any idea what happened/why did it break? I wanted to implement fix suggested by @RungKutta (thanks btw!) but now I am thinking that there is a different issue/easier fix to this and this would just band-aid it.

[lastbestmatt]

@Darkyenus , I've looked briefly at the changes, and I also don't see an obvious reason why this broke. I wonder if it has to do with a change in RoboVM?

I first noticed this problem when upgrading my setup from libgdx 1.7.0 -> 1.9.1 and RoboVM 1.8.0 -> 1.12.0. There seem to have been other changes related to orientation; in particular, switching from landscape to portrait and back is much smoother (looks like a normal iOS app) in the newer version. And while the touch coordinates generally seemed to work before, it wasn't 100% consistent. There were times that I had to rotate back and forth between landscape and portrait to get everything (including both touch coordinates and graphics coordinates) working right.

[Darkyenus]

I don't think that this is caused by RoboVM, they provide just a wrapper around iOS API.
I have created a diff of things that have changed since 1.7.0 in RoboVM backend and there isn't anything that stands out as a culprit.

There is a change in how the backend is build near the top, which I don't fully understand but I doubt that this could cause this. GLES3 has been added but is disabled by default and the same codepath is taken for GLES2, so that can't be it. My bound calculation is just getters, so that should be fine.

GLKView creation has been changed slightly (the CGRect calculation) but I believe that the values actually passed are still the same, so that should be fine, but may be worth investigating. Rest are changes in input handling and stubs.

And finally, what I think has the biggest probability of being problematic, is that instead of didRotate we now override viewDidLayoutSubviews in IOSGraphics.java. Can you perhaps try moving the code from viewDidLayoutSubviews to didRotate and see what happens? Also note that the original didRotate had super.didRotate(), try it with and without that, maybe there is a default behavior, somewhere?

I have tried running old Xcode to be able to test on old simulators, but that didn't work, so I'll probably have to try it on a virtual machine. Thanks Apple.

[lastbestmatt]

Hey, check out this change in toTouchEvents that I noticed when browsing through your diff:

Old: CGPoint loc = touch.getLocationInView(touch.getView());
New: CGPoint loc = touch.getLocationInView(touch.getWindow());

If I switch this line back to use getView instead of getWindow, the touch coordinates seem to work fine again.

I'm not sure about all of the implications of this, and whether it was changed for a reason or not, but it seems to fix the problem at least on my old iOS 6 setup.

[Darkyenus]

Good catch! So the axis flipping is not needed anymore after this?

[lastbestmatt]

Yeah, not in IOSInput. Still needed in getBounds() though.

[Darkyenus]

Ok, can you please try again with #3709 applied? I have amended the commits and it should be fixed, I don't know what it was using window instead of view, probably a mistake. And thanks for help!

[lastbestmatt]

Yes, this one works for me. Thanks!