feat: Support orientation change on iOS

[ldalzottomp]

What

While integrating the react-native-vision-camera library in our application, we stumbled upon an issue.
When rotating the device, the camera feed were not folloing the interface orientation.

Changes

I tried to add support for auto orientation on iOS by listening to the orientation change notification.
Then, update the connection input orientations based on UI interface orientation.
I also updated the input orientation on session start.

I have tried to integrate this feature as close as possible as the current architecture. As I am not familiar with camera and your project, feel free to tell me what to change.

I did not observed the initial faulty behaviour on different Android devices.

Before :

before_compressed.mov

After : I entered in the device home screen at some point

after_compressed.mov

Tested on

iPad Pro (11-inch) (2nd generation). With both orientation locked screen and unlocked.
iPhone 11. No orientation change as the UI is orientation locked.

Related issues

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
react-native-vision-camera	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 8, 2024 1:29pm

[mrousavy]

Thanks for your PR, but I wonder why you didn't use the setOrientation extension I already wrote?

react-native-vision-camera/package/ios/Extensions/AVCaptureOutput+mirror.swift

Lines 24 to 58 in 6b71d4f

	/**
	Sets the target orientation of the video output.
	This does not always physically rotate image buffers.
	- For Preview, an orientation hint is used to rotate the layer/view itself.
	- For Photos, an EXIF tag is used.
	- For Videos, the buffers are physically rotated if available, since we use an AVCaptureVideoDataOutput instead of an AVCaptureMovieFileOutput.
	*/
	func setOrientation(_ orientation: Orientation) {
	// Set orientation for each connection
	for connection in connections {
	#if swift(>=5.9)
	if #available(iOS 17.0, *) {
	// Camera Sensors are always in landscape rotation (90deg).
	// We are setting the target rotation here, so we need to rotate by landscape once.
	let cameraOrientation = orientation.rotateBy(orientation: .landscapeLeft)
	let degrees = cameraOrientation.toDegrees()
	// TODO: Don't rotate the video output because it adds overhead. Instead just use EXIF flags for the .mp4 file if recording.
	// Does that work when we flip the camera?
	if connection.isVideoRotationAngleSupported(degrees) {
	connection.videoRotationAngle = degrees
	}
	} else {
	if connection.isVideoOrientationSupported {
	connection.videoOrientation = orientation.toAVCaptureVideoOrientation()
	}
	}
	#else
	if connection.isVideoOrientationSupported {
	connection.videoOrientation = orientation.toAVCaptureVideoOrientation()
	}
	#endif
	}
	}

Is that different from your code?

[ldalzottomp]

but I wonder why you didn't use the setOrientation extension I already wrote?

@mrousavy I did that as a first try !

The issue that I encountered is that it's updating the output.

In the case of a phone.
The user can put his phone on landscape mode.
And in this case, the developer may want the output in landscape mode with the "orientation" prop. But may not want to switch the input orientation because the UI orientation is locked.

Basically, that would be good for tablet, but not for phones or locked tablets.

Here, I feel that we really want something "UI wise". Which is : does the camera feed orientation follows the UI orientation or not.
Basically, I feel that we need to have a distinction between inputOrientation and outputOrientation ("orientation" prop). To overcome the issue I just described.

---

So the difference that I see is that the piece of code you highlighted apply to the output (AVCaptureOuput) of the photo capture. Which I don't think this is always what we want. Because the input (camera feed) and the output can be different.

But the logic of this extension can be extracted somewhere else to be reused.

[ajp8164]

This may be related.
#1619

[zero2603]

Hi, I am using react native v0.70.4 and react-native-vision-camera v2.16.2, I solved camera orientation for both iOS and Android. And I want to share this trick for everyone. Please follow below steps:

• Install react-native-sensors. Then create custom hook useDeviceRotationSensor.js:

import { useLayoutEffect, useRef } from 'react';
import { Platform } from 'react-native';
import { setUpdateIntervalForType, SensorTypes, gravity } from "react-native-sensors";

/**
 * 
 * Reference: https://github.com/mrousavy/react-native-vision-camera/issues/624
 */
export const useDeviceRotationSensor = (callback) => {
    setUpdateIntervalForType(SensorTypes.gravity, 1000);

    const callbackRef = useRef(callback);
    callbackRef.current = callback;

    useLayoutEffect(() => {
        // We use gravity sensor here because react-native-orientation
        // can't detect landscape orientation when the device's orientation is locked
        const subscription = gravity.subscribe(({ x, y }) => {
            const radian = Math.atan2(y, x);
            const degree = (radian * 180) / Math.PI;

            let rotation = 'landscapeLeft';

            if (Platform.OS === 'android') {
                if (degree > -135) rotation = 'portraitUpsideDown';
                if (degree > -45) rotation = 'landscapeRight';
                if (degree > 45) rotation = 'portrait';
                if (degree > 135) rotation = 'landscapeLeft';
            } else {
                if (degree > -135) rotation = 'portrait';
                if (degree > -45) rotation = 'landscapeLeft';
                if (degree > 45) rotation = 'portraitUpsideDown';
                if (degree > 135) rotation = 'landscapeRight';
            }

            callbackRef.current(rotation, degree);
        });

        return () => subscription.unsubscribe();
    }, []);
};

• Open /node_modules/react-native-vision-camera/ios/CameraView+Orientation.swift, then replace current code with below code (reference to comment: 🐛 [iPadOS] If app is set to landscape only, vision camera gets preview orientation wrong #1455 (comment)) :

//
//  CameraView+Orientation.swift
//  VisionCamera
//
//  Created by Marc Rousavy on 04.01.22.
//  Copyright © 2022 mrousavy. All rights reserved.
//

import Foundation
import UIKit

extension CameraView {
  /// Returns the current _interface_ orientation of the main window
  private var windowInterfaceOrientation: UIInterfaceOrientation {
    if #available(iOS 13.0, *) {
      return UIApplication.shared.windows.first?.windowScene?.interfaceOrientation ?? .unknown
    } else {
      return UIApplication.shared.statusBarOrientation
    }
  }

  /// Orientation of the input connection (preview)
  private var inputOrientation: UIInterfaceOrientation {
    return windowInterfaceOrientation
  }

  // Orientation of the output connections (photo, video, frame processor)
  private var outputOrientation: UIInterfaceOrientation {
    if let userOrientation = orientation as String?,
       let parsedOrientation = try? UIInterfaceOrientation(withString: userOrientation) {
      // user is overriding output orientation
      return parsedOrientation
    } else {
      // use same as input orientation
      return inputOrientation
    }
  }

  internal func updateOrientation() {
    // Updates the Orientation for all rotable connections (outputs) as well as for the preview layer
    DispatchQueue.main.async {
      // `windowInterfaceOrientation` and `videoPreviewLayer` should only be accessed from UI thread
      let isMirrored = self.videoDeviceInput?.device.position == .front

      self.videoPreviewLayer.connection?.setInterfaceOrientation(self.inputOrientation)

      let connectionOrientation = self.outputOrientation
      self.cameraQueue.async {
        // Run those updates on cameraQueue since they can be blocking.
        self.captureSession.outputs.forEach { output in
          output.connections.forEach { connection in
            if connection.isVideoMirroringSupported {
              connection.automaticallyAdjustsVideoMirroring = false
              connection.isVideoMirrored = isMirrored
            }
            connection.setInterfaceOrientation(connectionOrientation)
          }
        }
      }
    }
  }
}

• Install patch-package, then run command: yarn patch-package react-native-vision-camera
• In your file which is using , you can code as below:

...
const [deviceOrientation, setDeviceOrientation] = useState('portrait');
useDeviceRotationSensor(rotation => {
    if (rotation != deviceOrientation && !isRecording) {
        setDeviceOrientation(rotation);
    }
});
...
return (
    <Camera
        // ...otherProps
        orientation={deviceOrientation}
    />
)

Hope this helps!!

[IamHungry113]

any update here ? cause I encountered the same problem, in version 3.9.1, if the app orientation is locked, the output of the camera is not the same as expected， just like @ldalzottomp said
so any patch here ?

[ldalzottomp]

@IamHungry113 The changes in this PR were enough to cover the use case that I mentioned. So you can grab it and apply it to a local fork. My public fork is here (https://github.com/ldalzottomp/react-native-vision-camera). However there is build issue regarding older iOS version. If your app don't target iOS lower than the requested verion, you can add if conditions to prevent build errors.

I discussed with @mrousavy about this PR and this is not going to be merged as the video feed/frame processor have not been thoroughly tested.

[MCervenka]

Thank you, this fixed my problem.