✨ Implement Orientation ($8,000)

[mrousavy]

What feature or enhancement are you suggesting?

Currently, there is no direct orientation support for VisionCamera.

This feature request contains of multiple parts:

• Support orientation prop: When this is passed, the Camera will rotate itself to the target orientation
• Take Camera Sensor orientation (often 90deg) into consideration for relative orientation
• Save correct orientation in captured photos (relative to sensor and relative to device/orientation prop)
• Save correct orientation in recorded videos (relative to sensor and relative to device/orientation prop)
• Pass correct orientation in Frame in Frame Processor (relative to sensor and relative to device/orientation prop)
• Documentation on how to properly handle orientation (see iOS Camera app, the camera itself should not rotate, only controls should rotate)

This comes with a ton of research, development and testing on both iOS and Android so it's quite complex to handle properly.

Also, fun fact: On Android this is even more annoying. Check out this blog post - Samsung exposes orientation wrong, meaning I have to add special handling for such devices. Insane.

The Android documentation is quite okay though.

$8,000

I'll implement Orientation for $8,000. Donate to the polar pool to support the feature.

[mrousavy]

Related: https://stackoverflow.com/questions/48406497/

[mrousavy]

Also related: https://developer.android.com/codelabs/android-camera2-preview#5

[vanenshi]

This issue is causing problems with Google MLKit Face detection.
here is the library I am trying to build vanenshi/vision-camera-face-detector

Technically MLKit needs the correct orientation to be able to detect the face, so when I manually set it to 90 deg, it works in portrait mode

[drLeonymous]

Hi, all!
I'm not a Java/Kotlin friend but I did some research and here is a patch that works on the emulator Pixel_2_XL and Samsung Note10 and some other old Samsung or Huawei devices.

--- a/package/android/src/main/java/com/mrousavy/camera/CameraView+TakePhoto.kt
+++ b/package/android/src/main/java/com/mrousavy/camera/CameraView+TakePhoto.kt
@@ -14,6 +14,7 @@ import com.facebook.react.bridge.WritableMap
 import com.mrousavy.camera.core.CameraSession
 import com.mrousavy.camera.core.InsufficientStorageError
 import com.mrousavy.camera.types.Flash
+import com.mrousavy.camera.types.Orientation
 import com.mrousavy.camera.types.QualityPrioritization
 import com.mrousavy.camera.utils.*
 import java.io.File
@@ -46,7 +47,7 @@ suspend fun CameraView.takePhoto(optionsMap: ReadableMap): WritableMap {
     enableShutterSound,
     enableAutoStabilization,
     enablePrecapture,
-    orientation
+    cameraSession.orientation
  
+private val matrix = Matrix()
 private fun writePhotoToFile(photo: CameraSession.CapturedPhoto, file: File) {
   val byteBuffer = photo.image.planes[0].buffer
-  if (photo.isMirrored) {
     val imageBytes = ByteArray(byteBuffer.remaining()).apply { byteBuffer.get(this) }
     val bitmap = BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.size)
-    val matrix = Matrix()
-    matrix.preScale(-1f, 1f)
-    val processedBitmap =
-      Bitmap.createBitmap(bitmap, 0, 0, bitmap.width, bitmap.height, matrix, false)
+    matrix.reset()
+    if(photo.isMirrored) {
+      matrix.preScale(-1f, 1f)
+    }
+    matrix.postRotate(90f)
+    val processedBitmap = Bitmap.createBitmap(bitmap, 0, 0, bitmap.width, bitmap.height, matrix, false)
     FileOutputStream(file).use { stream ->
       processedBitmap.compress(Bitmap.CompressFormat.JPEG, 100, stream)
     }
-  } else {
-    val channel = FileOutputStream(file).channel
-    channel.write(byteBuffer)
-    channel.close()
-  }
 }
 
 private suspend fun savePhotoToFile(
diff --git a/package/android/src/main/java/com/mrousavy/camera/types/Orientation.kt b/package/android/src/main/java/com/mrousavy/camera/types/Orientation.kt
index 64dc5bcf..86275519 100644
--- a/package/android/src/main/java/com/mrousavy/camera/types/Orientation.kt
+++ b/package/android/src/main/java/com/mrousavy/camera/types/Orientation.kt
@@ -3,17 +3,17 @@ package com.mrousavy.camera.types
 import com.mrousavy.camera.core.CameraDeviceDetails
 
 enum class Orientation(override val unionValue: String) : JSUnionValue {
-  PORTRAIT("portrait"),
   LANDSCAPE_RIGHT("landscape-right"),
-  PORTRAIT_UPSIDE_DOWN("portrait-upside-down"),
-  LANDSCAPE_LEFT("landscape-left");
+  PORTRAIT("portrait"),
+  LANDSCAPE_LEFT("landscape-left"),
+  PORTRAIT_UPSIDE_DOWN("portrait-upside-down");
 
   fun toDegrees(): Int =
     when (this) {
-      PORTRAIT -> 0
-      LANDSCAPE_LEFT -> 90
-      PORTRAIT_UPSIDE_DOWN -> 180
-      LANDSCAPE_RIGHT -> 270
+        LANDSCAPE_LEFT -> 0
+        PORTRAIT -> 90
+        LANDSCAPE_RIGHT -> 180
+        PORTRAIT_UPSIDE_DOWN -> 270
     }
 
   fun toSensorRelativeOrientation(deviceDetails: CameraDeviceDetails): Orientation {
@@ -26,7 +26,7 @@ enum class Orientation(override val unionValue: String) : JSUnionValue {
     }
 
     // Rotate sensor rotation by target rotation
-    val newRotationDegrees = (deviceDetails.sensorOrientation.toDegrees() + rotationDegrees + 360) % 360
+    val newRotationDegrees = (deviceDetails.sensorOrientation.toDegrees() + rotationDegrees + 270) % 360
 
     return fromRotationDegrees(newRotationDegrees)
   }
@@ -34,19 +34,19 @@ enum class Orientation(override val unionValue: String) : JSUnionValue {
   companion object : JSUnionValue.Companion<Orientation> {
     override fun fromUnionValue(unionValue: String?): Orientation =
       when (unionValue) {
+        "landscape-left" -> LANDSCAPE_LEFT
         "portrait" -> PORTRAIT
         "landscape-right" -> LANDSCAPE_RIGHT
         "portrait-upside-down" -> PORTRAIT_UPSIDE_DOWN
-        "landscape-left" -> LANDSCAPE_LEFT
         else -> PORTRAIT
       }
 
     fun fromRotationDegrees(rotationDegrees: Int): Orientation =
       when (rotationDegrees) {
-        in 45..135 -> LANDSCAPE_LEFT
-        in 135..225 -> PORTRAIT_UPSIDE_DOWN
-        in 225..315 -> LANDSCAPE_RIGHT
-        else -> PORTRAIT
+          in 45..135 -> PORTRAIT
+          in 135..225 -> LANDSCAPE_RIGHT
+          in 225..315 -> PORTRAIT_UPSIDE_DOWN
+          else -> LANDSCAPE_LEFT
       }
   }
 }

[simon-assistiq]

The problem is that the live visualisation is 90 deg inverted on iPad landscape mode. I'm just using the barcode scanner (I'm not saving any image)

Having the same issue as @matheusqfql. I'm a bit confused; is there no way to ensure the preview output of the camera on the screen is not rotated (I would need to rotate the preview by 90 degrees clockwise for it to be correct on the iPad in landscape) ? I'm migrating from 2.13.2 where the preview output was in the correct orientation, so I'm just wondering whether I'm just missing something new or some change was introduced. I don't need it to be able to respond to orientation changes, just want it to be always displayed in landscape on the iPad.

EDIT: Tried using orientation='landscape-right' on the <Camera> but it didn't seem to have any effect, which I guess is the point of this ticket. But the inline doc comment on that prop says that if this value isn't provided it should be using the device's orientation anyway.

I am also seeing this presumably related warning in the console:

[Orientation] BUG IN CLIENT OF UIKIT: Setting UIDevice.orientation is not supported. Please use UIWindowScene.requestGeometryUpdate(_:)

I have only tried iOS 16 so far, so I wonder if it would behave better on iOS 17.

[JesseLawler]

Does anyone have a good solution for this when applying to video?

Rotating an image is no big deal; I'm unsure of any reasonable solution for video...

[pvedula7]

I'm migrating from 2.13.2 where the preview output was in the correct orientation, so I'm just wondering whether I'm just missing something new or some change was introduced. I don't need it to be able to respond to orientation changes, just want it to be always displayed in landscape on the iPad.

Facing something somewhat similar where we are recording a video in standard portrait mode, but the frame.orientation always prints landscape-left. When attempting to overlay things onto the video with coordinates from frame processor, it's always off presumably due to some frame difference. Worked fine in V2 for us.

[mrousavy]

btw., I wrote an explanation on why rotation/orientation isn't as straight forward as it sounds here: #2807 (comment)

[mars-lan]

After reverting back to CameraX in 4.x, this doesn't seem to be an issue anymore.

[mrousavy]

This still is an issue - Frame Processors don't pass you a transformation matrix, and the preview view orientation cannot be frozen while outputs should rotate - that's how most Camera apps work.

Also, on iOS this still isn't implemented. For now, only portrait works perfectly.