🐛 Camera is not aligned properly when using `useSkiaFrameProcessor` #2762

karventhan13 · 2024-04-20T09:06:51Z

What's happening?

🐛 [^4.0.0-beta.16] - Camera is not aligned properly when using useSkiaFrameProcessor but it's work with

Reproduceable Code

return (
    <SafeAreaView style={styles.container}>
      {!hasPermission && <Text>No Camera Permission.</Text>}
      {hasPermission && device != null && (
        <Camera
          style={StyleSheet.absoluteFill}
          device={device}
          isActive={true}
          photo={true}
          video={true}
          format={format}
          onError={(err)=> { console.log("err **",err)}}
          fps={fps}
          enableFpsGraph={true}
          frameProcessor={frameProcessor}  
          videoHdr={format.supportsVideoHdr}
          photoHdr={format.supportsPhotoHdr}
          enableZoomGesture={false}
        />
      )}
    </SafeAreaView>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
  },
});

Relevant log output

none

Camera Device

none

Device

vivo y21

VisionCamera Version

[^4.0.0-beta.16]

Can you reproduce this issue in the VisionCamera Example app?

I didn't try (⚠️ your issue might get ignored & closed if you don't try this)

Additional information

I am using Expo
I have enabled Frame Processors (react-native-worklets-core)
I have read the Troubleshooting Guide
I agree to follow this project's Code of Conduct
I searched for similar issues in this repository and found none.

mrousavy · 2024-04-21T21:47:32Z

I will never understand why people ignore the fact that I need log output. I will write a bot in the future to auto-close issues that don't have valid log output.

mrousavy · 2024-04-21T22:03:27Z

Maybe play around with useSkiaFrameProcessor hook, specifically the render() function in there. If you can figure out what's wrong, please submit a PR.
I'm on vacation this week

Space6188 · 2024-04-22T13:40:50Z

This happened when you using resizeMode={'contain'}

Try to delete this from Camera props

mrousavy · 2024-04-22T13:57:43Z

@Space6188 does it properly display without the resizeMode?

mrousavy · 2024-04-22T13:58:00Z

I can also reproduce this when using resizeMode="cover"...

mrousavy · 2024-04-22T14:00:27Z

Try playing around with the code from useSkiaFrameProcessor:

react-native-vision-camera/package/src/skia/useSkiaFrameProcessor.ts

Lines 165 to 175 in 3ff02cc

    
           'worklet' 
        
           // rotate canvas to properly account for Frame orientation 
        
           canvas.save() 
        
           const rotation = getRotationDegrees(frame.orientation) 
        
           canvas.rotate(rotation, frame.width / 2, frame.height / 2) 
        
           // render the Camera Frame to the Canvas 
        
           if (paint != null) canvas.drawImage(image, 0, 0, paint) 
        
           else canvas.drawImage(image, 0, 0) 
        
           // restore transforms/rotations again 
        
           canvas.restore()

Maybe we're doing something wrong.

Also, the ImageProxy (which is what the native Frame type holds) contains a getSensorToBufferTransformMatrix() method - maybe we can use this Matrix to transform the Frame safely to view dimensions? cc @rodgomesc if you wanna look into this.

rodgomesc · 2024-04-22T14:02:54Z

I will never understand why people ignore the fact that I need log output. I will write a bot in the future to auto-close issues that don't have valid log output.

then you will have zero issues in vc 😅

rodgomesc · 2024-04-22T14:04:37Z

i tried all resizeMode possibilities, also all fit possibilities, the problem is not there

mrousavy · 2024-04-22T14:05:50Z

then it's probably the sensor to buffer transform matrix.

Space6188 · 2024-04-22T15:18:09Z

@Space6188правильно ли он отображается без resizeMode?

No,my bad,sorry, it is not working too

rodgomesc · 2024-04-29T01:51:55Z

i think that i found a fix for this, should send a pr some time tomorrow

AndyRC6 · 2024-04-29T17:51:37Z

In case it helps - I am able to reproduce this by passing in the format prop with a custom videoResolution to the camera. Removing the format prop seems to fix the issue.

Edit: Actually, removing format only fixes the white bar that appears to the left of the preview, and moves it down to the bottom. Which is less noticeable as it looks like that's just where the camera preview ends. But in my case, the preview should take up the entire screen, but doesn't when using the Skia frame processor.

rodgomesc · 2024-04-30T02:53:52Z

it seems that canvas.drawImage draws the image with a default clip and matrix that are messed by the canvas rotation somehow? that’s confusing since if we use canvas.drawImageRect with the same src/dst props it draws correctly! in my tests it seem to solve the issue

diff --git a/package/src/skia/useSkiaFrameProcessor.ts b/package/src/skia/useSkiaFrameProcessor.ts
index 3feb336..fd69ca1 100644
--- a/package/src/skia/useSkiaFrameProcessor.ts
+++ b/package/src/skia/useSkiaFrameProcessor.ts
@@ -6,8 +6,34 @@ import { VisionCameraProxy, wrapFrameProcessorWithRefCounting } from '../FramePr
 import type { DrawableFrameProcessor } from '../types/CameraProps'
 import type { ISharedValue, IWorkletNativeApi } from 'react-native-worklets-core'
 import { WorkletsProxy } from '../dependencies/WorkletsProxy'
-import type { SkCanvas, SkPaint, SkImage, SkSurface } from '@shopify/react-native-skia'
+import { type SkCanvas, type SkPaint, type SkImage, type SkSurface, ColorType, AlphaType, Skia } from '@shopify/react-native-skia'
 import { SkiaProxy } from '../dependencies/SkiaProxy'
+import { InteractionManager } from 'react-native'
+
+// Mocked Frame Data for testings
+const mockedImageWidth = 1920
+const mockedImageHeight = 1080
+
+const pixels = new Uint8Array(mockedImageWidth * mockedImageHeight * 4)
+pixels.fill(255)
+let i = 0
+for (let x = 0; x < mockedImageWidth; x++) {
+  for (let y = 0; y < mockedImageHeight; y++) {
+    pixels[i++] = (x * y) % 255
+  }
+}
+
+const data = Skia.Data.fromBytes(pixels)
+const image = Skia.Image.MakeImage(
+  {
+    width: mockedImageWidth,
+    height: mockedImageHeight,
+    alphaType: AlphaType.Opaque,
+    colorType: ColorType.RGBA_8888,
+  },
+  data,
+  mockedImageWidth * 4,
+)!
 
 /**
  * Represents a Camera Frame that can be directly drawn to using Skia.
@@ -155,7 +181,7 @@ export function createSkiaFrameProcessor(
 
     // Convert Frame to SkImage/Texture
     const nativeBuffer = (frame as FrameInternal).getNativeBuffer()
-    const image = Skia.Image.MakeImageFromNativeBuffer(nativeBuffer.pointer)
+    // const image = Skia.Image.MakeImageFromNativeBuffer(nativeBuffer.pointer)
 
     return new Proxy(frame as DrawableFrame, {
       get: (_, property: keyof DrawableFrame) => {
@@ -165,11 +191,36 @@ export function createSkiaFrameProcessor(
             'worklet'
             // rotate canvas to properly account for Frame orientation
             canvas.save()
-            const rotation = getRotationDegrees(frame.orientation)
-            canvas.rotate(rotation, frame.width / 2, frame.height / 2)
-            // render the Camera Frame to the Canvas
-            if (paint != null) canvas.drawImage(image, 0, 0, paint)
-            else canvas.drawImage(image, 0, 0)
+
+            const rotationAngle = 0 //getRotationDegrees('portrait')
+            const { width: frameWidth, height: frameHeight } = getPortraitSize(frame)
+
+            // center of the canvas
+            const centerX = frameWidth / 2
+            const centerY = frameHeight / 2
+
+            const currentPaint = paint ?? Skia.Paint()
+
+            // only draw the rect by 40% of the frame size so we can vizualize the whole thing to debug :)
+            const reductionFactor = 0.4
+            const rectWidth = frameWidth * reductionFactor
+            const rectHeight = frameHeight * reductionFactor
+
+            // apply rotation around the center of the canvas
+            canvas.rotate(rotationAngle, centerX, centerY)
+
+            const rectX = centerX - rectWidth / 2
+            const rectY = centerY - rectHeight / 2
+
+            const srcRect = Skia.XYWHRect(0, 0, image.width(), image.height())
+            // Define the destination rectangle on the canvas
+            const dstRect = Skia.XYWHRect(rectX, rectY, rectWidth, rectHeight)
+            // Draw the image on the canvas
+
+            canvas.drawImageRect(image, srcRect, dstRect, currentPaint)
 
             // restore transforms/rotations again
             canvas.restore()
@@ -185,7 +236,7 @@ export function createSkiaFrameProcessor(
           return () => {
             'worklet'
             // dispose the Frame and the SkImage/Texture
-            image.dispose()
+            // image.dispose() // we do not dispose our mocked image :)
             nativeBuffer.delete()
           }
         }

Note

I'm doing some tricks to print the preview at 40% of its original size for better debugging However, at the end of the day, this code just simplifies to:

const srcRect = Skia.XYWHRect(0, 0, frameWidth, frameHeight);
const dstRect = Skia.XYWHRect(0, 0, frameWidth, frameHeight);
canvas.drawImageRect(image, srcRect, dstRect, currentPaint);

portrait 0deg	landscape-left 90deg
portrait-upside-down 180deg	landscape-right 270deg

rodgomesc · 2024-04-30T03:02:34Z

Oh wait, this solves the misalignment of the frame inside the canvas; however, it causes another issue: the image buffer is now rotated 90 degrees clockwise inside a (0 deg portrait) preview for me. I think this is expected since we are lacking orientation support for Android, @mrousavy?

mrousavy · 2024-04-30T07:04:38Z

I am applying the orientation inside the render(..) function, so actually it is expected that the Preview looks correct.

rodgomesc · 2024-04-30T10:04:05Z

I am applying the orientation inside the render(..) function, so actually it is expected that the Preview looks correct.

i mean, frame.orientation will always returns landscape-right since orientation support not being complete on android, and in render you are just doing getRotationDegrees(frame.orientation)
so i can't visualize the full picture here, of how did you got the image buffers always in portrait without a counter-clockwise rotation of -90deg in the current code that's running in render() {...}

react-native-vision-camera/package/android/src/main/java/com/mrousavy/camera/frameprocessors/Frame.java

Lines 89 to 93 in a53166b

    
           public Orientation getOrientation() throws FrameInvalidError { 
        
               assertIsValid(); 
        
               int degrees = imageProxy.getImageInfo().getRotationDegrees(); 
        
               return Orientation.Companion.fromRotationDegrees(degrees); 
        
           }

mrousavy · 2024-04-30T13:15:58Z

i mean, frame.orientation will always returns landscape-right since orientation support not being complete on android,

no, frame.orientation is an exception here - this is the sensor relative orientation of the ImageProxy, so this is correct as it is.

pweglik · 2024-05-08T08:07:02Z

Any news on this? I'm also experiencing this misplacement. I observed that it depends on format and video resolution:

Without passing format props to Camera:

Default format got from useCasmeraFormat():

  const format = useCameraFormat(device, []);
  console.log(format);

Log:

{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 2160, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 3840}

const format = useCameraFormat(device, [
    { videoResolution: { width: 600, height: 400 } },
  ]);
  console.log(format);

Log:

{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 480, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 640}

const format = useCameraFormat(device, [
    { videoResolution: { width: 1200, height: 800 } },
  ]);
  console.log(format);

Log:

{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 720, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 1280}

![2024-05-08 10 01 42]()

So in case of lower resolutions (default or set manually) only bottom part is cut off, while in higher resolutions entire frame is misplacement. Maybe that somehow helps you

OleksiiZdaly · 2024-05-14T08:51:09Z

Є новини з цього приводу? Я також відчуваю це неправильне розташування. Я помітив, що це залежить від формату та роздільної здатності відео:

Без передачі параметрів формату в камеру:

2. Формат за замовчуванням отримано з `useCasmeraFormat()`:
  const format = useCameraFormat(device, []);
  console.log(format);
Журнал:
{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 2160, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 3840}
const format = useCameraFormat(device, [
    { videoResolution: { width: 600, height: 400 } },
  ]);
  console.log(format);
Журнал:
{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 480, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 640}
const format = useCameraFormat(device, [
    { videoResolution: { width: 1200, height: 800 } },
  ]);
  console.log(format);
Журнал:
{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 720, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 1280}
Таким чином, у разі нижчої роздільної здатності (за замовчуванням або встановленої вручну) обрізається лише нижня частина, тоді як у вищій роздільній здатності зміщується весь кадр. Можливо, це якось тобі допоможе

For some reason, the format parameters don't change anything for me, the behavior is the same as yours in the second example. And the same problem with useSkiaFrameProcessor. On the emulator, the screen is completely black, and on real devices, part of it is. Another strange thing is that the frame itself processes frames rotated 90 degrees, I noticed this when my model started drawing rectangle around object on a black screen turned to the horizontal position. In IOS all ok.

mrousavy · 2024-05-21T15:48:44Z

Another strange thing is that the frame itself processes frames rotated 90 degrees

That's because of how Cameras work. Sensors are in 90deg orientation. See frame.orientation.

karventhan13 added the 🐛 bug Something isn't working label Apr 20, 2024

This comment was marked as spam.

Sign in to view

mrousavy changed the title ~~🐛 [^4.0.0-beta.16] - Camera is not aligned properly when using useSkiaFrameProcessor~~ 🐛 Camera is not aligned properly when using useSkiaFrameProcessor Apr 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Camera is not aligned properly when using `useSkiaFrameProcessor` #2762

🐛 Camera is not aligned properly when using `useSkiaFrameProcessor` #2762

karventhan13 commented Apr 20, 2024

This comment was marked as spam.

mrousavy commented Apr 21, 2024

mrousavy commented Apr 21, 2024

Space6188 commented Apr 22, 2024

mrousavy commented Apr 22, 2024

mrousavy commented Apr 22, 2024

mrousavy commented Apr 22, 2024

rodgomesc commented Apr 22, 2024

rodgomesc commented Apr 22, 2024

mrousavy commented Apr 22, 2024

Space6188 commented Apr 22, 2024

rodgomesc commented Apr 29, 2024

AndyRC6 commented Apr 29, 2024 •

edited

rodgomesc commented Apr 30, 2024 •

edited

rodgomesc commented Apr 30, 2024 •

edited

mrousavy commented Apr 30, 2024

rodgomesc commented Apr 30, 2024 •

edited

mrousavy commented Apr 30, 2024

pweglik commented May 8, 2024

OleksiiZdaly commented May 14, 2024

mrousavy commented May 21, 2024

🐛 Camera is not aligned properly when using useSkiaFrameProcessor #2762

🐛 Camera is not aligned properly when using useSkiaFrameProcessor #2762

Comments

karventhan13 commented Apr 20, 2024

What's happening?

Reproduceable Code

Relevant log output

Camera Device

Device

VisionCamera Version

Can you reproduce this issue in the VisionCamera Example app?

Additional information

This comment was marked as spam.

mrousavy commented Apr 21, 2024

mrousavy commented Apr 21, 2024

Space6188 commented Apr 22, 2024

mrousavy commented Apr 22, 2024

mrousavy commented Apr 22, 2024

mrousavy commented Apr 22, 2024

rodgomesc commented Apr 22, 2024

rodgomesc commented Apr 22, 2024

mrousavy commented Apr 22, 2024

Space6188 commented Apr 22, 2024

rodgomesc commented Apr 29, 2024

AndyRC6 commented Apr 29, 2024 • edited

rodgomesc commented Apr 30, 2024 • edited

rodgomesc commented Apr 30, 2024 • edited

mrousavy commented Apr 30, 2024

rodgomesc commented Apr 30, 2024 • edited

mrousavy commented Apr 30, 2024

pweglik commented May 8, 2024

OleksiiZdaly commented May 14, 2024

mrousavy commented May 21, 2024

🐛 Camera is not aligned properly when using `useSkiaFrameProcessor` #2762

🐛 Camera is not aligned properly when using `useSkiaFrameProcessor` #2762

AndyRC6 commented Apr 29, 2024 •

edited

rodgomesc commented Apr 30, 2024 •

edited

rodgomesc commented Apr 30, 2024 •

edited

rodgomesc commented Apr 30, 2024 •

edited