🐛 iOS pause audio/video output incorrect #2790

xHeinrich · 2024-04-24T03:45:04Z

What's happening?

When taking a video and pausing/resuming the video on iOS, the audio and video are chopped into different segments rather than one continuous video. For example, I start recording and say

one two three

pause recording and say four five six

resume recording and say seven eight nine

stop recording

Example video output:

broken-video.mp4

Reproduceable Code

Full repo with a minimal reproduction: https://github.com/xHeinrich/vision-camera-reproduction

const camera = useRef<Camera>(null)

    let device = useCameraDevice('back', {
        physicalDevices: [
            'ultra-wide-angle-camera',
            'wide-angle-camera',
            'telephoto-camera'
        ]
    })
 
    const [targetFps, setTargetFps] = useState(30)

    let filters = [
        {fps: targetFps},
        {videoStabilizationMode: 'auto'},
        {
            videoResolution: {
                width: 1280,
                height: 720
            },
        },
        {
            photoResolution: {
                width: 1280,
                height: 720
            },
        }
    ];

    const format = useCameraFormat(device, filters)
    const [torchOn, setTorchOn] = useState('off')
    const onError = useCallback((error: any) => {
        console.error(error)
    }, [])
                      
                   <Camera device={device}
                                zoom={device.neutralZoom}
                                ref={camera}
                                format={format}
                                enableZoomGesture={true}
                                exposure={0}
                                style={StyleSheet.absoluteFill}
                                isActive={isActive}
                                torch={torchOn}
                                orientation={'portrait'}
                                audio={micPermissionState.hasPermission}
                                photo={permissionState.hasPermission}
                                video={permissionState.hasPermission}
                                onError={onError}
                        />


            camera.current!.takePhoto({
                qualityPrioritization: 'speed',
                enableShutterSound: false
            }).then((file: PhotoFile) => {
            }).catch((error) => {
                reject(error)
            })

// sleep 3 seconds

        camera.current.startRecording({
            flash: 'on',
            fileType: "mp4",
            videoCodec: "h264",
            videoBitRate: 5, // 5mbps as target, affected by target fps
            onRecordingFinished: (video: VideoFile) => {
            },
            onRecordingError: (error) => console.error(error)
        })


// sleep 3 seconds
        await camera.current.pauseRecording()

// sleep 3 seconds


        await camera.current.resumeRecording()

// sleep 3 seconds

       await camera.current.stopRecording()

Relevant log output

13:23:26.080: [info] :camera_with_flash: VisionCamera.didSetProps(_:): Updating 27 props: [onInitialized, cameraId, position, enableBufferCompression, preview, onStarted, onCodeScanned, collapsable, top, right, isActive, video, onViewReady, onError, onStopped, enableFrameProcessor, format, orientation, left, bottom, audio, enableZoomGesture, exposure, torch, photo, onShutter, zoom]
13:23:26.081: [info] :camera_with_flash: VisionCamera.configure(_:): configure { ... }: Waiting for lock...
13:23:26.082: [info] :camera_with_flash: VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: true, outputsChanged: true, videoStabilizationChanged: true, orientationChanged: true, formatChanged: true, sidePropsChanged: true, torchChanged: true, zoomChanged: true, exposureChanged: true, audioSessionChanged: true, locationChanged: true)
13:23:26.082: [info] :camera_with_flash: VisionCamera.configureDevice(configuration:): Configuring Input Device...
13:23:26.082: [info] :camera_with_flash: VisionCamera.configureDevice(configuration:): Configuring Camera com.apple.avfoundation.avcapturedevice.built-in_video:7...
13:23:26.086: [info] :camera_with_flash: VisionCamera.configureDevice(configuration:): Successfully configured Input Device!
13:23:26.086: [info] :camera_with_flash: VisionCamera.configureOutputs(configuration:): Configuring Outputs...
13:23:26.086: [info] :camera_with_flash: VisionCamera.configureOutputs(configuration:): Adding Photo output...
13:23:26.088: [info] :camera_with_flash: VisionCamera.configureOutputs(configuration:): Adding Video Data output...
13:23:26.088: [info] :camera_with_flash: VisionCamera.configureOutputs(configuration:): Successfully configured all outputs!
13:23:26.089: [info] :camera_with_flash: VisionCamera.configureFormat(configuration:device:): Configuring Format (2112x1188 | 1280x720@60.0 (ISO: 34.0..3264.0))...
13:23:26.089: [info] :camera_with_flash: VisionCamera.configureFormat(configuration:device:): Successfully configured Format!
13:23:26.090: [info] :camera_with_flash: VisionCamera.getPixelFormat(for:): Available Pixel Formats: ["420v", "420f", "BGRA", "&8v0", "-8v0", "&8f0", "-8f0", "&BGA", "-BGA"], finding best match... (pixelFormat="yuv", enableHdr={false}, enableBufferCompression={true})
13:23:26.090: [info] :camera_with_flash: VisionCamera.getPixelFormat(for:): Using PixelFormat: -8f0...
13:23:26.485: [info] :camera_with_flash: VisionCamera.onCameraStarted(): Camera started!
13:23:26.485: [info] :camera_with_flash: VisionCamera.onSessionInitialized(): Camera initialized!
13:23:26.486: [info] :camera_with_flash: VisionCamera.configure(_:): Beginning AudioSession configuration...
13:23:26.486: [info] :camera_with_flash: VisionCamera.configureAudioSession(configuration:): Configuring Audio Session...
13:23:26.486: [info] :camera_with_flash: VisionCamera.configureAudioSession(configuration:): Adding Audio input...
13:23:26.487: [info] :camera_with_flash: VisionCamera.configure(_:): Beginning Location Output configuration...
13:23:26.490: [info] :camera_with_flash: VisionCamera.configureAudioSession(configuration:): Adding Audio Data output...
13:23:26.491: [info] :camera_with_flash: VisionCamera.configure(_:): Committed AudioSession configuration!
13:23:26.495: [info] :camera_with_flash: VisionCamera.configure(_:): Finished Location Output configuration!
13:23:44.113: [info] :camera_with_flash: VisionCamera.takePhoto(options:promise:): Capturing photo...
13:23:47.175: [info] :camera_with_flash: VisionCamera.startRecording(options:onVideoRecorded:onError:): Starting Video recording...
13:23:47.177: [info] :camera_with_flash: VisionCamera.startRecording(options:onVideoRecorded:onError:): Will record to temporary file: /private/var/mobile/Containers/Data/Application/0F78673F-DB67-4F14-8017-D356A019D118/tmp/ReactNative/01121C78-1A01-4C95-9FFF-38CC93C40AF1.mp4
13:23:47.186: [info] :camera_with_flash: VisionCamera.startRecording(options:onVideoRecorded:onError:): Enabling Audio for Recording...
13:23:47.186: [info] :camera_with_flash: VisionCamera.activateAudioSession(): Activating Audio Session...
13:23:47.194: [info] :camera_with_flash: VisionCamera.initializeAudioWriter(withSettings:format:): Initializing Audio AssetWriter with settings: ["AVSampleRateKey": 48000, "AVNumberOfChannelsKey": 1, "AVFormatIDKey": 1633772320]
13:23:47.194: [info] :camera_with_flash: VisionCamera.updateCategory(_:mode:options:): Changing AVAudioSession category from AVAudioSessionCategoryPlayAndRecord -> AVAudioSessionCategoryPlayAndRecord
13:23:47.357: [info] :camera_with_flash: VisionCamera.updateCategory(_:mode:options:): AVAudioSession category changed!
13:23:47.362: [info] :camera_with_flash: VisionCamera.didSetProps(_:): Updating 1 props: [torch]
13:23:47.362: [info] :camera_with_flash: VisionCamera.configure(_:): configure { ... }: Waiting for lock...
13:23:47.773: [info] :camera_with_flash: VisionCamera.activateAudioSession(): Audio Session activated!
13:23:47.785: [info] :camera_with_flash: VisionCamera.initializeAudioWriter(withSettings:format:): Initialized Audio AssetWriter.
13:23:47.795: [info] :camera_with_flash: VisionCamera.recommendedVideoSettings(forOptions:): Using codec AVVideoCodecType(_rawValue: avc1)...
13:23:47.795: [info] :camera_with_flash: VisionCamera.recommendedVideoSettings(forOptions:): Setting Video Bit-Rate from 14358528.0 bps to 5000000.0 bps...
13:23:47.795: [info] :camera_with_flash: VisionCamera.initializeVideoWriter(withSettings:): Initializing Video AssetWriter with settings: ["AVVideoCompressionPropertiesKey": ["AverageNonDroppableFrameRate": 30, "Priority": 80, "RealTime": 1, "ExpectedFrameRate": 60, "MaxAllowedFrameQP": 41, "H264EntropyMode": CABAC, "MaxKeyFrameIntervalDuration": 1, "AverageBitRate": 5000000, "AllowFrameReordering": 0, "MinAllowedFrameQP": 15, "QuantizationScalingMatrixPreset": 3, "ProfileLevel": H264_High_AutoLevel], "AVVideoWidthKey": 720, "AVVideoHeightKey": 1280, "AVVideoCodecKey": avc1]
13:23:47.819: [info] :camera_with_flash: VisionCamera.initializeVideoWriter(withSettings:): Initialized Video AssetWriter.
13:23:47.819: [info] :camera_with_flash: VisionCamera.start(clock:): Starting Asset Writer(s)...
13:23:48.441: [info] :camera_with_flash: VisionCamera.start(clock:): Asset Writer(s) started!
13:23:48.442: [info] :camera_with_flash: VisionCamera.start(clock:): Started RecordingSession at time: 127794.095554541
13:23:48.442: [info] :camera_with_flash: VisionCamera.startRecording(options:onVideoRecorded:onError:): RecordingSesssion started in 1266.5345ms!
13:23:48.443: [info] :camera_with_flash: VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: false, formatChanged: false, sidePropsChanged: false, torchChanged: true, zoomChanged: false, exposureChanged: false, audioSessionChanged: false, locationChanged: false)
13:24:00.905: [info] :camera_with_flash: VisionCamera.stop(clock:): Requesting stop at 127806.559768166 seconds for AssetWriter with status "writing"...
13:24:00.988: [info] :camera_with_flash: VisionCamera.appendBuffer(_:clock:type:): Successfully appended last audio Buffer (at 127806.56072916667 seconds), finishing RecordingSession...
13:24:00.988: [info] :camera_with_flash: VisionCamera.finish(): Stopping AssetWriter with status "writing"...
13:24:01.008: [info] :camera_with_flash: VisionCamera.startRecording(options:onVideoRecorded:onError:): RecordingSession finished with status completed.
13:24:01.008: [info] :camera_with_flash: VisionCamera.deactivateAudioSession(): Deactivating Audio Session...
13:24:01.013: [info] :camera_with_flash: VisionCamera.deactivateAudioSession(): Audio Session deactivated!
13:24:01.034: [info] :camera_with_flash: VisionCamera.didSetProps(_:): Updating 1 props: [torch]
13:24:01.034: [info] :camera_with_flash: VisionCamera.configure(_:): configure { ... }: Waiting for lock...
13:24:01.034: [info] :camera_with_flash: VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: false, formatChanged: false, sidePropsChanged: false, torchChanged: true, zoomChanged: false, exposureChanged: false, audioSessionChanged: false, locationChanged: false)

Camera Device

{
 "id": "com.apple.avfoundation.avcapturedevice.built-in_video:7",
 "formats": [],
 "hasFlash": true,
 "name": "Back Triple Camera",
 "minExposure": -8,
 "neutralZoom": 2,
 "physicalDevices": [
  "ultra-wide-angle-camera",
  "wide-angle-camera",
  "telephoto-camera"
 ],
 "supportsFocus": true,
 "supportsRawCapture": false,
 "isMultiCam": true,
 "minZoom": 1,
 "minFocusDistance": 2,
 "maxZoom": 61.875,
 "maxExposure": 8,
 "supportsLowLightBoost": false,
 "sensorOrientation": "landscape-right",
 "position": "back",
 "hardwareLevel": "full",
 "hasTorch": true
}

Device

iPhone 13 Pro

VisionCamera Version

4.0.0

Can you reproduce this issue in the VisionCamera Example app?

Yes, I can reproduce the same issue in the Example app here

Additional information

I am using Expo
I have enabled Frame Processors (react-native-worklets-core)
I have read the Troubleshooting Guide
I agree to follow this project's Code of Conduct
I searched for similar issues in this repository and found none.

thanhtungkhtn · 2024-05-07T10:08:28Z

Any update on this? facing similar issue

"react": "18.2.0",
"react-native": "0.74.1",
"react-native-vision-camera": "^4.0.3"

xHeinrich · 2024-05-08T00:20:45Z

Think it must be some timing issue with the asset writer but I don't know enough swift to find actual issue.

xHeinrich added the 🐛 bug Something isn't working label Apr 24, 2024

xHeinrich changed the title ~~🐛 iOS Pause Output Incorrect~~ 🐛 iOS pause audio/video output incorrect Apr 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 iOS pause audio/video output incorrect #2790

🐛 iOS pause audio/video output incorrect #2790

xHeinrich commented Apr 24, 2024 •

edited

thanhtungkhtn commented May 7, 2024

xHeinrich commented May 8, 2024

🐛 iOS pause audio/video output incorrect #2790

🐛 iOS pause audio/video output incorrect #2790

Comments

xHeinrich commented Apr 24, 2024 • edited

What's happening?

Reproduceable Code

Relevant log output

Camera Device

Device

VisionCamera Version

Can you reproduce this issue in the VisionCamera Example app?

Additional information

thanhtungkhtn commented May 7, 2024

xHeinrich commented May 8, 2024

xHeinrich commented Apr 24, 2024 •

edited