feat: ✨ V3 ✨ #1466
Merged
feat: ✨ V3 ✨ #1466
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* chore: Upgrade Example to RN 0.71 * chore: Upgrade all libs * fix: Fix CameraRoll installation * Update Gradle Tools * fix: Fix buildscripts * Clean out build.gradle * fix: Fix Kotlin setup * fix: Move kotlin-android dependency to lib * Move `_setGlobalConsole` * Update gradle-wrapper.properties * Rebuild lockfiles * chore: Update build:gradle * Update StatusBarBlurBackground.tsx * Use Java 11 in Workflows * Update MediaPage.tsx * Add `google` repository to build.gradle * Double Java Heap size * Increase heap size * Alternative args * Update build.gradle
This comment was marked as off-topic.
This comment was marked as off-topic.
41 tasks
* Setup RN Worklets * Use RN Worklets on iOS * Fix console * Add `installFrameProcessorBindings()` function * Add `FrameProcessorPlugins` proxy (BREAKING CHANGE) * Clean up docs * Update FRAME_PROCESSORS.mdx * Use RN Worklets 0.2.5 * feat: Android build setup * Rewrite Android Frame Processor Part * Update CMakeLists.txt * fix: Add react-native-worklets Gradle dependency * Update Podfile.lock * fix build * gradle:7.4.1 * Init JSI Bindings in method on Android * Fix Folly flags * fix: Init `FrameProcessorRuntimeManager` later * fix: Wrap in `<GestureHandlerRootView>` * Refactor plugins * fix: Remove enableFrameProcessors * Install RN Worklets from current GH master * Update babel.config.js * Update CameraViewModule.kt * Update ImageProxyUtils.java * feat: Upgrade to Reanimated v3 * fix: Fix crash on Worklet init * Update RN Worklets to latest master * fix: Simplify FP Plugins Proxy
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
…1472) Before, Frame Processors ran on a separate Thread. After, Frame Processors run fully synchronous and always at the same FPS as the Camera. Two new functions have been introduced: * `runAtTargetFps(fps: number, func: () => void)`: Runs the given code as often as the given `fps`, effectively throttling it's calls. * `runAsync(frame: Frame, func: () => void)`: Runs the given function on a separate Thread for Frame Processing. A strong reference to the Frame is held as long as the function takes to execute. You can use `runAtTargetFps` to throttle calls to a specific API (e.g. if your Camera is running at 60 FPS, but you only want to run face detection at ~25 FPS, use `runAtTargetFps(25, ...)`.) You can use `runAsync` to run a heavy algorithm asynchronous, so that the Camera is not blocked while your algorithm runs. This is useful if your main sync processor draws something, and your async processor is doing some image analysis on the side. You can also combine both functions. Examples: ```js const frameProcessor = useFrameProcessor((frame) => { 'worklet' console.log("I'm running at 60 FPS!") }, []) ``` ```js const frameProcessor = useFrameProcessor((frame) => { 'worklet' console.log("I'm running at 60 FPS!") runAtTargetFps(10, () => { 'worklet' console.log("I'm running at 10 FPS!") }) }, []) ``` ```js const frameProcessor = useFrameProcessor((frame) => { 'worklet' console.log("I'm running at 60 FPS!") runAsync(frame, () => { 'worklet' console.log("I'm running on another Thread, I can block for longer!") }) }, []) ``` ```js const frameProcessor = useFrameProcessor((frame) => { 'worklet' console.log("I'm running at 60 FPS!") runAtTargetFps(10, () => { 'worklet' runAsync(frame, () => { 'worklet' console.log("I'm running on another Thread at 10 FPS, I can block for longer!") }) }) }, []) ```
* fix: Fix CI for "Build Android" * update versions * Update Gemfile.lock * format swift * fix: Fix swift lint * Update .swiftlint.yml * Use C++17 for lint * fix: Fix C++ lints
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as resolved.
This comment was marked as resolved.
* Create Shaders.ts * Add `previewType` and `enableFpsGraph` * Add RN Skia native dependency * Add Skia Preview View on iOS * Pass 1 * Update FrameHostObject.mm * Wrap Canvas * Lockfiles * fix: Fix stuff * chore: Upgrade RNWorklets * Add `previewType` to set the Preview * feat: Add Example * Update project.pbxproj * `enableFpsGraph` * Cache the `std::shared_ptr<FrameHostObject>` * Update CameraView+RecordVideo.swift * Update SkiaMetalCanvasProvider.mm * Android: Integrate Skia Dependency * fix: Use new Prefix * Add example for rendering shader * chore: Upgrade CameraX * Remove KTX * Enable `viewBinding` * Revert "Enable `viewBinding`" This reverts commit f2a603f. * Revert "chore: Upgrade CameraX" This reverts commit 8dc832c. * Remove unneeded `ProcessCameraProvider.getInstance()` call * fix: Add REA hotfix patch * fix: Fix FrameHostObject dead in runAsync * fix: Make `runAsync` run truly async by dropping new Frames while executing * chore: Upgrade RN Worklets to latest * chore: Upgrade RN Skia * Revert "Remove KTX" This reverts commit 253f586. * Make Skia optional in CMake * Fix import * Update CMakeLists.txt * Update build.gradle * Update CameraView.kt * Update CameraView.kt * Update CameraView.kt * Update Shaders.ts * Center Blur * chore: Upgrade RN Worklets * feat: Add `toByteArray()`, `orientation`, `isMirrored` and `timestamp` to `Frame` (#1487) * feat: Implement `orientation` and `isMirrored` on Frame * feat: Add `toArrayBuffer()` func * perf: Do faster buffer copy * feat: Implement `toArrayBuffer()` on Android * feat: Add `orientation` and `isMirrored` to Android * feat: Add `timestamp` to Frame * Update Frame.ts * Update JImageProxy.h * Update FrameHostObject.cpp * Update FrameHostObject.cpp * Update CameraPage.tsx * fix: Format Swift
Ahh, thank you!
Edited the comment :)
Yep I mainly focused on iOS for now, am currently looking more into Android. thanks for the logs, will investigate! |
Update 21.2.2023:I just released VisionCamera v3.0.0-rc.2, which includes the following features:
Try it: Things to test: all of the features above |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
…me`'s lifecycle (#1488) * fix: fix C++ lint * fix: attach `InvalidateCacheOnDestroy` to `jsi::Runtime`
|
Okay I had a mistake in |
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See #1376
Breaking Changes
runAsyncinside a Frame Processor__prefix, but rather stored directly in theFrameProcessorPluginsobject exported by react-native-vision-camera. (e.g. replace__scanQRCodes(frame)withFrameProcessorPlugins.scanQRCodes(frame))frameProcessorFpsno longer exists. UserunAtTargetFpsinside a Frame Processor to throttle some calls.onFrameProcessorPerformanceSuggestionAvailableno longer exists. Use the FPS display (enableFpsGraph={true}) to see how your Frame Processor performs over time. This is more in-line with how React Native works (Dev Tools / Perf Monitor)Progress
You can test the latest V3 release by creating a new RN project with RN 0.71 and installing VisionCamera + RNWorklets:
Things to test:
.tflitemodel!! ✨ (see this PR for more info, will be a separate library soon)frame.toArrayBuffer()to get the Frame's byte content in JSrunAtTargetFpsandrunAsyncin Frame ProcessorsOverall V3 is close to completion. I have a few things to do the coming days so not sure how much work I can put into this. If anyone wants to support the development of v3, I'd appreciate donations / sponsors: https://github.com/sponsors/mrousavy ❤️ :)
Related issues
features
preset#117...and then pretty much every Android issue lol
.. maybe also (not tested):
SIGSEGVon Android #1635[capture/inactive-source]The recording failed because the source becomes inactive and stops sending frames. One case is that if camera is closed due to lifecycle stopped, the active recording will be finalized with this error, and the output will be generated, containing the frames produced before camera closing. Attempting to start a new recording will be finalized immediately if the source remains inactive and no output will be generated. #959