feat: add integration with vision camera v5 by NorbertKlockiewicz · Pull Request #810 · software-mansion/react-native-executorch

NorbertKlockiewicz · 2026-02-16T09:41:46Z

Description

Example of how to use the API with vision camera v5: https://gist.github.com/NorbertKlockiewicz/5d62915d16955979c029303591912d6a

For now this PR is in experimental phase so when reviewing please focus on the user facing API + implementation of ObjectDetection both on TypeScript and Native Side. The JSI part of the code isn't production ready yet and requires refactor + comprehensive comments

Introduces a breaking change?

Yes
No

Type of change

Bug fix (change which fixes an issue)
New feature (change which adds functionality)
Documentation update (improves or adds clarity to existing documentation)
Other (chores, tests, code style improvements etc.)

Tested on

iOS
Android

Testing instructions

Screenshots

Related issues

Checklist

I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have updated the documentation accordingly
My changes generate no new warnings

Additional notes

msluszniak

Also handle there:

msluszniak · 2026-02-20T11:04:26Z

apps/computer-vision/app/object_detection/index.tsx

+      // Create a simple 320x320 test image (all zeros - black image)
+      // In a real scenario, you would load actual image pixel data here
+      const width = 320;
+      const height = 320;
+      const channels = 3; // RGB
+
+      // Create a black image (you can replace this with actual pixel data)
+      const rgbData = new Uint8Array(width * height * channels);
+
+      // Optionally, add some test pattern (e.g., white square in center)
+      for (let y = 100; y < 220; y++) {
+        for (let x = 100; x < 220; x++) {
+          const idx = (y * width + x) * 3;
+          rgbData[idx + 0] = 255; // R
+          rgbData[idx + 1] = 255; // G
+          rgbData[idx + 2] = 255; // B
+        }
+      }
+
+      const pixelData: PixelData = {
+        dataPtr: rgbData,
+        sizes: [height, width, channels],
+        scalarType: ScalarType.BYTE,
+      };
+
+      console.log('Running forward with hardcoded pixel data...', {
+        sizes: pixelData.sizes,
+        dataSize: pixelData.dataPtr.byteLength,
+      });
+
+      // Run inference using unified forward() API
+      const output = await ssdLite.forward(pixelData, 0.3);
+      console.log('Pixel data result:', output.length, 'detections');
+      setResults(output);
+    } catch (e) {
+      console.error('Error in runForwardPixels:', e);


I think all the comments from here as the code is self-describing.

msluszniak · 2026-02-20T11:07:15Z

...ages/react-native-executorch/common/rnexecutorch/models/object_detection/ObjectDetection.cpp

+  // Get target size from model input shape
+  const std::vector<int32_t> tensorDims = getAllInputShapes()[0];
+  cv::Size tensorSize = cv::Size(tensorDims[tensorDims.size() - 1],
+                                 tensorDims[tensorDims.size() - 2]);
+
+  cv::Mat rgb;
+
+  // Convert RGBA/BGRA to RGB if needed (for VisionCamera frames)
+  if (frame.channels() == 4) {
+// Platform-specific color conversion:
+// iOS uses BGRA format, Android uses RGBA format
+#ifdef __APPLE__
+    // iOS: BGRA → RGB
+    cv::cvtColor(frame, rgb, cv::COLOR_BGRA2RGB);
+#else
+    // Android: RGBA → RGB
+    cv::cvtColor(frame, rgb, cv::COLOR_RGBA2RGB);
+#endif
+  } else if (frame.channels() == 3) {
+    // Already RGB


Again these comments are not needed, only comment Only resize if dimensions don't match seems to be valid one.

msluszniak · 2026-02-20T11:07:43Z

...ages/react-native-executorch/common/rnexecutorch/models/object_detection/ObjectDetection.cpp

-  auto [inputTensor, originalSize] =
-      image_processing::readImageToTensor(imageSource, getAllInputShapes()[0]);
+ObjectDetection::runInference(cv::Mat image, double detectionThreshold) {
+  std::lock_guard<std::mutex> lock(inference_mutex_);


Scoped lock is superior to lock_guard, and since we use c++ >= 17, use only scoped_lock in such situations.

Suggested change

std::lock_guard<std::mutex> lock(inference_mutex_);

std::scoped_lock<std::mutex> lock(inference_mutex_);

msluszniak · 2026-02-20T11:12:18Z

...ages/react-native-executorch/common/rnexecutorch/models/object_detection/ObjectDetection.cpp

+  // Store original size for postprocessing
+  cv::Size originalSize = image.size();
+
+  // Preprocess the image using model-specific preprocessing
+  cv::Mat preprocessed = preprocessFrame(image);
+
+  // Create tensor and run inference


These comments are redundant.

msluszniak · 2026-02-20T11:13:03Z

...ages/react-native-executorch/common/rnexecutorch/models/object_detection/ObjectDetection.cpp

-} // namespace rnexecutorch::models::object_detection
+
+std::vector<types::Detection>
+ObjectDetection::generateFromString(std::string imageSource,


Why you use passing string by copy and not const reference? If because of the fact that this function is called via JSI and const ref fails here, please resolve this comment.

msluszniak · 2026-02-20T11:44:54Z

packages/react-native-executorch/src/hooks/useModule.ts

        await moduleInstance.load(model, setDownloadProgress);
        setIsReady(true);
+
+        // Extract runOnFrame worklet from VisionModule if available


Suggested change

// Extract runOnFrame worklet from VisionModule if available

msluszniak · 2026-02-20T11:47:12Z

packages/react-native-executorch/src/modules/computer_vision/VisionModule.ts

+    // Extract pure JSI function reference (runs on JS thread)
+    const nativeGenerateFromFrame = this.nativeModule.generateFromFrame;
+
+    // Return worklet that captures ONLY the JSI function


Suggested change

// Extract pure JSI function reference (runs on JS thread)

const nativeGenerateFromFrame = this.nativeModule.generateFromFrame;

// Return worklet that captures ONLY the JSI function

const nativeGenerateFromFrame = this.nativeModule.generateFromFrame;

msluszniak · 2026-02-20T11:48:09Z

packages/react-native-executorch/src/modules/computer_vision/VisionModule.ts

+
+    // Type detection and routing
+    if (typeof input === 'string') {
+      // String path → generateFromString()


Suggested change

// String path → generateFromString()

msluszniak · 2026-02-20T11:48:16Z

packages/react-native-executorch/src/modules/computer_vision/VisionModule.ts

+      'scalarType' in input &&
+      input.scalarType === ScalarType.BYTE
+    ) {
+      // Pixel data → generateFromPixels()


Suggested change

// Pixel data → generateFromPixels()

msluszniak · 2026-02-20T11:48:40Z

packages/react-native-executorch/src/modules/computer_vision/VisionModule.ts

+      typeof input === 'object' &&
+      'dataPtr' in input &&
+      input.dataPtr instanceof Uint8Array &&
+      'sizes' in input &&
+      Array.isArray(input.sizes) &&
+      input.sizes.length === 3 &&
+      'scalarType' in input &&
+      input.scalarType === ScalarType.BYTE


Huuuh, abstract it into smaller function ;p

msluszniak

Also handle there:

NorbertKlockiewicz changed the title ~~@nk/vision camera~~ feat: add integration with vision camera v5 Feb 18, 2026

NorbertKlockiewicz force-pushed the @nk/vision-camera branch from ee215f9 to 96f2c14 Compare February 19, 2026 16:33

NorbertKlockiewicz added 10 commits February 19, 2026 21:47

fix: correct frame data extraction

7e8c81a

feat: frame extractor for zero-copy approach

4338acc

chore: num minSdkVersion to 26

4c9f64f

feat: unify frame extraction and preprocessing

aea9c26

feat: remove unused bindJSIMethods

e4e0f95

feat: initial version of vision model API

3527d01

refactor: errors, logs, unnecessary comments, use existing TensorPtr

f552710

fix: change Frame import in BaseModule

62cfb01

feat: use TensorPtrish type for Pixel data input

ff53e57

refactor: add or remove empty lines

f3e17e2

NorbertKlockiewicz force-pushed the @nk/vision-camera branch from 96f2c14 to f3e17e2 Compare February 19, 2026 21:03

NorbertKlockiewicz added 2 commits February 19, 2026 22:34

fix: errors after rebase

09589b0

fix: remove redundant preprocessing step

3e15a25

msluszniak requested changes Feb 20, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

feat: add integration with vision camera v5#810

feat: add integration with vision camera v5#810
NorbertKlockiewicz wants to merge 12 commits intomainfrom
@nk/vision-camera

NorbertKlockiewicz commented Feb 16, 2026 •

edited

Loading

Uh oh!

msluszniak left a comment

Uh oh!

msluszniak Feb 20, 2026

Uh oh!

msluszniak Feb 20, 2026

Uh oh!

msluszniak Feb 20, 2026

Uh oh!

msluszniak Feb 20, 2026

Uh oh!

msluszniak Feb 20, 2026

Uh oh!

msluszniak Feb 20, 2026

Uh oh!

msluszniak Feb 20, 2026

Uh oh!

msluszniak Feb 20, 2026

Uh oh!

msluszniak Feb 20, 2026

Uh oh!

msluszniak Feb 20, 2026

Uh oh!

msluszniak left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	std::lock_guard<std::mutex> lock(inference_mutex_);
	std::scoped_lock<std::mutex> lock(inference_mutex_);

Comments

Conversation

NorbertKlockiewicz commented Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Introduces a breaking change?

Type of change

Tested on

Testing instructions

Screenshots

Related issues

Checklist

Additional notes

Uh oh!

msluszniak left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

msluszniak left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

NorbertKlockiewicz commented Feb 16, 2026 •

edited

Loading