Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
name: Tests

on:
push:
branches: [main, master]
pull_request:
branches: [main, master]

jobs:
typescript-tests:
name: TypeScript Tests
runs-on: self-hosted

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'

- name: Install dependencies
run: npm ci --legacy-peer-deps

- name: Run TypeScript tests
run: npm test

- name: Run TypeScript tests with coverage
run: npm run test:coverage
continue-on-error: true

android-tests:
name: Android Unit Tests
runs-on: self-hosted

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup Java
uses: actions/setup-java@v4
with:
distribution: 'temurin'
java-version: '17'

- name: Setup Android SDK
uses: android-actions/setup-android@v3

- name: Run Android unit tests
working-directory: android
run: ./gradlew test --no-daemon

# Uncomment when MacBook Air is set up as a self-hosted runner
# ios-tests:
# name: iOS Unit Tests
# runs-on: self-hosted-macos # Update with your macOS runner label
#
# steps:
# - name: Checkout
# uses: actions/checkout@v4
#
# - name: Run iOS tests
# working-directory: ios
# run: |
# xcodebuild test \
# -scheme VisionML \
# -destination 'platform=iOS Simulator,name=iPhone 15' \
# -only-testing:VisionMLTests \
# | xcpretty
245 changes: 245 additions & 0 deletions FEATURE_PARITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,245 @@
# Feature Parity: iOS vs Android

This document tracks the feature parity between iOS and Android implementations of react-native-vision-ml.

## Summary

| Category | iOS | Android | Status |
|----------|-----|---------|--------|
| Core ONNX Inference | ✅ | ✅ | **Full Parity** |
| Video Analysis | ✅ | ✅ | **Full Parity** |
| Vision/ML Kit | ✅ | ⚠️ | **Partial** |
| Sensitive Content Analysis | ✅ | ❌ | **iOS Only** |
| Live Activity | ✅ | ⚠️ | **Partial** |

---

## Detailed Feature Comparison

### Core ONNX Inference ✅ Full Parity

| Feature | iOS | Android | Notes |
|---------|-----|---------|-------|
| Model Loading | ✅ CoreML | ✅ NNAPI | Both use hardware acceleration |
| Image Decode | ✅ CGImage | ✅ Bitmap | HEIC/JPEG/PNG supported |
| NCHW Conversion | ✅ | ✅ | Identical tensor format |
| YOLO v8 Parsing | ✅ | ✅ | Same output format |
| NMS Algorithm | ✅ | ✅ | Identical implementation |
| Multiple Detectors | ✅ | ✅ | Concurrent instances supported |
| Resource Cleanup | ✅ | ✅ | dispose/disposeAll methods |

**Performance:**
- iOS: ~120-150ms per image (CoreML acceleration)
- Android: ~120-150ms per image (NNAPI acceleration)

### Video Analysis ✅ Full Parity

All 5 video scan modes are implemented identically:

| Mode | iOS | Android | Description |
|------|-----|---------|-------------|
| `quick_check` | ✅ | ✅ | 3 frames: start, middle, end |
| `sampled` | ✅ | ✅ | Regular interval sampling |
| `thorough` | ✅ | ✅ | Human detection + ONNX on human frames |
| `binary_search` | ✅ | ✅ | Adaptive region expansion |
| `full_short_circuit` | ✅ | ✅ | Progressive scan with early exit |

**Frame Extraction:**
- iOS: AVAssetImageGenerator
- Android: MediaMetadataRetriever.getFrameAtTime()

### Vision Framework / ML Kit ⚠️ Partial Parity

| Feature | iOS (Vision) | Android (ML Kit) | Status |
|---------|--------------|------------------|--------|
| Animal Detection | ✅ VNRecognizeAnimalsRequest | ✅ Image Labeling | **Parity** |
| Human Pose | ✅ VNDetectHumanBodyPoseRequest | ✅ Pose Detection | **Parity** |
| Face Detection | ✅ VNDetectFaceRectanglesRequest | ✅ Face Detection | **Parity** |
| Text Recognition | ✅ VNRecognizeTextRequest | ✅ Text Recognition | **Parity** |
| Scene Classification | ✅ VNClassifyImageRequest | ❌ Not Available | **Gap** |
| Rectangle Detection | ✅ VNDetectRectanglesRequest | ❌ Not Available | **Gap** |

#### Gap: Scene Classification
- **iOS**: VNClassifyImageRequest provides ~1000 scene categories
- **Android**: ML Kit doesn't have an equivalent API
- **Workaround**: Would require custom TensorFlow Lite model

#### Gap: Rectangle Detection
- **iOS**: VNDetectRectanglesRequest for screenshot/document detection
- **Android**: Not implemented
- **Impact**: `likelyScreenshot` field always false on Android

### Sensitive Content Analysis ❌ iOS Only

| Feature | iOS | Android | Notes |
|---------|-----|---------|-------|
| Single Image SCA | ✅ iOS 17+ | ❌ | Apple proprietary |
| Batch Image SCA | ✅ iOS 17+ | ❌ | Apple proprietary |
| Video SCA | ✅ iOS 17+ | ❌ | Apple proprietary |
| Batch Video SCA | ✅ iOS 17+ | ❌ | Apple proprietary |
| SCA Settings | ✅ | ❌ | Apple proprietary |

**Why No Android Support:**
Apple's Sensitive Content Analysis framework is a proprietary iOS 17+ feature with no Android equivalent. Google does not provide a similar on-device content analysis API.

**Workaround for Android:**
Use the ONNX-based detection which works on both platforms and provides detailed bounding boxes and confidence scores.

### Live Activity ⚠️ Partial Parity

| Feature | iOS | Android | Notes |
|---------|-----|---------|-------|
| Start Activity | ✅ Dynamic Island | ⚠️ Stub | Foreground service infrastructure ready |
| Update Progress | ✅ Real-time | ⚠️ Stub | Notification UI not implemented |
| End Activity | ✅ Final status | ⚠️ Stub | |

**Android Implementation Status:**
- Foreground service infrastructure is in place
- Notification progress UI is stubbed (always returns success but no visible UI)
- Full implementation requires custom notification layout

---

## API Compatibility

### Methods with Full Parity

```typescript
// These work identically on both platforms
createDetector(modelPath, classLabels, inputSize)
detect(detectorId, imageUri, options)
disposeDetector(detectorId)
disposeAllDetectors()
analyzeVideo(detectorId, assetId, options)
quickCheckVideo(detectorId, assetId, threshold)
analyzeAnimals(assetId)
analyzeHumanPose(assetId)
```

### Methods with Platform Differences

```typescript
// Works but may return partial results on Android
analyzeComprehensive(assetId)
// - Android: scenes=[], rectangles=0, likelyScreenshot=false

// iOS 17+ only - Android returns {available: false}
getSensitiveContentAnalysisStatus()
analyzeSensitiveContent(assetId)
batchAnalyzeSensitiveContent(assetIds)
analyzeVideoSensitiveContent(assetId)
batchAnalyzeVideosSensitiveContent(assetIds)

// iOS 16.1+ only - Android returns stub responses
isLiveActivityAvailable()
startVideoScanActivity(name, duration, mode)
updateVideoScanActivity(progress, phase, count, frames)
endVideoScanActivity(count, frames, isNSFW)
```

---

## Recommended Usage Patterns

### Cross-Platform Detection

```typescript
// Works on both iOS and Android
const detector = await createDetector(modelPath, labels, 320);
const result = await detect(detector.detectorId, imageUri, {
confidenceThreshold: 0.6,
iouThreshold: 0.45,
});
```

### Platform-Specific Features

```typescript
import { Platform } from 'react-native';

// Use SCA on iOS 17+, fall back to ONNX on Android
if (Platform.OS === 'ios') {
const status = await getSensitiveContentAnalysisStatus();
if (status.available && status.enabled) {
const result = await analyzeSensitiveContent(assetId);
if (result.isSensitive) {
// Run ONNX for detailed bounding boxes
const detailed = await detect(detectorId, imageUri);
}
}
} else {
// Android: Use ONNX directly
const result = await detect(detectorId, imageUri);
}
```

### Comprehensive Analysis

```typescript
const analysis = await analyzeComprehensive(assetId);

// Handle platform differences
const hasSceneData = Platform.OS === 'ios' && analysis.scenes?.length > 0;
const isScreenshot = Platform.OS === 'ios' && analysis.likelyScreenshot;
```

---

## Test Coverage

### TypeScript Tests (Jest)
- API integration tests
- NMS algorithm unit tests
- YOLO parser unit tests
- Platform parity verification

Run with: `npm test`

### Android Tests (JUnit)
- NMS algorithm tests
- YOLO parser tests
- Coordinate transformation tests

Run with: `./gradlew test` in android/

### iOS Tests (XCTest)
- NMS algorithm tests
- YOLO parser tests
- Coordinate transformation tests

Run with: Xcode Test Navigator

---

## Roadmap

### Planned Improvements

1. **Android Live Activity** (P1)
- Implement notification progress UI
- Match iOS Dynamic Island UX

2. **Android Scene Classification** (P2)
- Research custom TensorFlow Lite models
- Consider Google Cloud Vision API fallback

3. **Android Rectangle Detection** (P3)
- Implement ML Kit custom model integration
- Or use OpenCV-based detection

### Not Planned

- **Android SCA**: No equivalent API exists; use ONNX detection instead

---

## Version History

| Version | Changes |
|---------|---------|
| 1.0.5 | Video SCA fixes |
| 1.0.4 | Added video SCA support |
| 1.0.3 | Added batch SCA support |
| 1.0.2 | Full Android implementation |
| 1.0.1 | Initial Android stubs |
| 1.0.0 | Initial iOS-only release |
Loading
Loading