📚 Table of Contents

• 1. Team
• 2. Project Overview
• 3. Key Features
• 4. Core Technologies
• 5. Technical Challenges and Solutions
• 6. System Design
• 7. Project Structure
• 8. Tech Stack
• 9. Project Schedule Management
• 10. License

---

1. Team

🧑‍💼 Jinhyuk Jang @jinhyuk2me

• Implemented the Unity/Blender synthetic data pipeline
• Designed and built the real-time bird strike risk analysis AI system (BDS)
• Developed deep learning models for BDS
• Developed deep learning models for the IDS ground monitoring system
• Implemented RedWing ground-guidance assistance for pilots
• Built RedWing TTS warning and auto-response features

🧑‍💼 Jongmyung Kim @jongbob1918

• Project lead (documentation and schedule management)
• Built the ground-object detection AI system (IDS)
• Researched and implemented ground-object detection models
• Investigated and tested ArUco-based map coordinate mapping

🧑‍💼 Jiyeon Kim @heyjay1002

• Generated pose keypoints and synthetic datasets with Blender
• Built a custom YOLO pose model for fall detection
• Designed and implemented the Hawkeye ATC GUI
• Researched LLM/STT/TTS for the RedWing pilot AI service

🧑‍💼 Hyojin Park @Park-hyojin

• Led system design and backend
• Built and maintained the main server
• Designed and managed the database
• Defined system interfaces and communication architecture
• Designed the ArUco-based coordinate mapping logic

---

2. Project Overview

Major airports worldwide continue to report severe incidents such as bird strikes, Foreign Object Debris (FOD) accidents, and runway incursions. These incidents are usually the result of a combination of factors: high cognitive load for controllers and pilots, sensor limitations, and delayed information handoffs.

Case	Year	Root Cause
Muan Airport bird strike	2024	Lack of detection system
Concorde FOD accident	2000	Debris left on runway
Austin runway incursion	2023	Control error + situational awareness failure

FALCON was created to raise the safety and efficiency of flight operations and pursues three core values.

💡 FALCON's Core Values

• Real-time risk discovery
  Automatically detects threats humans may miss to eliminate blind spots

• Decision-support assistance
  Hand signal interpretation, risk assessment, and voice responses reduce cognitive load

• Immediate information delivery
  Provides risk information with GUI/TTS so crews can respond faster

---

3. Key Features

🛫 Air Traffic Controller AI Service: Hawkeye

• Ground hazard detection

  • CCTV-based video analytics
  • When birds, FOD, people, or vehicles are detected, show GUI popups and map markers
  • Continuously update risk levels and write logs
  • Watch video

• Ground fall detection

  • Recognizes civilians/workers who have fallen
  • Visualizes risk gauge (location, time, severity)
  • Provides visual summary to help decide if rescue is needed
  • Watch video

• Ground access control

  • Configure zone-based clearance levels (Level 1–3)
  • Detect and alert on access violations automatically
  • Reflect clearance updates on the GUI in real time
  • Watch video

---

✈️ Pilot AI Service: RedWing

• Flight risk alerts

  • Real-time TTS warnings for bird strikes and runway hazards
  • Connects video analytics with the risk assessment model
  • Watch video

• Risk inquiry auto-response

  • Voice query (STT) → LLM classification → Voice response (TTS)
  • Example: “Runway Alpha status?” → “Runway Alpha is CLEAR.”
  • Watch video

• Ground guidance assistance

  • Recognizes marshaller hand signals (stop, proceed, turn left/right) in CCTV footage
  • Converts the signal into spoken instructions for pilots
  • Watch video

---

4. Core Technologies

1) Simulation-based Training and Prediction

• Unity-based airport environment simulator (RunwaySim)
  • Models runways and surrounding infrastructure
  • Generates automated pipelines for training ground-object detection models

• Unity-based real-time bird strike risk simulator (BirdRiskSim)
  • Predicts bird positions from fixed CCTV footage
  • Calculates collision probability using relative distances and velocities between birds, aircraft, and air routes

2) Object Detection

Airport risks are divided into ground and aerial zones, and each zone uses a dedicated detection pipeline.

🧱 Ground Object Detection

• Detection classes: birds, FOD, people, wildlife, aircraft, and vehicles (6 total)

• Dataset composition:

  • Hybrid dataset of Unity-simulated imagery and miniature airport photos
  • Automated labeling pipeline using Polycam, Blender, and Unity-based 3D scans
  • Simulated diverse lighting, angles, and environments to increase variation
  • Automatically generate ~3,000 labeled images per hour

Bird

FOD

Wild animal

Vehicle

_{Automated labeling pipeline using Blender}

• Model architecture and training setup:

  • YOLOv8n-box (960×960 input, 150 epochs, batch size 8)
  • Dataset split: Train 69.4% / Validation 20.9% / Test 9.8%

• Post-processing classification:

  • Detect hi-vis (HV) vests with OpenCV to determine if a person is authorized
  • Use vehicle color features to distinguish work vehicles vs. general vehicles

Authorized / unauthorized personnel

Authorized / unauthorized vehicles

• Model performance (v0.3):

  • mAP@0.5: 0.9902
  • mAP@0.5:0.95: 0.9005
  • Precision: 0.9928
  • Recall: 0.9672

• Key improvements:

  • ~50% lighter and faster than the initial YOLOv11-seg model
  • Added negative samples to eliminate ArUco marker false positives

🛩️ Aerial Object Detection

Specialized YOLOv8-based model for detecting aerial threats such as birds. Powers the BDS (Bird Detection System) and provides the flight risk alert feature.

• Training data

  • Total epochs: 72, final learning rate: 0.000495
  • Framework: YOLOv8

• Performance summary

  Metric	Epoch 69 (best)	Epoch 72 (final)
  mAP@0.5	0.9455	0.9438
  mAP@0.5:0.95	0.8278	0.8342
  Precision	0.9850	0.9787
  Recall	0.8949	0.9031

---

3) Object Tracking

(1) Ground object tracking

• Uses the ByteTrack algorithm (Ultralytics built-in)
• Combines low-score detection with a Kalman Filter
• Meets both real-time and accuracy requirements

(2) Aerial object tracking

To predict and respond to aerial threats such as bird strikes, FALCON integrates triangulation-based localization, ByteTrack-based object tracking, and Unity-driven risk computation.

• 📌 Feasibility validation
  • Reconstructed the bird flock trajectory from the 2024 Muan Airport incident using nearby CCTV after the fact.
  • Confirmed that triangulation and tracking can deliver real-time bird-strike risk scores.

• 🌐 Simulation environment
  • Modeled actual airport terrain in Unity
  • Generated multiple weather and flight scenarios
  • Aircraft follow Bézier-curve paths and support multi-flight simulations

• 🛰️ CCTV-based bird localization (triangulation) and path tracking
  • Two synchronized CCTV feeds inside the Unity simulator
  • Detect 2D bird and aircraft positions in each view
  • Estimate 3D coordinates using triangulation
  • Track frame-to-frame positions with ByteTrack using the 3D data

Triangulation-based localization

ByteTrack trajectory tracking

• 🧠 Real-time bird strike risk scoring
  • Analyze relative distance, velocity, and direction of birds vs. aircraft
    Output qualitative risk levels (e.g., BR_MEDIUM)
  • Deliver warnings to controllers and pilots via GUI and voice interface

---

4) Pose Estimation

Combined static and dynamic pose estimation to precisely interpret marshaller gestures.

(1) Static pose estimation

• YOLOv8n-pose extracts 17 keypoints
• Trained on 683 Blender-generated synthetic images + real captures
• Detects falls by analyzing keypoint tilt

(2) Dynamic pose estimation

• Model

  • Temporal Convolutional Network (TCN)
  • Input: 17 joints × (x, y) coordinates → 34 features over 30 frames
  • Output classes: stop, forward, left, right

• Dataset

  • 3,984 sequences (train 80%, test 20%)
  • MediaPipe-based 17-joint coordinates

• Performance

  • Accuracy: 98.99%
  • Precision: 99.00%, Recall: 98.99%, F1-Score: 98.99%
  • Avg. confidence: 98.62%, Std: 6.64%

• Per-class metrics (test set)

  Gesture	Precision	Recall	F1-Score
  Stop	98.55%	99.51%	99.03%
  Forward	99.46%	97.87%	98.66%
  Left	98.57%	99.52%	99.04%
  Right	99.49%	98.98%	99.23%

---

5) Coordinate Mapping

• ArUco-based world coordinate mapping

  • Uses OpenCV perspectiveTransform()
  • Maps ArUco marker pixel centers to real-world coordinates
  • Achieves ±5 mm/pixel accuracy

• Object center correction

  • Converts detected bounding-box centers into real-world positions
  • Used to check zone intrusions and access violations

---

5. Technical Challenges and Solutions

📉 YOLO accuracy degradation

Problem

• Low detection accuracy during real-world tests

Original YOLO PR curve	Original real-world test

Solution

• Built a hybrid dataset combining real and synthetic data
• Retrained using YOLOv8n-box

Hybrid model PR curve	Hybrid model real-world test

---

🧍‍♂️ Pose keypoint detection errors

Problem

• Keypoints were inaccurate when subjects lay down or were upside down

Solution

• Generated 683 Blender-based synthetic poses
• Retrained YOLOv8n-pose → higher fall-detection accuracy

Previous pose model	Improved pose model

---

6. System Design

System Architecture

ER Diagram

---

7. Project Structure

FALCON/
├── src/
│   ├── systems/              # Core systems
│   │   ├── bds/              # Bird Detection System
│   │   └── ids/              # Intrusion Detection System
│   │
│   ├── simulation/          # Simulation
│   │   ├── bird_sim/        # Bird strike simulation
│   │   └── runway_sim/      # Runway simulation
│   │
│   ├── interfaces/          # User interfaces
│   │   ├── hawkeye/         # ATC GUI
│   │   └── redwing/         # Pilot GUI
│   │
│   ├── infrastructure/      # System infrastructure
│   │   └── server/          # Server code
│   │
│   ├── shared/              # Shared modules
│   │   └── utils/           # Utilities
│   │
│   └── tests/               # Tests
│       └── technical_test/  # Technical validation
│
├── docs/                    # Documentation
├── assets/                  # Assets
├── tools/                   # Tools
└── README.md                # Project overview

---

8. Tech Stack

Category	Technologies
ML / DL
GUI
Database
Networking / Communication
Analytics / Visualization
Simulation / Synthetic Data

---

9. Project Schedule Management

Managed the program with Confluence and Jira. Visualized task assignment, development progress, and issue tracking to keep collaboration on track.

---

10. License

This project is open-sourced under the Apache License 2.0.
See the LICENSE file for details.