d

.gitignore

@@ -104,3 +104,4 @@ Thumbs.db
 *.pkl
 *.joblib
 *.model
+.vercel

README.md

@@ -6,6 +6,59 @@
 
 This project serves both as a practical cybersecurity tool and as a demonstration of applying neural networks and AI in cybersecurity for academic purposes.
 
+---
+## 🚀 Quickstart (cloned repo)
+
+Prerequisites
+- Python 3.10+ (3.11/3.12/3.13 supported)
+- Node.js 18+ and npm
+
+1) Clone and enter the folder
+```bash
+git clone https://github.com/PraneeshRV/RedCalibur.git
+cd RedCalibur
+```
+
+2) Create and activate a virtual environment, install deps
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+python -m pip install -U pip setuptools wheel
+python -m pip install -r requirements.txt
+python -m pip install -r api/requirements.txt
+```
+
+3) Configure environment variables (optional but recommended)
+```bash
+cp .env.example .env
+# edit .env and add keys as needed: SHODAN_API_KEY, VIRUSTOTAL_API_KEY, GEMINI_API_KEY
+```
+
+4) Start the backend API (runs in background)
+```bash
+chmod +x scripts/*.sh
+./scripts/start_api.sh
+# Health check: http://127.0.0.1:8000/health
+```
+
+5) Start the frontend (development)
+```bash
+cd frontend
+npm install
+npm run dev
+# App: http://localhost:5173 (Vite may choose 5174 if 5173 is busy)
+```
+
+Stop the backend
+```bash
+./scripts/stop_api.sh
+```
+
+Optional: run tests
+```bash
+python -m pytest tests/
+```
+
 ---
 ## ⚔️ Features
 
@@ -32,7 +85,7 @@ RedCalibur integrates traditional red teaming techniques with modern AI, offerin
     * **Document Metadata Analysis**: Analyze metadata from PDF documents.
     * **Reverse Image Search**: Find where an image appears online (placeholder).
 * **Social Media Reconnaissance**:
-    * **Username Footprinting**: Gather information from social media handles (placeholder).
+  * **Username Footprinting**: Multi-platform probes via direct HTTP checks (no external CLI required).
     * **LinkedIn Scraping**: Scrape company and employee data (placeholder).
     * **Twitter OSINT**: Analyze user data and activity (placeholder).
 
@@ -122,16 +175,14 @@ pip install -r requirements.txt
 ## ⚙️ Configuration
 
 ### Environment Variables
-```bash
-# Required for full functionality
-export SHODAN_API_KEY="your_shodan_api_key"
-export OPENAI_API_KEY="your_openai_api_key"
-export ANTHROPIC_API_KEY="your_anthropic_api_key"
-
-# Optional
-export REDCALIBUR_OUTPUT_DIR="./reports"
-export REDCALIBUR_LOG_LEVEL="INFO"
-```
+Use `.env` at the project root (copy from `.env.example`). Keys are optional but enable richer results.
+
+Core keys used by the current API/UI
+- SHODAN_API_KEY: Enables Shodan enrichment on network scan
+- VIRUSTOTAL_API_KEY: Enables full URL malware scanning; without it, a basic URL health check is used
+- GEMINI_API_KEY: Enables AI summarization of recon data (Google Generative AI)
+
+Additional optional variables in `.env.example` are for future/extended tooling (e.g., Hunter.io, OpenAI/Anthropic); they are not required to run the local UI and core flows.
 
 ### Configuration Check
 ```bash
@@ -284,15 +335,22 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
 This repository includes a lightweight API server and a modern React dashboard.
 
 ### Backend API
-1) Copy `.env.example` to `.env` and fill your keys.
-2) Install and run:
+1) Copy `.env.example` to `.env` and fill keys as needed (optional).
+2) Install deps (see Quickstart above) and run:
 
+Dev (foreground, auto-reload):
 ```bash
-pip install -r requirements.txt
 python -m api.run
 ```
 
-API is served at http://localhost:8000 (health: `/health`).
+Background (recommended local service):
+```bash
+chmod +x scripts/*.sh
+./scripts/start_api.sh
+# Stop with: ./scripts/stop_api.sh
+```
+
+API default: http://127.0.0.1:8000 (health: `/health`).
 
 ### Frontend
 In a second terminal:
@@ -303,13 +361,37 @@ npm install
 npm run dev
 ```
 
-App is served at http://localhost:5173 and proxies `/api/*` to the backend.
+App is served at http://localhost:5173 (or 5174) and proxies `/api/*` to http://127.0.0.1:8000 in development.
 
 ### UI Highlights
 - Neon-red cyber theme, accessible contrast, responsive layout
 - Domain recon (WHOIS, DNS, subdomains, SSL), AI summary and basic risk score
 - Network scan with optional Shodan enrichment
-- Username lookup (Sherlock)
+- Username lookup (direct HTTP probes; no Sherlock dependency)
 - URL malware scan (VirusTotal)
 
-Note: Ensure external tools and keys are configured (Shodan, VirusTotal, Gemini, Sherlock).
+Note: Keys are optional. Shodan and VirusTotal enrich results when provided. Gemini powers AI summaries.
+
+### Optional: systemd service (Linux)
+If you prefer the backend to run automatically on boot:
+
+1) Review and, if needed, edit `deploy/systemd/redcalibur.service` paths.
+2) Install and enable the service:
+```bash
+sudo cp deploy/systemd/redcalibur.service /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable redcalibur
+sudo systemctl start redcalibur
+```
+3) Check status and logs:
+```bash
+systemctl status redcalibur --no-pager
+journalctl -u redcalibur -n 100 --no-pager
+```
+
+### Troubleshooting
+- Backend health check: http://127.0.0.1:8000/health
+- Frontend can’t reach API in dev: ensure the backend is running and Vite proxy is active
+- Port conflicts: Vite will choose another port (5174) if 5173 is busy; change API port with `--port` in `api/run.py` if needed
+- Scripts say venv missing: create the venv at `.venv` and install requirements (see Quickstart)
+- Missing keys: the app will still run, but some features return reduced data

api/app.py

@@ -10,9 +10,12 @@
 from redcalibur.osint.domain_infrastructure.subdomain_discovery import discover_subdomains
 from redcalibur.osint.domain_infrastructure.port_scanning import perform_port_scan
 from redcalibur.osint.domain_infrastructure.ssl_tls_details import get_ssl_details
+from concurrent.futures import ThreadPoolExecutor, as_completed, TimeoutError
+import time
 from redcalibur.osint.network_threat_intel.shodan_integration import perform_shodan_scan
 from redcalibur.osint.user_identity.username_lookup import lookup_username
 from redcalibur.osint.virustotal_integration import scan_url
+from redcalibur.osint.url_health_check import basic_url_health
 from redcalibur.osint.ai_enhanced.recon_summarizer import summarize_recon_data
 from redcalibur.osint.ai_enhanced.risk_scoring import calculate_risk_score
 
@@ -65,44 +68,77 @@ def domain_recon(req: DomainRequest):
     results: Dict[str, Any] = {"target": req.target, "timestamp": datetime.now().isoformat()}
     errors: Dict[str, Any] = {}
 
-    # WHOIS
-    if req.whois or req.all:
-        try:
-            results["whois"] = perform_whois_lookup(req.target)
-        except Exception as e:
-            logger.error(f"WHOIS error: {e}")
-            errors["whois"] = str(e)
-
-    # DNS
-    if req.dns or req.all:
-        try:
-            results["dns"] = enumerate_dns_records(req.target)
-        except Exception as e:
-            logger.error(f"DNS error: {e}")
-            errors["dns"] = str(e)
+    tasks = []
 
-    # Subdomains
-    if req.subdomains or req.all:
+    def wrap(name, fn, *args, **kwargs):
         try:
-            results["subdomains"] = discover_subdomains(req.target, config.SUBDOMAIN_WORDLIST)
+            res = fn(*args, **kwargs)
+            results[name] = res
         except Exception as e:
-            logger.error(f"Subdomain discovery error: {e}")
-            errors["subdomains"] = str(e)
+            logger.error(f"{name} error: {e}")
+            errors[name] = str(e)
 
-    # SSL
-    if req.ssl or req.all:
-        try:
-            results["ssl"] = get_ssl_details(req.target)
-        except Exception as e:
-            logger.error(f"SSL details error: {e}")
-            errors["ssl"] = str(e)
+    ex = ThreadPoolExecutor(max_workers=4)
+    try:
+        fut_to_name: Dict[Any, str] = {}
+        if req.whois or req.all:
+            fut = ex.submit(wrap, "whois", perform_whois_lookup, req.target)
+            tasks.append(fut)
+            fut_to_name[fut] = "whois"
+        if req.dns or req.all:
+            fut = ex.submit(wrap, "dns", enumerate_dns_records, req.target)
+            tasks.append(fut)
+            fut_to_name[fut] = "dns"
+        if req.subdomains or req.all:
+            fut = ex.submit(wrap, "subdomains", discover_subdomains, req.target, config.SUBDOMAIN_WORDLIST)
+            tasks.append(fut)
+            fut_to_name[fut] = "subdomains"
+        if req.ssl or req.all:
+            fut = ex.submit(wrap, "ssl", get_ssl_details, req.target)
+            tasks.append(fut)
+            fut_to_name[fut] = "ssl"
+
+        deadline = time.time() + 12.0
+        pending = set(tasks)
+        while time.time() < deadline and pending:
+            try:
+                for fut in as_completed(list(pending), timeout=0.5):
+                    try:
+                        fut.result()
+                    except Exception as e:
+                        logger.error(f"Task future error: {e}")
+                    finally:
+                        pending.discard(fut)
+            except TimeoutError:
+                # No futures completed in this slice; loop again until deadline
+                pass
+
+        # Mark any remaining as timed out by task name
+        for fut in list(pending):
+            name = fut_to_name.get(fut, "task")
+            errors[name] = "timeout"
+    finally:
+        # Do not wait for running tasks; return partial results promptly
+        ex.shutdown(wait=False, cancel_futures=True)
 
     # AI enrichment (optional)
     if req.all:
         try:
             import json
             raw_data = json.dumps(results, indent=2, default=str)
-            results["ai_summary"] = summarize_recon_data(raw_data[:2000])
+            # Run summarization with a strict timeout to avoid long external calls
+            ex_ai = ThreadPoolExecutor(max_workers=1)
+            try:
+                fut = ex_ai.submit(summarize_recon_data, raw_data[:2000])
+                try:
+                    results["ai_summary"] = fut.result(timeout=6.0)
+                except TimeoutError:
+                    errors["ai"] = "timeout"
+            finally:
+                # Don't wait for the AI call to finish if it's slow
+                ex_ai.shutdown(wait=False, cancel_futures=True)
+
+            # Risk scoring is local and fast
             features = [
                 len(results.get("subdomains", [])),
                 1 if isinstance(results.get("ssl", {}), dict) and "error" not in results.get("ssl", {}) else 0,
@@ -152,17 +188,23 @@ def username(req: UsernameRequest):
 @app.post("/urlscan")
 def urlscan(req: URLScanRequest):
     try:
-        if not config.VIRUSTOTAL_API_KEY:
-            raise HTTPException(status_code=400, detail="VIRUSTOTAL_API_KEY not configured")
-        data = scan_url(config.VIRUSTOTAL_API_KEY, req.url)
-        if not data:
-            raise HTTPException(status_code=502, detail="VirusTotal returned no data")
-        return data
-    except HTTPException:
-        raise
+        # Run the scan in a short-lived thread with a strict timeout
+        def runner():
+            if not config.VIRUSTOTAL_API_KEY:
+                return {"note": "VIRUSTOTAL_API_KEY not configured", "health": basic_url_health(req.url)}
+            return scan_url(config.VIRUSTOTAL_API_KEY, req.url) or {"error": "no_data"}
+
+        with ThreadPoolExecutor(max_workers=1) as ex_url:
+            fut = ex_url.submit(runner)
+            try:
+                return fut.result(timeout=10.0)
+            except TimeoutError:
+                return {"error": "timeout"}
+            finally:
+                ex_url.shutdown(wait=False, cancel_futures=True)
     except Exception as e:
         logger.error(f"URL scan failed: {e}")
-        raise HTTPException(status_code=500, detail=str(e))
+        return {"error": str(e)}
 
 
 class SummarizeRequest(BaseModel):

api/smoke_domain.py

@@ -0,0 +1,18 @@
+import json
+import requests
+
+def main():
+    url = "http://127.0.0.1:8000/domain"
+    payload = {"target": "example.com", "all": True}
+    try:
+        r = requests.post(url, json=payload, timeout=20)
+        print("Status:", r.status_code)
+        try:
+            print(json.dumps(r.json(), indent=2))
+        except Exception:
+            print(r.text)
+    except Exception as e:
+        print("Request failed:", e)
+
+if __name__ == "__main__":
+    main()

deploy/systemd/redcalibur.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=RedCalibur FastAPI Service
+After=network.target
+
+[Service]
+User=akvnn
+Group=akvnn
+WorkingDirectory=/home/akvnn/Downloads/aa/RedCalibur
+Environment=PYTHONUNBUFFERED=1
+# Optional: if you want to load .env via systemd rather than python-dotenv
+#EnvironmentFile=/home/akvnn/Downloads/aa/RedCalibur/.env
+ExecStart=/home/akvnn/Downloads/aa/RedCalibur/.venv/bin/python -m uvicorn api.app:app --host 0.0.0.0 --port 8000 --workers 2
+Restart=always
+RestartSec=3
+TimeoutStopSec=15
+
+[Install]
+WantedBy=multi-user.target