Implement Dark Web Crawler with knowledge base integration

[Copilot]

Adds a web crawler for automated content discovery and knowledge base population, completing the Dark RAG pipeline.

Implementation

• DarkCrawler: Configurable crawler with rate limiting, robots.txt compliance, depth/page limits, and timeout handling
• HTMLContentExtractor: Extracts text, titles, and links from HTML while filtering scripts/styles
• URL Filtering: Domain-based (create_domain_filter) and regex pattern-based (create_pattern_filter) filters with composability
• Knowledge Base Integration: crawl_and_store() method for direct storage with content length filtering

Usage

from dark_crawler import DarkCrawler, create_domain_filter
from dark_rag import DarkKnowledgeBase, DarkRAG

# Crawl and populate knowledge base
crawler = DarkCrawler(delay=2.0, max_depth=2, max_pages=100)
kb = DarkKnowledgeBase()

domain_filter = create_domain_filter(['docs.example.com'])
stats = crawler.crawl_and_store(
    seed_urls=['https://docs.example.com'],
    knowledge_base=kb,
    filter_func=domain_filter
)

# Query with augmented context
dark_rag = DarkRAG(knowledge_base=kb)
result = dark_rag.generate("How do I configure the API?")

Testing

• 10 test cases covering extraction, filtering, integration, and edge cases
• No network dependencies (uses mocks)
• CodeQL scan: 0 vulnerabilities

Documentation

• DARK_CRAWLER.md: Architecture, usage patterns, best practices
• crawler_examples.py: 8 runnable examples demonstrating configurations
• CRAWLER_IMPLEMENTATION_SUMMARY.md: Design decisions and implementation details

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Dark Web Crawler</issue_title>
<issue_description></issue_description>

<agent_instructions>An optional prompt.</agent_instructions>

Comments on the Issue (you are @copilot in this section)

• Fixes Dark Web Crawler #17

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.