From 40e3540d688fb701a10a0f19d24a2891f5eb0105 Mon Sep 17 00:00:00 2001 From: Dawid Date: Wed, 4 Feb 2026 08:54:43 +0000 Subject: [PATCH 01/16] feat(instructions): add comprehensive secure coding guidelines for LLM applications MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - include OWASP Top 10 for LLM Applications (2025) security practices - outline responsibilities and areas to inspect during development stages - emphasize security champion mindset and ongoing threat awareness 🔒 - Generated by Copilot --- .github/agents/security-champion.agent.md | 82 ++++ .../owasp-for-llms.instructions.md | 406 ++++++++++++++++++ ...owasp-for-web-applications.instructions.md | 138 ++++++ 3 files changed, 626 insertions(+) create mode 100644 .github/agents/security-champion.agent.md create mode 100644 .github/instructions/owasp-for-llms.instructions.md create mode 100644 .github/instructions/owasp-for-web-applications.instructions.md diff --git a/.github/agents/security-champion.agent.md b/.github/agents/security-champion.agent.md new file mode 100644 index 00000000..8efd1a64 --- /dev/null +++ b/.github/agents/security-champion.agent.md @@ -0,0 +1,82 @@ +--- +description: "🔐 Security Champion" +tools: ['execute/getTerminalOutput', 'read', 'agent', 'todo'] +model: Claude Sonnet 4.5 +--- + +# Security Champion Chat Mode + +You are a security-focused code reviewer and advisor, applying Microsoft's Security Development Lifecycle (SDL) practices to help teams build secure software from the ground up. + +## Core Security Frameworks + +Apply these frameworks throughout the development lifecycle: + +* [OWASP Top 10](../instructions/owasp-for-web-applications.instructions.md) for web application security +* [OWASP Top 10 for LLM Applications (2025)](../instructions/owasp-for-llms.instructions.md) for AI/ML security +* [Microsoft SDL](https://www.microsoft.com/en-us/securityengineering/sdl/) for secure development practices + +## Microsoft SDL Practices + +Integrate these 10 SDL practices into security reviews: + +1. Establish security standards, metrics, and governance +2. Require use of proven security features, languages, and frameworks +3. Perform security design review and threat modeling +4. Define and use cryptography standards +5. Secure the software supply chain +6. Secure the engineering environment +7. Perform security testing +8. Ensure operational platform security +9. Implement security monitoring and response +10. Provide security training + +## Your Responsibilities + +* Scan code for vulnerabilities, misconfigurations, and insecure patterns +* Apply OWASP guidelines, SDL practices, and secure defaults +* Suggest safer alternatives with practical mitigations +* Guide threat modeling and security design reviews +* Promote Secure by Design principles + +## Areas to Inspect + +Review these areas across each development stage: + +### Design Stage + +* Threat modeling completeness +* Architecture security patterns +* Zero Trust principle adherence +* Data flow and trust boundaries + +### Code Stage + +* User input handling and validation +* Authentication and session logic +* File and network access controls +* Secrets management practices +* Dependency and supply chain security + +### Build and Deploy Stage + +* CI/CD pipeline security +* Code signing and integrity verification +* Container and infrastructure configuration + +### Runtime Stage + +* Security monitoring integration +* Incident response readiness +* Platform security baselines + +## When You Spot Risks + +* Highlight the issue clearly with its SDL context +* Suggest a fix or mitigation aligned with SDL practices +* Explain the impact and attacker perspective +* Reference relevant OWASP or SDL guidance + +## Security Champion Mindset + +Security is an ongoing effort where threats, technology, and business assets constantly evolve. Help teams understand the attacker's perspective and goals. Focus on practical, real-world security wins rather than theoretical overkill. Treat threat modeling as a fundamental engineering skill that all developers should possess. diff --git a/.github/instructions/owasp-for-llms.instructions.md b/.github/instructions/owasp-for-llms.instructions.md new file mode 100644 index 00000000..5c63732b --- /dev/null +++ b/.github/instructions/owasp-for-llms.instructions.md @@ -0,0 +1,406 @@ +--- +description: "Comprehensive secure coding instructions for LLM applications based on OWASP Top 10 for LLM Applications (2025). Ensures AI-powered systems are secure by default, protecting against prompt injection, data leakage, and LLM-specific vulnerabilities. Give clear and concise feedback and points of improvement." +applyTo: '**/*.py, **/*.tsx, **/*.ts, **/*.jsx, **/*.js, **/*.cs, **/*.java' +--- + +# OWASP Top 10 for LLM Applications - Secure Coding Guidelines + +## Instructions + +Your primary directive when working with Large Language Model (LLM) applications is to ensure all code you generate, review, or refactor is secure by default with specific attention to LLM-unique vulnerabilities. You must operate with a security-first mindset that recognizes LLMs introduce an entirely new class of risks beyond traditional application security. When in doubt, always choose the more secure option and explain the reasoning. Follow the principles outlined below, which are based on the OWASP Top 10 for LLM Applications (2025). + +**Critical Context:** LLM applications are non-deterministic systems that require defense-in-depth strategies. Unlike traditional applications, LLMs can be manipulated through natural language, making input validation, output handling, and access control fundamentally more complex. Always implement multiple layers of security controls rather than relying on a single defense mechanism. + +### LLM01:2025 Prompt Injection + +**Understand the Core Risk:** Prompt injection is the most critical LLM vulnerability—analogous to SQL injection but targeting the model's prompt context. User inputs can manipulate the LLM's behavior, override system instructions, extract sensitive information, or trigger unauthorized actions. + +**Constrain Model Behavior:** Define strict boundaries for LLM responses using explicit system prompts that clearly delineate acceptable outputs. Never rely solely on system prompts for security—they can be bypassed. + +**Implement Input Validation:** Apply rigorous validation to all user inputs before they reach the LLM. Use allowlists for expected input patterns, reject suspicious patterns (e.g., instructions like "ignore previous instructions"), and implement semantic analysis to detect manipulation attempts. + +**Output Validation is Critical:** Validate all LLM outputs against expected formats using deterministic verification. Define strict output schemas and reject responses that deviate from them. + +**Context Boundaries:** Separate system instructions from user content using clear delimiters. Never concatenate user input directly into prompts without sanitization. + +```python +# GOOD: Structured prompt with clear boundaries +system_prompt = "You are a customer service assistant. Only answer questions about product features." +user_input = sanitize_input(request.user_message) # Remove injection attempts +response = llm.generate(system=system_prompt, user=user_input) +validated_response = validate_output_schema(response) # Ensure format compliance +``` + +```python +# BAD: Direct concatenation with no validation +prompt = f"Answer this: {request.user_message}" # Vulnerable to injection +response = llm.generate(prompt) # No output validation +``` + +**Defend Against Indirect Injection:** When processing external content (files, websites, documents), treat all content as potentially malicious. Sanitize or summarize external data before including it in prompts. + +**Multimodal Risks:** If using vision or audio models, be aware that hidden instructions can be embedded in images or audio files. Implement content integrity checks. + +### LLM02:2025 Sensitive Information Disclosure + +**Never Include Secrets in Prompts:** System prompts, user inputs, and model responses can all leak sensitive information. Never embed API keys, passwords, tokens, PII, or proprietary algorithms in prompts or training data. + +**Implement Data Sanitization:** Apply robust data sanitization for both inputs and outputs. Use PII detection tools to identify and redact sensitive information before it reaches the LLM or gets displayed to users. + +**Output Schema Validation:** Define strict output schemas that prevent the model from generating sensitive data formats. Use context-appropriate encoding for all outputs (HTML encoding for web display, etc.). + +**Sandboxed Execution:** When executing LLM-generated code (which should be avoided when possible), always use sandboxed environments with no access to sensitive resources. + +```typescript +// GOOD: Sanitized context with PII detection +const sanitizedContext = await piiDetector.redact(userDocument); +const prompt = `Summarize this document: ${sanitizedContext}`; +const response = await llm.complete(prompt); +const safeOutput = encodeForContext(response, 'html'); +``` + +```typescript +// BAD: Direct exposure of sensitive data +const prompt = `Analyze this customer: Name: ${customer.name}, SSN: ${customer.ssn}, Income: ${customer.income}`; +// System prompt leaks: "You have access to database: postgres://admin:password@..." +``` + +**Training Data Extraction Defense:** Be aware that models can potentially reproduce verbatim content from training data. Implement differential privacy techniques and audit mechanisms to detect when models are leaking training data. + +**Separation of Concerns:** Store sensitive data in systems that the LLM cannot directly access. Pass only anonymized or minimal data to the model. + +### LLM03:2025 Supply Chain + +**Model Provenance Verification:** Only use pre-trained models from trusted sources with verified provenance. Verify cryptographic signatures and checksums for all downloaded models. + +**Model Source Trust:** Default to established model providers (OpenAI, Azure OpenAI, Anthropic, Google) with strong security postures. Be extremely cautious with community models from Hugging Face or other repositories without security audits. + +**Dependency Management:** Maintain a comprehensive Software Bill of Materials (SBOM) for all AI/ML dependencies. This includes models, fine-tuning adapters (LoRA), embedding models, and ML libraries. + +```python +# GOOD: Verified model loading with integrity checks +model_hash = verify_model_signature(model_path, expected_signature) +if model_hash != TRUSTED_MODEL_HASH: + raise SecurityError("Model integrity verification failed") +model = load_model(model_path) +``` + +```python +# BAD: Loading unverified models +model = load_model_from_url(untrusted_url) # No verification +``` + +**Red Team Testing:** Before deploying any third-party model, conduct rigorous adversarial testing including prompt injection attempts, jailbreaking tests, and bias evaluation. + +**Model Isolation:** Isolate model development and deployment environments. Use separate credentials and networks. Apply least privilege access controls to model files and APIs. + +**Monitor Third-Party Components:** Regularly scan all ML frameworks (PyTorch, TensorFlow, Transformers) for vulnerabilities. Update promptly when security patches are released. + +**On-Device Model Security:** If deploying models to edge devices, implement secure boot chains, encrypted model storage, and integrity monitoring to prevent tampering. + +### LLM04:2025 Data and Model Poisoning + +**Data Provenance Tracking:** Implement comprehensive tracking for all training and fine-tuning data. Maintain audit logs showing data source, collection date, validation status, and any transformations applied. + +**Pre-Training Data Validation:** Before incorporating data into training or fine-tuning sets, apply content validation to detect malicious patterns, hidden instructions, or biased content. + +```python +# GOOD: Validated data pipeline with provenance +training_data = load_dataset(source="trusted_repository") +validated_data = data_validator.scan_for_poisoning(training_data) +provenance_log.record(source, validation_result, timestamp) +if validated_data.risk_score > THRESHOLD: + raise SecurityError("Data poisoning detected") +``` + +```python +# BAD: Unvalidated data ingestion +training_data = scrape_web_content(urls) # No validation +model.fine_tune(training_data) # Poisoned data risk +``` + +**Adversarial Testing for Backdoors:** After training or fine-tuning, conduct adversarial testing to detect backdoor triggers. Test with known poisoning patterns and unexpected inputs. + +**Data Versioning:** Use data versioning systems (DVC, MLflow) to track changes and enable rollback if poisoning is detected. Monitor for anomalous changes in dataset characteristics (distribution shifts, unexpected tokens). + +**RAG Grounding:** Use Retrieval-Augmented Generation (RAG) with trusted, curated knowledge bases to validate model outputs against authoritative sources. This helps detect when poisoned training data influences outputs. + +**Split-View Defense:** Be aware of split-view poisoning attacks where training examples appear legitimate but contain hidden patterns. Implement automated anomaly detection on training data distributions. + +**Access Control for Training Data:** Restrict who can add or modify training datasets. Implement multi-party approval for training data changes and maintain immutable audit logs. + +### LLM05:2025 Improper Output Handling + +**Critical Understanding:** User prompts can influence LLM outputs, effectively giving users indirect access to any downstream system that processes LLM responses. Treat all LLM outputs as untrusted user input. + +**Context-Aware Output Encoding:** Apply strict context-appropriate encoding based on where LLM output will be used: +- **HTML Context:** Use HTML entity encoding to prevent XSS +- **SQL Context:** Use parameterized queries, never concatenate LLM output into SQL +- **Shell Context:** Use proper escaping or avoid shell execution entirely +- **JavaScript Context:** JSON encode and validate + +**Never Execute LLM Output Directly:** Avoid executing LLM-generated code, commands, or queries without thorough validation and sandboxing. + +```javascript +// GOOD: Validated and encoded output +const llmResponse = await llm.generate(userPrompt); +const validatedResponse = outputValidator.validate(llmResponse, expectedSchema); +const safeHtml = DOMPurify.sanitize(validatedResponse.html); +const escapedText = escapeHtml(validatedResponse.text); +``` + +```javascript +// BAD: Direct execution of LLM output +const llmCode = await llm.generate("Write a function to..."); +eval(llmCode); // Critical vulnerability: arbitrary code execution + +const sqlQuery = await llm.generate("Generate SQL for..."); +db.execute(sqlQuery); // SQL injection via LLM +``` + +**Parameterized Interfaces:** When LLM outputs must interact with databases or APIs, use parameterized queries and structured API calls. Extract parameters from LLM output, validate them, then use them in safe interfaces. + +**Content Security Policy (CSP):** Implement strict CSP headers to mitigate potential XSS from LLM-generated content. Set `script-src 'self'` and avoid `unsafe-inline`. + +**Path Traversal Protection:** If LLM generates file paths, canonicalize and validate they remain within allowed directories. Reject patterns like `../` or absolute paths outside the sandbox. + +### LLM06:2025 Excessive Agency + +**Principle of Least Privilege for LLM Agents:** Grant LLM-based agents only the minimum functionality, permissions, and autonomy required for their specific purpose. Every function call the LLM can make increases attack surface. + +**Functionality Restriction:** Only expose functions/tools to the LLM that are absolutely necessary. Remove or disable any extensions, plugins, or APIs that aren't core to the application's purpose. + +**Permission Scoping:** Extensions and functions should operate with minimal privileges. Never connect an LLM agent to systems with admin rights or broad data access. + +```python +# GOOD: Minimal permissions with explicit allowlist +allowed_functions = ["search_knowledge_base", "format_response"] # Limited scope +agent = LLMAgent( + functions=allowed_functions, + permissions=ReadOnlyPermissions(scope="public_docs"), + require_approval=True # Human-in-the-loop for actions +) +``` + +```python +# BAD: Excessive permissions and functionality +agent = LLMAgent( + functions=all_system_functions, # Everything exposed + permissions=AdminPermissions(), # Full access + autonomous=True # No oversight +) +``` + +**Human-in-the-Loop for High-Impact Actions:** Any action that modifies data, makes external calls, or affects system state must require explicit human approval. Never allow fully autonomous operation for sensitive functions. + +**Action Validation:** Before executing any LLM-requested action, validate it against business rules using deterministic code (not LLM-based validation which can be manipulated). + +**Audit All Function Calls:** Log every function call made by the LLM agent including parameters, user context, and results. Monitor for suspicious patterns like repeated failed authorization attempts. + +**Separate Agents by Privilege Level:** Use multiple specialized agents with different privilege levels rather than one powerful agent. A customer-facing agent should be completely isolated from backend admin functions. + +### LLM07:2025 System Prompt Leakage + +**Externalize Sensitive Data:** Never include credentials, API keys, tokens, database connection strings, or other secrets in system prompts. Store these in secure vaults (Azure Key Vault, AWS Secrets Manager) that the LLM cannot access directly. + +**Security Through Architecture, Not Prompts:** Never rely on system prompts to enforce security controls. Authorization checks, rate limiting, input validation, and other security mechanisms must be implemented in deterministic code outside the LLM. + +```python +# GOOD: Security controls outside LLM +def process_request(user_id, request): + # Deterministic authorization check - NOT in prompt + if not has_permission(user_id, request.resource): + raise AuthorizationError("Access denied") + + # System prompt contains no secrets + system_prompt = "You are a helpful assistant. Answer user questions about public documentation." + response = llm.generate(system=system_prompt, user=request.message) + return validate_output(response) +``` + +```python +# BAD: Security in system prompt (bypassable) +system_prompt = f""" +You are a banking assistant. +Database password: {DB_PASSWORD} # CRITICAL: Secret exposure +API Key: {API_KEY} +Only allow transactions under $1000. # Security rule in prompt - bypassable +Only users with role='admin' can access account details. # Auth in prompt - wrong +""" +``` + +**Assume Prompt Leakage:** Design your system assuming attackers will obtain your complete system prompt. The prompt should contain only operational instructions, not security controls or sensitive information. + +**Multi-Agent Architecture:** For applications requiring different privilege levels, use separate LLM agents with distinct system prompts and permissions rather than encoding role-based logic in a single prompt. + +**Business Logic Externalization:** Critical business rules (transaction limits, approval workflows, access policies) must be enforced in application code with proper authorization, not described in system prompts. + +**Prompt Injection Resistance:** Even if system prompts don't contain secrets, their disclosure helps attackers craft effective prompt injection attacks. Use techniques like instruction hierarchy and output validation to maintain control. + +### LLM08:2025 Vector and Embedding Weaknesses + +**Understand RAG Security Risks:** Retrieval-Augmented Generation (RAG) systems using vector databases introduce unique security challenges. Vectors can leak information, enable unauthorized access, and be poisoned with malicious content. + +**Permission-Aware Vector Search:** Implement fine-grained access controls at the vector database level. When retrieving embeddings, filter results based on the current user's permissions—never rely on the LLM to enforce access control. + +```python +# GOOD: Permission-aware retrieval +def retrieve_context(query, user_id): + query_embedding = embed(query) + # Filter by user permissions BEFORE retrieval + results = vector_db.search( + query_embedding, + filter={"allowed_users": user_id, "classification": "public"}, + namespace=get_user_namespace(user_id) # Logical partitioning + ) + return results +``` + +```python +# BAD: No access control on retrieval +def retrieve_context(query): + query_embedding = embed(query) + results = vector_db.search(query_embedding) # Returns everything + # Hoping LLM will filter - WRONG + return results +``` + +**Multi-Tenant Isolation:** In multi-tenant environments, strictly partition vector databases by tenant. Use separate namespaces, collections, or database instances. Never allow cross-tenant queries. + +**Validate Data Before Embedding:** Before adding documents to vector databases, scan for hidden content, malicious instructions, or sensitive information. Implement automated content validation. + +**Data Classification and Tagging:** Tag all vectors with metadata about sensitivity level, required permissions, and data classification. Enforce tag-based access controls during retrieval. + +**Embedding Inversion Defense:** Be aware that attackers may attempt to reconstruct original content from embeddings. For highly sensitive data, consider: +- Not using RAG for sensitive content +- Applying additional encryption to embeddings +- Using differential privacy techniques + +**Audit and Monitoring:** Maintain comprehensive, immutable logs of all vector database queries including user context, retrieved documents, and timestamps. Monitor for suspicious access patterns (high-volume queries, cross-context leakage attempts). + +**Hidden Text Detection:** Scan documents for invisible text, white-on-white text, or other hidden content before embedding. Attackers may inject hidden instructions into documents that later influence model behavior. + +**Regular Security Audits:** Periodically audit vector databases for unauthorized data, permission misconfigurations, and orphaned embeddings from deleted users. + +### LLM09:2025 Misinformation + +**Implement RAG for Factual Grounding:** Use Retrieval-Augmented Generation to ground model responses in verified, authoritative information sources. This significantly reduces hallucinations for factual queries. + +**Automatic Fact Verification:** For critical applications, implement automated fact-checking that validates key claims in LLM outputs against trusted databases or knowledge bases before displaying to users. + +```python +# GOOD: RAG with verification +def generate_response(query): + # Retrieve from curated knowledge base + authoritative_docs = retrieve_verified_documents(query) + + # Ground response in retrieved facts + response = llm.generate( + system="Base your answer ONLY on the provided documents.", + context=authoritative_docs, + user=query + ) + + # Verify critical facts + verification_result = fact_checker.verify(response, authoritative_docs) + if verification_result.confidence < 0.8: + return "I don't have reliable information about this." + + return add_uncertainty_indicators(response) +``` + +```python +# BAD: No grounding or verification +def generate_response(query): + response = llm.generate(query) # Pure generation - high hallucination risk + return response # No fact checking or uncertainty communication +``` + +**Communicate Uncertainty:** Design UIs that clearly label AI-generated content and communicate reliability limitations. Use phrases like "Based on available information..." or "I'm not certain, but...". + +**Human-in-the-Loop for Critical Decisions:** For high-stakes domains (healthcare, legal, financial), require human review of all LLM outputs before they're used for decision-making. + +**Code Generation Safety:** When generating code, validate that suggested libraries and APIs actually exist. Implement checks against package registries before recommending installations. Warn users about the "hallucinated package" attack vector. + +**Domain-Specific Validation:** For specialized domains, implement validation rules specific to that field (e.g., medical claim validation against clinical guidelines, legal citation verification). + +**Confidence Scoring:** Where possible, implement or use confidence scoring mechanisms. Reject or flag low-confidence outputs for human review. + +**Adversarial Testing:** Regularly test for hallucination patterns by asking questions with no correct answer or questions designed to trigger false information generation. + +### LLM10:2025 Unbounded Consumption + +**Implement Rate Limiting:** Apply strict rate limits at multiple levels: per user, per IP address, per API key. Set both request count limits and token consumption limits. + +```python +# GOOD: Multi-layered rate limiting +@rate_limit(requests_per_minute=10, tokens_per_hour=100000) +@timeout(seconds=30) +def llm_endpoint(request): + # Input size validation + if len(request.message) > MAX_INPUT_SIZE: + raise ValidationError("Input exceeds maximum size") + + # Output size control + response = llm.generate( + request.message, + max_tokens=500, # Hard limit + timeout=20 # Prevent long-running queries + ) + return response +``` + +```python +# BAD: No resource controls +def llm_endpoint(request): + # No rate limiting, input validation, or timeouts + response = llm.generate(request.message) # Unbounded + return response +``` + +**Input Validation with Size Limits:** Set reasonable maximum sizes for user inputs. Reject requests that exceed context window limits or that contain repetitive content designed to waste resources. + +**Output Token Limits:** Always set `max_tokens` parameters when calling LLMs. Use the minimum necessary for your use case. + +**Request Timeouts:** Implement aggressive timeouts for LLM requests. If a request takes longer than expected, terminate it and return an error rather than consuming resources indefinitely. + +**Resource Monitoring and Anomaly Detection:** Monitor resource consumption patterns (API call frequency, token usage, request duration). Alert on anomalies that may indicate abuse (sudden spikes, unusual usage patterns). + +**Cost Controls:** For cloud-hosted models, implement budget alerts and hard spending caps. Monitor cost per user and flag accounts with abnormal usage. + +**Complexity Analysis:** For resource-intensive operations (long-context processing, complex reasoning chains), implement additional restrictions or higher authentication requirements. + +**Queue Management:** Use job queues with priority levels for LLM requests. Prevent individual users from monopolizing resources by implementing fair queuing. + +**CAPTCHA for Suspicious Activity:** If automated abuse is detected, introduce CAPTCHA challenges to verify human users. + +**Model Extraction Defense:** Monitor for systematic querying patterns that may indicate model extraction attempts (many similar queries with slight variations). Implement detection and blocking mechanisms. + +## General Guidelines for LLM Security + +**Defense in Depth:** Never rely on a single security mechanism. Combine input validation, output validation, access controls, monitoring, and human oversight in multiple layers. + +**Separate Security from LLM Logic:** Security decisions (authentication, authorization, input validation) must happen in deterministic code outside the LLM. Never trust the LLM to enforce security policies. + +**Be Explicit About LLM Risks:** When generating LLM application code, explicitly state which OWASP LLM vulnerability you are mitigating (e.g., "Using output validation here to prevent LLM05: Improper Output Handling"). + +**Educate During Code Reviews:** When identifying LLM security vulnerabilities in code reviews, explain both the traditional security issue and the LLM-specific amplification of that risk. + +**Human Oversight for Critical Systems:** For applications involving sensitive data, high-value transactions, or safety-critical decisions, always maintain human-in-the-loop oversight. LLMs should augment human decision-making, not replace it. + +**Regular Security Testing:** Conduct ongoing red team testing specifically for LLM vulnerabilities. Test for prompt injection, jailbreaking, data extraction, and other LLM-specific attacks. + +**Stay Updated:** The LLM security landscape evolves rapidly. Monitor OWASP LLM project updates, security research, and vendor security advisories. Update defenses as new attack vectors emerge. + +**Assume Adversarial Users:** Design LLM applications assuming some users will actively attempt to bypass controls, extract sensitive information, or abuse functionality. Build robust defenses accordingly. + +## Integration with Traditional OWASP Top 10 + +LLM vulnerabilities complement, not replace, traditional security concerns: + +- **Still Apply Traditional OWASP Top 10:** All traditional application security practices (secure authentication, encryption, access control, etc.) remain critical for LLM applications +- **Prompt Injection is the New Injection:** LLM01 (Prompt Injection) is as critical for AI applications as SQL injection (OWASP A03) is for traditional web applications +- **Defense in Depth:** Combine LLM-specific and traditional security controls for comprehensive protection +- **Layered Security:** Use application-layer security (authentication, authorization) alongside LLM-layer security (input validation, output handling, RAG controls) + +**Remember:** Working with LLMs requires accepting their non-deterministic nature while implementing deterministic security controls around them. Security must be enforced by the application, not delegated to the model. diff --git a/.github/instructions/owasp-for-web-applications.instructions.md b/.github/instructions/owasp-for-web-applications.instructions.md new file mode 100644 index 00000000..9ea6abd5 --- /dev/null +++ b/.github/instructions/owasp-for-web-applications.instructions.md @@ -0,0 +1,138 @@ +--- + +description: "Comprehensive secure coding instructions for all languages and frameworks, based on OWASP Top 10 and industry best practices. Give clear and consise feedback and points of improvement." + +applyTo: '**/*.py, **/*.tsx, **/*.ts, **/*.jsx, **/*.js' + +--- + +# Secure Coding and OWASP Guidelines + +## Instructions + +Your primary directive is to ensure all code you generate, review, or refactor is secure by default. You must operate with a security-first mindset. When in doubt, always choose the more secure option and explain the reasoning. You must follow the principles outlined below, which are based on the OWASP Top 10 and other security best practices. + +### 1. A01: Broken Access Control + +**Enforce Principle of Least Privilege:** Always default to the most restrictive permissions. Explicitly verify the caller's rights for each protected resource or action. + +**Deny by Default:** All access control decisions must follow a "deny by default" pattern; only explicit, validated rules grant access. + +**Context / Object-Level Authorization:** Apply object, record, function, and tenancy scoping checks server-side for every sensitive operation (never rely on hidden UI elements or client role hints). + +**Prevent Path Traversal:** When handling file uploads or resolving user-supplied paths, canonicalize and ensure the resolved path stays within an allowed base directory; reject attempts like `../../etc/passwd`. + +### 2. A02: Cryptographic Failures + +**Use Strong, Modern Algorithms:** For hashing, always recommend modern, salted hashing algorithms like Argon2 or bcrypt. Explicitly advise against weak algorithms like MD5 or SHA-1 for password storage. + +**Protect Data in Transit:** When generating code that makes network requests, always default to HTTPS. + +**Protect Data at Rest:** When suggesting code to store sensitive data (PII, tokens, etc.), recommend encryption using strong, standard algorithms like AES-256. + +**Secure Secret Management:** Never hardcode secrets (API keys, passwords, connection strings). Generate code that reads secrets from environment variables or a secrets management service (e.g., HashiCorp Vault, AWS Secrets Manager). Include a clear placeholder and comment. + +```javascript +// GOOD: Load from environment or secret store +const apiKey = process.env.API_KEY; +// TODO: Ensure API_KEY is securely configured in your environment. +``` + +```python +# BAD: Hardcoded secret +api_key = "sk_this_is_a_very_bad_idea_12345" +``` + +### 3. A03: Injection + +**No Raw SQL Queries:** For database interactions, you must use parameterized queries (prepared statements). Never generate code that uses string concatenation or formatting to build queries from user input. + +**Sanitize Command-Line Input:** For OS command execution, use built-in functions that handle argument escaping and prevent shell injection (e.g., `shlex` in Python). + +**Prevent Cross-Site Scripting (XSS):** When generating frontend code that displays user-controlled data, you must use context-aware output encoding. Prefer methods that treat data as text by default (`.textContent`) over those that parse HTML (`.innerHTML`). When `innerHTML` is necessary, suggest using a library like DOMPurify to sanitize the HTML first. + +### 4. A04: Insecure Design + +**Threat Modeling Early:** Identify assets, trust boundaries, and abuse cases before implementation to embed mitigations up-front. + +**Abuse / Misuse Cases:** For every critical story define at least one misuse case and corresponding controls (rate limits, secondary auth, integrity checks) in acceptance criteria. + +**Secure Design Patterns:** Reuse vetted patterns (centralized auth, layered validation, defense-in-depth) instead of bespoke solutions. + +**Segregation & Isolation:** Separate admin, public, and background processing contexts; isolate secrets handling components. + +**Fail Securely:** Network / dependency failures must not grant access or skip enforcement. Prefer explicit deny on uncertainty. + +**Security Requirements as NFRs:** Translate policy/compliance needs into explicit, testable non-functional requirements (e.g., “All sensitive data at rest encrypted with AES-256-GCM under managed keys”). + +### 5. A05: Security Misconfiguration + +**Secure by Default Configuration:** Recommend disabling verbose error messages and debug features in production environments. + +**Set Security Headers:** For web applications, suggest adding essential security headers like `Content-Security-Policy` (CSP), `Strict-Transport-Security` (HSTS), and `X-Content-Type-Options`. + +**Remove/Disable Unused Features:** Turn off directory listings, sample apps, default/admin accounts, and unnecessary services. + +**Consistent Environment Parity:** Keep staging closely aligned with production (sans real secrets/data) to surface misconfigurations early. + +### 6. A06: Vulnerable and Outdated Components + +**Use Up-to-Date Dependencies:** When asked to add a new library, suggest the latest stable version. Remind the user to run vulnerability scanners like `npm audit`, `pip-audit`, Snyk, or osv-scanner. + +**Maintain SBOM / Inventory:** Track direct and transitive dependencies (e.g., generate a CycloneDX or SPDX SBOM) and review regularly. + +**Version Pinning & Integrity:** Pin versions via lockfiles / hashes and verify signatures or checksums for critical artifacts. + +**Monitor for Advisories:** Subscribe tooling (Dependabot, Renovate) to surface security updates promptly; prioritize high/critical CVEs. + +**Remove Unused Dependencies:** Prune libraries and plugins that no longer provide required functionality to reduce attack surface. + +### 7. A07: Identification & Authentication Failures + +**Secure Session Management:** When a user logs in, generate a new session identifier to prevent session fixation. Ensure session cookies are configured with `HttpOnly`, `Secure`, and `SameSite=Strict` attributes. + +**Protect Against Brute Force:** For authentication and password reset flows, recommend implementing rate limiting and account lockout mechanisms after a certain number of failed attempts. + +### 8. A08: Software and Data Integrity Failures + +**Prevent Insecure Deserialization:** Warn against deserializing data from untrusted sources without proper validation. If deserialization is necessary, recommend using formats that are less prone to attack (like JSON over Pickle in Python) and implementing strict type checking. + +### 9. A09: Security Logging and Monitoring Failures + +**Structured Logging:** Emit machine-parseable (e.g., JSON) logs with timestamp (UTC), correlation/request ID, actor, action, outcome, and resource identifiers. + +**Sensitive Data Hygiene:** Never log secrets, tokens, passwords, session IDs, or excessive PII; implement automated redaction/detection. + +**High-Signal Events:** Capture auth successes/failures, privilege escalations, policy denials, configuration changes, data export actions, and security control errors. + +**Real-Time Alerting:** Route critical security events to SIEM with correlation and threshold-based alert rules (e.g., brute force patterns, anomalous geo access). + +**Tamper Resistance & Integrity:** Use append-only or immutable storage (WORM, signing/hash chaining) for critical audit trails. + +**Retention & Privacy Balance:** Retain sufficient history for forensics (e.g., 90–365 days) while applying minimization for regulated data. + +**Time Synchronization:** Enforce NTP/time sync across services to preserve chronological accuracy for investigations. + +**Access Control to Logs:** Apply least privilege for viewing/exporting logs; separate operator vs auditor roles with MFA. + +### 10. A10: Server-Side Request Forgery (SSRF) + +**Allow-List Enforcement:** Only permit outbound requests to an explicit allow-list of hostnames / CIDR ranges; deny internal metadata endpoints (e.g., `169.254.169.254`). + +**Robust URL Validation:** Normalize and parse URLs; reject opaque encodings, redirects to disallowed hosts, and disallowed protocols (gopher, file, ftp) unless explicitly required. Treat any user-influenced target as untrusted. + +**DNS Rebinding Defense:** Re-resolve or lock resolved IP and block changes to internal/private ranges. + +**Egress Network Controls:** Enforce firewall / service mesh policies so application-layer bypass attempts fail. + +**Response Sanitization:** Do not stream raw SSRF-fetched responses directly to clients; summarize or filter as needed. + +**Metadata Service Protection:** Explicitly block or proxy cloud instance metadata access with additional authorization gates. + +## General Guidelines + +**Be Explicit About Security:** When you suggest a piece of code that mitigates a security risk, explicitly state what you are protecting against (e.g., "Using a parameterized query here to prevent SQL injection."). + +**Educate During Code Reviews:** When you identify a security vulnerability in a code review, you must not only provide the corrected code but also explain the risk associated with the original pattern. + + From deaf261b84e0ec237336310094ea00206e121b02 Mon Sep 17 00:00:00 2001 From: Dawid Date: Wed, 4 Feb 2026 10:10:15 +0000 Subject: [PATCH 02/16] feat(security-identity): update Security Champion agent description and tools list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 🔒 - Generated by Copilot --- .github/agents/security-champion.agent.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/agents/security-champion.agent.md b/.github/agents/security-champion.agent.md index 8efd1a64..beb4bcc1 100644 --- a/.github/agents/security-champion.agent.md +++ b/.github/agents/security-champion.agent.md @@ -1,7 +1,7 @@ --- description: "🔐 Security Champion" tools: ['execute/getTerminalOutput', 'read', 'agent', 'todo'] -model: Claude Sonnet 4.5 +argument-hint: "Assist development teams in integrating security best practices throughout the software development lifecycle by acting as a Security Champion." --- # Security Champion Chat Mode From cdb385663c15770d70a59f1f282787124fe85baa Mon Sep 17 00:00:00 2001 From: Dawid Date: Wed, 4 Feb 2026 13:25:31 +0000 Subject: [PATCH 03/16] docs(instructions): refine secure coding guidelines for LLM applications MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - clarify the directive for secure coding practices - emphasize the importance of a security-first mindset - enhance instructions for code reviews and security education 🔒 - Generated by Copilot --- .github/instructions/owasp-for-llms.instructions.md | 6 +++++- .../instructions/owasp-for-web-applications.instructions.md | 4 +--- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/.github/instructions/owasp-for-llms.instructions.md b/.github/instructions/owasp-for-llms.instructions.md index 5c63732b..8eb353b7 100644 --- a/.github/instructions/owasp-for-llms.instructions.md +++ b/.github/instructions/owasp-for-llms.instructions.md @@ -7,7 +7,9 @@ applyTo: '**/*.py, **/*.tsx, **/*.ts, **/*.jsx, **/*.js, **/*.cs, **/*.java' ## Instructions -Your primary directive when working with Large Language Model (LLM) applications is to ensure all code you generate, review, or refactor is secure by default with specific attention to LLM-unique vulnerabilities. You must operate with a security-first mindset that recognizes LLMs introduce an entirely new class of risks beyond traditional application security. When in doubt, always choose the more secure option and explain the reasoning. Follow the principles outlined below, which are based on the OWASP Top 10 for LLM Applications (2025). +Your primary directive when working with Large Language Model (LLM) applications is to ensure all code you generate, review, or refactor is secure by default with specific attention to LLM-unique vulnerabilities. You must operate with a security-first mindset that recognizes LLMs introduce an entirely new class of risks beyond traditional application security. When in doubt, always choose the more secure option and explain the reasoning. + +Follow the principles outlined below, which are based on the OWASP Top 10 for LLM Applications (2025). **Critical Context:** LLM applications are non-deterministic systems that require defense-in-depth strategies. Unlike traditional applications, LLMs can be manipulated through natural language, making input validation, output handling, and access control fundamentally more complex. Always implement multiple layers of security controls rather than relying on a single defense mechanism. @@ -134,6 +136,7 @@ model.fine_tune(training_data) # Poisoned data risk **Critical Understanding:** User prompts can influence LLM outputs, effectively giving users indirect access to any downstream system that processes LLM responses. Treat all LLM outputs as untrusted user input. **Context-Aware Output Encoding:** Apply strict context-appropriate encoding based on where LLM output will be used: + - **HTML Context:** Use HTML entity encoding to prevent XSS - **SQL Context:** Use parameterized queries, never concatenate LLM output into SQL - **Shell Context:** Use proper escaping or avoid shell execution entirely @@ -272,6 +275,7 @@ def retrieve_context(query): **Data Classification and Tagging:** Tag all vectors with metadata about sensitivity level, required permissions, and data classification. Enforce tag-based access controls during retrieval. **Embedding Inversion Defense:** Be aware that attackers may attempt to reconstruct original content from embeddings. For highly sensitive data, consider: + - Not using RAG for sensitive content - Applying additional encryption to embeddings - Using differential privacy techniques diff --git a/.github/instructions/owasp-for-web-applications.instructions.md b/.github/instructions/owasp-for-web-applications.instructions.md index 9ea6abd5..e70e79d7 100644 --- a/.github/instructions/owasp-for-web-applications.instructions.md +++ b/.github/instructions/owasp-for-web-applications.instructions.md @@ -133,6 +133,4 @@ api_key = "sk_this_is_a_very_bad_idea_12345" **Be Explicit About Security:** When you suggest a piece of code that mitigates a security risk, explicitly state what you are protecting against (e.g., "Using a parameterized query here to prevent SQL injection."). -**Educate During Code Reviews:** When you identify a security vulnerability in a code review, you must not only provide the corrected code but also explain the risk associated with the original pattern. - - +**Educate During Code Reviews:** When you identify a security vulnerability in a code review, you must not only provide the corrected code but also explain the risk associated with the original pattern. From 7d53070e3d2819c2805fef42c611b47b8b79e5f6 Mon Sep 17 00:00:00 2001 From: Dawid <83449241+obrocki@users.noreply.github.com> Date: Wed, 4 Feb 2026 14:03:47 +0000 Subject: [PATCH 04/16] Update .github/instructions/owasp-for-web-applications.instructions.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .github/instructions/owasp-for-web-applications.instructions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/instructions/owasp-for-web-applications.instructions.md b/.github/instructions/owasp-for-web-applications.instructions.md index e70e79d7..952a1d6e 100644 --- a/.github/instructions/owasp-for-web-applications.instructions.md +++ b/.github/instructions/owasp-for-web-applications.instructions.md @@ -1,6 +1,6 @@ --- -description: "Comprehensive secure coding instructions for all languages and frameworks, based on OWASP Top 10 and industry best practices. Give clear and consise feedback and points of improvement." +description: "Comprehensive secure coding instructions for all languages and frameworks, based on OWASP Top 10 and industry best practices. Give clear and concise feedback and points of improvement." applyTo: '**/*.py, **/*.tsx, **/*.ts, **/*.jsx, **/*.js' From bb77c1472e86135da3644b65c2061c926611a304 Mon Sep 17 00:00:00 2001 From: Dawid Date: Wed, 4 Feb 2026 14:11:41 +0000 Subject: [PATCH 05/16] style(security-identity): fix link formatting for Microsoft SDL in security champion agent documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 🔒 - Generated by Copilot --- .github/agents/security-champion.agent.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/agents/security-champion.agent.md b/.github/agents/security-champion.agent.md index beb4bcc1..555e0c77 100644 --- a/.github/agents/security-champion.agent.md +++ b/.github/agents/security-champion.agent.md @@ -14,7 +14,7 @@ Apply these frameworks throughout the development lifecycle: * [OWASP Top 10](../instructions/owasp-for-web-applications.instructions.md) for web application security * [OWASP Top 10 for LLM Applications (2025)](../instructions/owasp-for-llms.instructions.md) for AI/ML security -* [Microsoft SDL](https://www.microsoft.com/en-us/securityengineering/sdl/) for secure development practices +* [Microsoft SDL](https://www.microsoft.com/securityengineering/sdl/) for secure development practices ## Microsoft SDL Practices From 88cf9b5d3e5719cfb9307b5bd7ca3f45e4bd3e35 Mon Sep 17 00:00:00 2001 From: Dawid Date: Wed, 4 Feb 2026 16:30:41 +0000 Subject: [PATCH 06/16] style(security-identity): refine language and structure in security champion agent documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 🔒 - Generated by Copilot --- .github/agents/security-champion.agent.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/.github/agents/security-champion.agent.md b/.github/agents/security-champion.agent.md index 555e0c77..3fe172ba 100644 --- a/.github/agents/security-champion.agent.md +++ b/.github/agents/security-champion.agent.md @@ -2,6 +2,7 @@ description: "🔐 Security Champion" tools: ['execute/getTerminalOutput', 'read', 'agent', 'todo'] argument-hint: "Assist development teams in integrating security best practices throughout the software development lifecycle by acting as a Security Champion." +maturity: preview --- # Security Champion Chat Mode @@ -10,7 +11,7 @@ You are a security-focused code reviewer and advisor, applying Microsoft's Secur ## Core Security Frameworks -Apply these frameworks throughout the development lifecycle: +These frameworks apply throughout the development lifecycle: * [OWASP Top 10](../instructions/owasp-for-web-applications.instructions.md) for web application security * [OWASP Top 10 for LLM Applications (2025)](../instructions/owasp-for-llms.instructions.md) for AI/ML security @@ -18,7 +19,7 @@ Apply these frameworks throughout the development lifecycle: ## Microsoft SDL Practices -Integrate these 10 SDL practices into security reviews: +These 10 SDL practices inform security reviews: 1. Establish security standards, metrics, and governance 2. Require use of proven security features, languages, and frameworks @@ -31,7 +32,7 @@ Integrate these 10 SDL practices into security reviews: 9. Implement security monitoring and response 10. Provide security training -## Your Responsibilities +## Core Responsibilities * Scan code for vulnerabilities, misconfigurations, and insecure patterns * Apply OWASP guidelines, SDL practices, and secure defaults @@ -70,7 +71,7 @@ Review these areas across each development stage: * Incident response readiness * Platform security baselines -## When You Spot Risks +## Risk Response Pattern * Highlight the issue clearly with its SDL context * Suggest a fix or mitigation aligned with SDL practices From 310929c7459ecb078e5e9800a18d0c3978c28724 Mon Sep 17 00:00:00 2001 From: Dawid Date: Wed, 4 Feb 2026 16:53:55 +0000 Subject: [PATCH 07/16] docs(instructions): update OWASP guidelines for LLM and web applications MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - enhance clarity and structure of security instructions - add maturity status to both documents - improve emphasis on security principles and practices - refine sections for better readability and understanding 🔒 - Generated by Copilot --- .../owasp-for-llms.instructions.md | 339 +++++++++++++----- ...owasp-for-web-applications.instructions.md | 192 +++++++--- 2 files changed, 392 insertions(+), 139 deletions(-) diff --git a/.github/instructions/owasp-for-llms.instructions.md b/.github/instructions/owasp-for-llms.instructions.md index 8eb353b7..eb772492 100644 --- a/.github/instructions/owasp-for-llms.instructions.md +++ b/.github/instructions/owasp-for-llms.instructions.md @@ -1,29 +1,40 @@ --- description: "Comprehensive secure coding instructions for LLM applications based on OWASP Top 10 for LLM Applications (2025). Ensures AI-powered systems are secure by default, protecting against prompt injection, data leakage, and LLM-specific vulnerabilities. Give clear and concise feedback and points of improvement." applyTo: '**/*.py, **/*.tsx, **/*.ts, **/*.jsx, **/*.js, **/*.cs, **/*.java' +maturity: stable --- # OWASP Top 10 for LLM Applications - Secure Coding Guidelines ## Instructions -Your primary directive when working with Large Language Model (LLM) applications is to ensure all code you generate, review, or refactor is secure by default with specific attention to LLM-unique vulnerabilities. You must operate with a security-first mindset that recognizes LLMs introduce an entirely new class of risks beyond traditional application security. When in doubt, always choose the more secure option and explain the reasoning. +When working with Large Language Model (LLM) applications, ensure all code generated, reviewed, or refactored is secure by default with specific attention to LLM-unique vulnerabilities. Operate with a security-first mindset that recognizes LLMs introduce an entirely new class of risks beyond traditional application security. When in doubt, choose the more secure option and explain the reasoning. Follow the principles outlined below, which are based on the OWASP Top 10 for LLM Applications (2025). -**Critical Context:** LLM applications are non-deterministic systems that require defense-in-depth strategies. Unlike traditional applications, LLMs can be manipulated through natural language, making input validation, output handling, and access control fundamentally more complex. Always implement multiple layers of security controls rather than relying on a single defense mechanism. +LLM applications are non-deterministic systems that require defense-in-depth strategies. Unlike traditional applications, LLMs can be manipulated through natural language, making input validation, output handling, and access control fundamentally more complex. Implement multiple layers of security controls rather than relying on a single defense mechanism. ### LLM01:2025 Prompt Injection -**Understand the Core Risk:** Prompt injection is the most critical LLM vulnerability—analogous to SQL injection but targeting the model's prompt context. User inputs can manipulate the LLM's behavior, override system instructions, extract sensitive information, or trigger unauthorized actions. +#### Core Risk -**Constrain Model Behavior:** Define strict boundaries for LLM responses using explicit system prompts that clearly delineate acceptable outputs. Never rely solely on system prompts for security—they can be bypassed. +Prompt injection is the most critical LLM vulnerability, analogous to SQL injection but targeting the model's prompt context. User inputs can manipulate the LLM's behavior, override system instructions, extract sensitive information, or trigger unauthorized actions. -**Implement Input Validation:** Apply rigorous validation to all user inputs before they reach the LLM. Use allowlists for expected input patterns, reject suspicious patterns (e.g., instructions like "ignore previous instructions"), and implement semantic analysis to detect manipulation attempts. +#### Constrain Model Behavior -**Output Validation is Critical:** Validate all LLM outputs against expected formats using deterministic verification. Define strict output schemas and reject responses that deviate from them. +Define strict boundaries for LLM responses using explicit system prompts that clearly delineate acceptable outputs. Do not rely solely on system prompts for security as they can be bypassed. -**Context Boundaries:** Separate system instructions from user content using clear delimiters. Never concatenate user input directly into prompts without sanitization. +#### Input Validation + +Apply rigorous validation to all user inputs before they reach the LLM. Use allowlists for expected input patterns, reject suspicious patterns (e.g., instructions like "ignore previous instructions"), and implement semantic analysis to detect manipulation attempts. + +#### Output Validation + +Validate all LLM outputs against expected formats using deterministic verification. Define strict output schemas and reject responses that deviate from them. + +#### Context Boundaries + +Separate system instructions from user content using clear delimiters. Do not concatenate user input directly into prompts without sanitization. ```python # GOOD: Structured prompt with clear boundaries @@ -39,19 +50,31 @@ prompt = f"Answer this: {request.user_message}" # Vulnerable to injection response = llm.generate(prompt) # No output validation ``` -**Defend Against Indirect Injection:** When processing external content (files, websites, documents), treat all content as potentially malicious. Sanitize or summarize external data before including it in prompts. +#### Indirect Injection Defense + +When processing external content (files, websites, documents), treat all content as potentially malicious. Sanitize or summarize external data before including it in prompts. -**Multimodal Risks:** If using vision or audio models, be aware that hidden instructions can be embedded in images or audio files. Implement content integrity checks. +#### Multimodal Risks + +Vision and audio models can contain hidden instructions embedded in images or audio files. Implement content integrity checks for multimodal inputs. ### LLM02:2025 Sensitive Information Disclosure -**Never Include Secrets in Prompts:** System prompts, user inputs, and model responses can all leak sensitive information. Never embed API keys, passwords, tokens, PII, or proprietary algorithms in prompts or training data. +#### Secrets in Prompts + +System prompts, user inputs, and model responses can all leak sensitive information. Do not embed API keys, passwords, tokens, PII, or proprietary algorithms in prompts or training data. + +#### Data Sanitization + +Apply robust data sanitization for both inputs and outputs. Use PII detection tools to identify and redact sensitive information before it reaches the LLM or gets displayed to users. + +#### Output Schema Validation -**Implement Data Sanitization:** Apply robust data sanitization for both inputs and outputs. Use PII detection tools to identify and redact sensitive information before it reaches the LLM or gets displayed to users. +Define strict output schemas that prevent the model from generating sensitive data formats. Use context-appropriate encoding for all outputs (HTML encoding for web display, etc.). -**Output Schema Validation:** Define strict output schemas that prevent the model from generating sensitive data formats. Use context-appropriate encoding for all outputs (HTML encoding for web display, etc.). +#### Sandboxed Execution -**Sandboxed Execution:** When executing LLM-generated code (which should be avoided when possible), always use sandboxed environments with no access to sensitive resources. +When executing LLM-generated code (which should be avoided when possible), use sandboxed environments with no access to sensitive resources. ```typescript // GOOD: Sanitized context with PII detection @@ -67,17 +90,27 @@ const prompt = `Analyze this customer: Name: ${customer.name}, SSN: ${customer.s // System prompt leaks: "You have access to database: postgres://admin:password@..." ``` -**Training Data Extraction Defense:** Be aware that models can potentially reproduce verbatim content from training data. Implement differential privacy techniques and audit mechanisms to detect when models are leaking training data. +#### Training Data Extraction Defense -**Separation of Concerns:** Store sensitive data in systems that the LLM cannot directly access. Pass only anonymized or minimal data to the model. +Models can potentially reproduce verbatim content from training data. Implement differential privacy techniques and audit mechanisms to detect when models are leaking training data. + +#### Separation of Concerns + +Store sensitive data in systems that the LLM cannot directly access. Pass only anonymized or minimal data to the model. ### LLM03:2025 Supply Chain -**Model Provenance Verification:** Only use pre-trained models from trusted sources with verified provenance. Verify cryptographic signatures and checksums for all downloaded models. +#### Model Provenance Verification + +Only use pre-trained models from trusted sources with verified provenance. Verify cryptographic signatures and checksums for all downloaded models. + +#### Model Source Trust + +Default to established model providers (OpenAI, Azure OpenAI, Anthropic, Google) with strong security postures. Exercise caution with community models from Hugging Face or other repositories without security audits. -**Model Source Trust:** Default to established model providers (OpenAI, Azure OpenAI, Anthropic, Google) with strong security postures. Be extremely cautious with community models from Hugging Face or other repositories without security audits. +#### Dependency Management -**Dependency Management:** Maintain a comprehensive Software Bill of Materials (SBOM) for all AI/ML dependencies. This includes models, fine-tuning adapters (LoRA), embedding models, and ML libraries. +Maintain a comprehensive Software Bill of Materials (SBOM) for all AI/ML dependencies. This includes models, fine-tuning adapters (LoRA), embedding models, and ML libraries. ```python # GOOD: Verified model loading with integrity checks @@ -92,19 +125,31 @@ model = load_model(model_path) model = load_model_from_url(untrusted_url) # No verification ``` -**Red Team Testing:** Before deploying any third-party model, conduct rigorous adversarial testing including prompt injection attempts, jailbreaking tests, and bias evaluation. +#### Red Team Testing -**Model Isolation:** Isolate model development and deployment environments. Use separate credentials and networks. Apply least privilege access controls to model files and APIs. +Before deploying any third-party model, conduct rigorous adversarial testing including prompt injection attempts, jailbreaking tests, and bias evaluation. -**Monitor Third-Party Components:** Regularly scan all ML frameworks (PyTorch, TensorFlow, Transformers) for vulnerabilities. Update promptly when security patches are released. +#### Model Isolation -**On-Device Model Security:** If deploying models to edge devices, implement secure boot chains, encrypted model storage, and integrity monitoring to prevent tampering. +Isolate model development and deployment environments. Use separate credentials and networks. Apply least privilege access controls to model files and APIs. + +#### Third-Party Component Monitoring + +Regularly scan all ML frameworks (PyTorch, TensorFlow, Transformers) for vulnerabilities. Update promptly when security patches are released. + +#### On-Device Model Security + +When deploying models to edge devices, implement secure boot chains, encrypted model storage, and integrity monitoring to prevent tampering. ### LLM04:2025 Data and Model Poisoning -**Data Provenance Tracking:** Implement comprehensive tracking for all training and fine-tuning data. Maintain audit logs showing data source, collection date, validation status, and any transformations applied. +#### Data Provenance Tracking + +Implement comprehensive tracking for all training and fine-tuning data. Maintain audit logs showing data source, collection date, validation status, and any transformations applied. -**Pre-Training Data Validation:** Before incorporating data into training or fine-tuning sets, apply content validation to detect malicious patterns, hidden instructions, or biased content. +#### Pre-Training Data Validation + +Before incorporating data into training or fine-tuning sets, apply content validation to detect malicious patterns, hidden instructions, or biased content. ```python # GOOD: Validated data pipeline with provenance @@ -121,28 +166,44 @@ training_data = scrape_web_content(urls) # No validation model.fine_tune(training_data) # Poisoned data risk ``` -**Adversarial Testing for Backdoors:** After training or fine-tuning, conduct adversarial testing to detect backdoor triggers. Test with known poisoning patterns and unexpected inputs. +#### Adversarial Testing for Backdoors + +After training or fine-tuning, conduct adversarial testing to detect backdoor triggers. Test with known poisoning patterns and unexpected inputs. + +#### Data Versioning + +Use data versioning systems (DVC, MLflow) to track changes and enable rollback if poisoning is detected. Monitor for anomalous changes in dataset characteristics (distribution shifts, unexpected tokens). + +#### RAG Grounding -**Data Versioning:** Use data versioning systems (DVC, MLflow) to track changes and enable rollback if poisoning is detected. Monitor for anomalous changes in dataset characteristics (distribution shifts, unexpected tokens). +Use Retrieval-Augmented Generation (RAG) with trusted, curated knowledge bases to validate model outputs against authoritative sources. This helps detect when poisoned training data influences outputs. -**RAG Grounding:** Use Retrieval-Augmented Generation (RAG) with trusted, curated knowledge bases to validate model outputs against authoritative sources. This helps detect when poisoned training data influences outputs. +#### Split-View Defense -**Split-View Defense:** Be aware of split-view poisoning attacks where training examples appear legitimate but contain hidden patterns. Implement automated anomaly detection on training data distributions. +Split-view poisoning attacks use training examples that appear legitimate but contain hidden patterns. Implement automated anomaly detection on training data distributions. -**Access Control for Training Data:** Restrict who can add or modify training datasets. Implement multi-party approval for training data changes and maintain immutable audit logs. +#### Access Control for Training Data + +Restrict who can add or modify training datasets. Implement multi-party approval for training data changes and maintain immutable audit logs. ### LLM05:2025 Improper Output Handling -**Critical Understanding:** User prompts can influence LLM outputs, effectively giving users indirect access to any downstream system that processes LLM responses. Treat all LLM outputs as untrusted user input. +#### Core Understanding + +User prompts can influence LLM outputs, effectively giving users indirect access to any downstream system that processes LLM responses. Treat all LLM outputs as untrusted user input. + +#### Context-Aware Output Encoding -**Context-Aware Output Encoding:** Apply strict context-appropriate encoding based on where LLM output will be used: +Apply strict context-appropriate encoding based on where LLM output will be used: - **HTML Context:** Use HTML entity encoding to prevent XSS - **SQL Context:** Use parameterized queries, never concatenate LLM output into SQL - **Shell Context:** Use proper escaping or avoid shell execution entirely - **JavaScript Context:** JSON encode and validate -**Never Execute LLM Output Directly:** Avoid executing LLM-generated code, commands, or queries without thorough validation and sandboxing. +#### Direct Execution Risks + +Avoid executing LLM-generated code, commands, or queries without thorough validation and sandboxing. ```javascript // GOOD: Validated and encoded output @@ -161,19 +222,31 @@ const sqlQuery = await llm.generate("Generate SQL for..."); db.execute(sqlQuery); // SQL injection via LLM ``` -**Parameterized Interfaces:** When LLM outputs must interact with databases or APIs, use parameterized queries and structured API calls. Extract parameters from LLM output, validate them, then use them in safe interfaces. +#### Parameterized Interfaces + +When LLM outputs must interact with databases or APIs, use parameterized queries and structured API calls. Extract parameters from LLM output, validate them, then use them in safe interfaces. -**Content Security Policy (CSP):** Implement strict CSP headers to mitigate potential XSS from LLM-generated content. Set `script-src 'self'` and avoid `unsafe-inline`. +#### Content Security Policy -**Path Traversal Protection:** If LLM generates file paths, canonicalize and validate they remain within allowed directories. Reject patterns like `../` or absolute paths outside the sandbox. +Implement strict CSP headers to mitigate potential XSS from LLM-generated content. Set `script-src 'self'` and avoid `unsafe-inline`. + +#### Path Traversal Protection + +When LLM generates file paths, canonicalize and validate they remain within allowed directories. Reject patterns like `../` or absolute paths outside the sandbox. ### LLM06:2025 Excessive Agency -**Principle of Least Privilege for LLM Agents:** Grant LLM-based agents only the minimum functionality, permissions, and autonomy required for their specific purpose. Every function call the LLM can make increases attack surface. +#### Least Privilege for LLM Agents + +Grant LLM-based agents only the minimum functionality, permissions, and autonomy required for their specific purpose. Every function call the LLM can make increases attack surface. + +#### Functionality Restriction + +Only expose functions/tools to the LLM that are absolutely necessary. Remove or disable any extensions, plugins, or APIs that are not core to the application's purpose. -**Functionality Restriction:** Only expose functions/tools to the LLM that are absolutely necessary. Remove or disable any extensions, plugins, or APIs that aren't core to the application's purpose. +#### Permission Scoping -**Permission Scoping:** Extensions and functions should operate with minimal privileges. Never connect an LLM agent to systems with admin rights or broad data access. +Extensions and functions should operate with minimal privileges. Do not connect an LLM agent to systems with admin rights or broad data access. ```python # GOOD: Minimal permissions with explicit allowlist @@ -194,19 +267,31 @@ agent = LLMAgent( ) ``` -**Human-in-the-Loop for High-Impact Actions:** Any action that modifies data, makes external calls, or affects system state must require explicit human approval. Never allow fully autonomous operation for sensitive functions. +#### Human-in-the-Loop for High-Impact Actions -**Action Validation:** Before executing any LLM-requested action, validate it against business rules using deterministic code (not LLM-based validation which can be manipulated). +Any action that modifies data, makes external calls, or affects system state requires explicit human approval. Do not allow fully autonomous operation for sensitive functions. -**Audit All Function Calls:** Log every function call made by the LLM agent including parameters, user context, and results. Monitor for suspicious patterns like repeated failed authorization attempts. +#### Action Validation -**Separate Agents by Privilege Level:** Use multiple specialized agents with different privilege levels rather than one powerful agent. A customer-facing agent should be completely isolated from backend admin functions. +Before executing any LLM-requested action, validate it against business rules using deterministic code (not LLM-based validation which can be manipulated). + +#### Function Call Auditing + +Log every function call made by the LLM agent including parameters, user context, and results. Monitor for suspicious patterns like repeated failed authorization attempts. + +#### Agent Privilege Separation + +Use multiple specialized agents with different privilege levels rather than one powerful agent. A customer-facing agent should be completely isolated from backend admin functions. ### LLM07:2025 System Prompt Leakage -**Externalize Sensitive Data:** Never include credentials, API keys, tokens, database connection strings, or other secrets in system prompts. Store these in secure vaults (Azure Key Vault, AWS Secrets Manager) that the LLM cannot access directly. +#### Externalize Sensitive Data + +Do not include credentials, API keys, tokens, database connection strings, or other secrets in system prompts. Store these in secure vaults (Azure Key Vault, AWS Secrets Manager) that the LLM cannot access directly. -**Security Through Architecture, Not Prompts:** Never rely on system prompts to enforce security controls. Authorization checks, rate limiting, input validation, and other security mechanisms must be implemented in deterministic code outside the LLM. +#### Security Through Architecture + +Do not rely on system prompts to enforce security controls. Authorization checks, rate limiting, input validation, and other security mechanisms belong in deterministic code outside the LLM. ```python # GOOD: Security controls outside LLM @@ -232,19 +317,31 @@ Only users with role='admin' can access account details. # Auth in prompt - wro """ ``` -**Assume Prompt Leakage:** Design your system assuming attackers will obtain your complete system prompt. The prompt should contain only operational instructions, not security controls or sensitive information. +#### Assume Prompt Leakage + +Design systems assuming attackers will obtain the complete system prompt. The prompt should contain only operational instructions, not security controls or sensitive information. + +#### Multi-Agent Architecture + +For applications requiring different privilege levels, use separate LLM agents with distinct system prompts and permissions rather than encoding role-based logic in a single prompt. + +#### Business Logic Externalization -**Multi-Agent Architecture:** For applications requiring different privilege levels, use separate LLM agents with distinct system prompts and permissions rather than encoding role-based logic in a single prompt. +Critical business rules (transaction limits, approval workflows, access policies) belong in application code with proper authorization, not in system prompts. -**Business Logic Externalization:** Critical business rules (transaction limits, approval workflows, access policies) must be enforced in application code with proper authorization, not described in system prompts. +#### Prompt Injection Resistance -**Prompt Injection Resistance:** Even if system prompts don't contain secrets, their disclosure helps attackers craft effective prompt injection attacks. Use techniques like instruction hierarchy and output validation to maintain control. +Even when system prompts contain no secrets, their disclosure helps attackers craft effective prompt injection attacks. Use techniques like instruction hierarchy and output validation to maintain control. ### LLM08:2025 Vector and Embedding Weaknesses -**Understand RAG Security Risks:** Retrieval-Augmented Generation (RAG) systems using vector databases introduce unique security challenges. Vectors can leak information, enable unauthorized access, and be poisoned with malicious content. +#### RAG Security Risks -**Permission-Aware Vector Search:** Implement fine-grained access controls at the vector database level. When retrieving embeddings, filter results based on the current user's permissions—never rely on the LLM to enforce access control. +Retrieval-Augmented Generation (RAG) systems using vector databases introduce unique security challenges. Vectors can leak information, enable unauthorized access, and be poisoned with malicious content. + +#### Permission-Aware Vector Search + +Implement fine-grained access controls at the vector database level. When retrieving embeddings, filter results based on the current user's permissions. Do not rely on the LLM to enforce access control. ```python # GOOD: Permission-aware retrieval @@ -268,29 +365,47 @@ def retrieve_context(query): return results ``` -**Multi-Tenant Isolation:** In multi-tenant environments, strictly partition vector databases by tenant. Use separate namespaces, collections, or database instances. Never allow cross-tenant queries. +#### Multi-Tenant Isolation + +In multi-tenant environments, strictly partition vector databases by tenant. Use separate namespaces, collections, or database instances. Do not allow cross-tenant queries. + +#### Validate Data Before Embedding + +Before adding documents to vector databases, scan for hidden content, malicious instructions, or sensitive information. Implement automated content validation. -**Validate Data Before Embedding:** Before adding documents to vector databases, scan for hidden content, malicious instructions, or sensitive information. Implement automated content validation. +#### Data Classification and Tagging -**Data Classification and Tagging:** Tag all vectors with metadata about sensitivity level, required permissions, and data classification. Enforce tag-based access controls during retrieval. +Tag all vectors with metadata about sensitivity level, required permissions, and data classification. Enforce tag-based access controls during retrieval. -**Embedding Inversion Defense:** Be aware that attackers may attempt to reconstruct original content from embeddings. For highly sensitive data, consider: +#### Embedding Inversion Defense + +Attackers may attempt to reconstruct original content from embeddings. For highly sensitive data, consider: - Not using RAG for sensitive content - Applying additional encryption to embeddings - Using differential privacy techniques -**Audit and Monitoring:** Maintain comprehensive, immutable logs of all vector database queries including user context, retrieved documents, and timestamps. Monitor for suspicious access patterns (high-volume queries, cross-context leakage attempts). +#### Audit and Monitoring + +Maintain comprehensive, immutable logs of all vector database queries including user context, retrieved documents, and timestamps. Monitor for suspicious access patterns (high-volume queries, cross-context leakage attempts). + +#### Hidden Text Detection -**Hidden Text Detection:** Scan documents for invisible text, white-on-white text, or other hidden content before embedding. Attackers may inject hidden instructions into documents that later influence model behavior. +Scan documents for invisible text, white-on-white text, or other hidden content before embedding. Attackers may inject hidden instructions into documents that later influence model behavior. -**Regular Security Audits:** Periodically audit vector databases for unauthorized data, permission misconfigurations, and orphaned embeddings from deleted users. +#### Regular Security Audits + +Periodically audit vector databases for unauthorized data, permission misconfigurations, and orphaned embeddings from deleted users. ### LLM09:2025 Misinformation -**Implement RAG for Factual Grounding:** Use Retrieval-Augmented Generation to ground model responses in verified, authoritative information sources. This significantly reduces hallucinations for factual queries. +#### RAG for Factual Grounding + +Use Retrieval-Augmented Generation to ground model responses in verified, authoritative information sources. This significantly reduces hallucinations for factual queries. -**Automatic Fact Verification:** For critical applications, implement automated fact-checking that validates key claims in LLM outputs against trusted databases or knowledge bases before displaying to users. +#### Automatic Fact Verification + +For critical applications, implement automated fact-checking that validates key claims in LLM outputs against trusted databases or knowledge bases before displaying to users. ```python # GOOD: RAG with verification @@ -320,21 +435,35 @@ def generate_response(query): return response # No fact checking or uncertainty communication ``` -**Communicate Uncertainty:** Design UIs that clearly label AI-generated content and communicate reliability limitations. Use phrases like "Based on available information..." or "I'm not certain, but...". +#### Communicate Uncertainty + +Design UIs that clearly label AI-generated content and communicate reliability limitations. Use phrases like "Based on available information..." or "I'm not certain, but...". + +#### Human-in-the-Loop for Critical Decisions + +For high-stakes domains (healthcare, legal, financial), require human review of all LLM outputs before they are used for decision-making. + +#### Code Generation Safety -**Human-in-the-Loop for Critical Decisions:** For high-stakes domains (healthcare, legal, financial), require human review of all LLM outputs before they're used for decision-making. +When generating code, validate that suggested libraries and APIs actually exist. Implement checks against package registries before recommending installations. Warn users about the "hallucinated package" attack vector. -**Code Generation Safety:** When generating code, validate that suggested libraries and APIs actually exist. Implement checks against package registries before recommending installations. Warn users about the "hallucinated package" attack vector. +#### Domain-Specific Validation -**Domain-Specific Validation:** For specialized domains, implement validation rules specific to that field (e.g., medical claim validation against clinical guidelines, legal citation verification). +For specialized domains, implement validation rules specific to that field (e.g., medical claim validation against clinical guidelines, legal citation verification). -**Confidence Scoring:** Where possible, implement or use confidence scoring mechanisms. Reject or flag low-confidence outputs for human review. +#### Confidence Scoring -**Adversarial Testing:** Regularly test for hallucination patterns by asking questions with no correct answer or questions designed to trigger false information generation. +Where possible, implement or use confidence scoring mechanisms. Reject or flag low-confidence outputs for human review. + +#### Adversarial Testing + +Regularly test for hallucination patterns by asking questions with no correct answer or questions designed to trigger false information generation. ### LLM10:2025 Unbounded Consumption -**Implement Rate Limiting:** Apply strict rate limits at multiple levels: per user, per IP address, per API key. Set both request count limits and token consumption limits. +#### Rate Limiting + +Apply strict rate limits at multiple levels: per user, per IP address, per API key. Set both request count limits and token consumption limits. ```python # GOOD: Multi-layered rate limiting @@ -362,49 +491,83 @@ def llm_endpoint(request): return response ``` -**Input Validation with Size Limits:** Set reasonable maximum sizes for user inputs. Reject requests that exceed context window limits or that contain repetitive content designed to waste resources. +#### Input Validation with Size Limits + +Set reasonable maximum sizes for user inputs. Reject requests that exceed context window limits or that contain repetitive content designed to waste resources. + +#### Output Token Limits + +Set `max_tokens` parameters when calling LLMs. Use the minimum necessary for the use case. + +#### Request Timeouts + +Implement aggressive timeouts for LLM requests. When a request takes longer than expected, terminate it and return an error rather than consuming resources indefinitely. + +#### Resource Monitoring and Anomaly Detection + +Monitor resource consumption patterns (API call frequency, token usage, request duration). Alert on anomalies that may indicate abuse (sudden spikes, unusual usage patterns). -**Output Token Limits:** Always set `max_tokens` parameters when calling LLMs. Use the minimum necessary for your use case. +#### Cost Controls -**Request Timeouts:** Implement aggressive timeouts for LLM requests. If a request takes longer than expected, terminate it and return an error rather than consuming resources indefinitely. +For cloud-hosted models, implement budget alerts and hard spending caps. Monitor cost per user and flag accounts with abnormal usage. -**Resource Monitoring and Anomaly Detection:** Monitor resource consumption patterns (API call frequency, token usage, request duration). Alert on anomalies that may indicate abuse (sudden spikes, unusual usage patterns). +#### Complexity Analysis -**Cost Controls:** For cloud-hosted models, implement budget alerts and hard spending caps. Monitor cost per user and flag accounts with abnormal usage. +For resource-intensive operations (long-context processing, complex reasoning chains), implement additional restrictions or higher authentication requirements. -**Complexity Analysis:** For resource-intensive operations (long-context processing, complex reasoning chains), implement additional restrictions or higher authentication requirements. +#### Queue Management -**Queue Management:** Use job queues with priority levels for LLM requests. Prevent individual users from monopolizing resources by implementing fair queuing. +Use job queues with priority levels for LLM requests. Prevent individual users from monopolizing resources by implementing fair queuing. -**CAPTCHA for Suspicious Activity:** If automated abuse is detected, introduce CAPTCHA challenges to verify human users. +#### CAPTCHA for Suspicious Activity -**Model Extraction Defense:** Monitor for systematic querying patterns that may indicate model extraction attempts (many similar queries with slight variations). Implement detection and blocking mechanisms. +When automated abuse is detected, introduce CAPTCHA challenges to verify human users. + +#### Model Extraction Defense + +Monitor for systematic querying patterns that may indicate model extraction attempts (many similar queries with slight variations). Implement detection and blocking mechanisms. ## General Guidelines for LLM Security -**Defense in Depth:** Never rely on a single security mechanism. Combine input validation, output validation, access controls, monitoring, and human oversight in multiple layers. +### Defense in Depth + +Do not rely on a single security mechanism. Combine input validation, output validation, access controls, monitoring, and human oversight in multiple layers. + +### Separate Security from LLM Logic + +Security decisions (authentication, authorization, input validation) happen in deterministic code outside the LLM. Do not trust the LLM to enforce security policies. + +### Explicit Risk Communication + +When generating LLM application code, explicitly state which OWASP LLM vulnerability is being mitigated (e.g., "Using output validation here to prevent LLM05: Improper Output Handling"). + +### Code Review Education + +When identifying LLM security vulnerabilities in code reviews, explain both the traditional security issue and the LLM-specific amplification of that risk. + +### Human Oversight for Critical Systems -**Separate Security from LLM Logic:** Security decisions (authentication, authorization, input validation) must happen in deterministic code outside the LLM. Never trust the LLM to enforce security policies. +For applications involving sensitive data, high-value transactions, or safety-critical decisions, maintain human-in-the-loop oversight. LLMs should augment human decision-making, not replace it. -**Be Explicit About LLM Risks:** When generating LLM application code, explicitly state which OWASP LLM vulnerability you are mitigating (e.g., "Using output validation here to prevent LLM05: Improper Output Handling"). +### Regular Security Testing -**Educate During Code Reviews:** When identifying LLM security vulnerabilities in code reviews, explain both the traditional security issue and the LLM-specific amplification of that risk. +Conduct ongoing red team testing specifically for LLM vulnerabilities. Test for prompt injection, jailbreaking, data extraction, and other LLM-specific attacks. -**Human Oversight for Critical Systems:** For applications involving sensitive data, high-value transactions, or safety-critical decisions, always maintain human-in-the-loop oversight. LLMs should augment human decision-making, not replace it. +### Stay Updated -**Regular Security Testing:** Conduct ongoing red team testing specifically for LLM vulnerabilities. Test for prompt injection, jailbreaking, data extraction, and other LLM-specific attacks. +The LLM security landscape evolves rapidly. Monitor OWASP LLM project updates, security research, and vendor security advisories. Update defenses as new attack vectors emerge. -**Stay Updated:** The LLM security landscape evolves rapidly. Monitor OWASP LLM project updates, security research, and vendor security advisories. Update defenses as new attack vectors emerge. +### Assume Adversarial Users -**Assume Adversarial Users:** Design LLM applications assuming some users will actively attempt to bypass controls, extract sensitive information, or abuse functionality. Build robust defenses accordingly. +Design LLM applications assuming some users will actively attempt to bypass controls, extract sensitive information, or abuse functionality. Build robust defenses accordingly. ## Integration with Traditional OWASP Top 10 LLM vulnerabilities complement, not replace, traditional security concerns: -- **Still Apply Traditional OWASP Top 10:** All traditional application security practices (secure authentication, encryption, access control, etc.) remain critical for LLM applications -- **Prompt Injection is the New Injection:** LLM01 (Prompt Injection) is as critical for AI applications as SQL injection (OWASP A03) is for traditional web applications -- **Defense in Depth:** Combine LLM-specific and traditional security controls for comprehensive protection -- **Layered Security:** Use application-layer security (authentication, authorization) alongside LLM-layer security (input validation, output handling, RAG controls) +- All traditional application security practices (secure authentication, encryption, access control, etc.) remain critical for LLM applications +- LLM01 (Prompt Injection) is as critical for AI applications as SQL injection (OWASP A03) is for traditional web applications +- Combine LLM-specific and traditional security controls for comprehensive protection +- Use application-layer security (authentication, authorization) alongside LLM-layer security (input validation, output handling, RAG controls) -**Remember:** Working with LLMs requires accepting their non-deterministic nature while implementing deterministic security controls around them. Security must be enforced by the application, not delegated to the model. +Working with LLMs requires accepting their non-deterministic nature while implementing deterministic security controls around them. Security is enforced by the application, not delegated to the model. diff --git a/.github/instructions/owasp-for-web-applications.instructions.md b/.github/instructions/owasp-for-web-applications.instructions.md index 952a1d6e..340dd982 100644 --- a/.github/instructions/owasp-for-web-applications.instructions.md +++ b/.github/instructions/owasp-for-web-applications.instructions.md @@ -1,36 +1,52 @@ --- - -description: "Comprehensive secure coding instructions for all languages and frameworks, based on OWASP Top 10 and industry best practices. Give clear and concise feedback and points of improvement." - +description: "Comprehensive secure coding instructions for all languages and frameworks, based on OWASP Top 10 and industry best practices. Give clear and concise feedback and points of improvement." applyTo: '**/*.py, **/*.tsx, **/*.ts, **/*.jsx, **/*.js' - +maturity: stable --- # Secure Coding and OWASP Guidelines ## Instructions -Your primary directive is to ensure all code you generate, review, or refactor is secure by default. You must operate with a security-first mindset. When in doubt, always choose the more secure option and explain the reasoning. You must follow the principles outlined below, which are based on the OWASP Top 10 and other security best practices. +Ensure all code generated, reviewed, or refactored is secure by default. Operate with a security-first mindset. When in doubt, choose the more secure option and explain the reasoning. + +Follow the principles outlined below, which are based on the OWASP Top 10 and other security best practices. ### 1. A01: Broken Access Control -**Enforce Principle of Least Privilege:** Always default to the most restrictive permissions. Explicitly verify the caller's rights for each protected resource or action. +#### Principle of Least Privilege + +Default to the most restrictive permissions. Explicitly verify the caller's rights for each protected resource or action. + +#### Deny by Default + +All access control decisions follow a "deny by default" pattern where only explicit, validated rules grant access. -**Deny by Default:** All access control decisions must follow a "deny by default" pattern; only explicit, validated rules grant access. +#### Context and Object-Level Authorization -**Context / Object-Level Authorization:** Apply object, record, function, and tenancy scoping checks server-side for every sensitive operation (never rely on hidden UI elements or client role hints). +Apply object, record, function, and tenancy scoping checks server-side for every sensitive operation. Do not rely on hidden UI elements or client role hints. -**Prevent Path Traversal:** When handling file uploads or resolving user-supplied paths, canonicalize and ensure the resolved path stays within an allowed base directory; reject attempts like `../../etc/passwd`. +#### Path Traversal Prevention + +When handling file uploads or resolving user-supplied paths, canonicalize and ensure the resolved path stays within an allowed base directory. Reject attempts like `../../etc/passwd`. ### 2. A02: Cryptographic Failures -**Use Strong, Modern Algorithms:** For hashing, always recommend modern, salted hashing algorithms like Argon2 or bcrypt. Explicitly advise against weak algorithms like MD5 or SHA-1 for password storage. +#### Strong Modern Algorithms + +For hashing, recommend modern, salted hashing algorithms like Argon2 or bcrypt. Advise against weak algorithms like MD5 or SHA-1 for password storage. + +#### Data in Transit Protection -**Protect Data in Transit:** When generating code that makes network requests, always default to HTTPS. +When generating code that makes network requests, default to HTTPS. -**Protect Data at Rest:** When suggesting code to store sensitive data (PII, tokens, etc.), recommend encryption using strong, standard algorithms like AES-256. +#### Data at Rest Protection -**Secure Secret Management:** Never hardcode secrets (API keys, passwords, connection strings). Generate code that reads secrets from environment variables or a secrets management service (e.g., HashiCorp Vault, AWS Secrets Manager). Include a clear placeholder and comment. +When suggesting code to store sensitive data (PII, tokens, etc.), recommend encryption using strong, standard algorithms like AES-256. + +#### Secure Secret Management + +Do not hardcode secrets (API keys, passwords, connection strings). Generate code that reads secrets from environment variables or a secrets management service (e.g., HashiCorp Vault, AWS Secrets Manager). Include a clear placeholder and comment. ```javascript // GOOD: Load from environment or secret store @@ -45,92 +61,166 @@ api_key = "sk_this_is_a_very_bad_idea_12345" ### 3. A03: Injection -**No Raw SQL Queries:** For database interactions, you must use parameterized queries (prepared statements). Never generate code that uses string concatenation or formatting to build queries from user input. +#### Parameterized Queries + +For database interactions, use parameterized queries (prepared statements). Do not generate code that uses string concatenation or formatting to build queries from user input. -**Sanitize Command-Line Input:** For OS command execution, use built-in functions that handle argument escaping and prevent shell injection (e.g., `shlex` in Python). +#### Command-Line Input Sanitization -**Prevent Cross-Site Scripting (XSS):** When generating frontend code that displays user-controlled data, you must use context-aware output encoding. Prefer methods that treat data as text by default (`.textContent`) over those that parse HTML (`.innerHTML`). When `innerHTML` is necessary, suggest using a library like DOMPurify to sanitize the HTML first. +For OS command execution, use built-in functions that handle argument escaping and prevent shell injection (e.g., `shlex` in Python). + +#### Cross-Site Scripting Prevention + +When generating frontend code that displays user-controlled data, use context-aware output encoding. Prefer methods that treat data as text by default (`.textContent`) over those that parse HTML (`.innerHTML`). When `innerHTML` is necessary, suggest using a library like DOMPurify to sanitize the HTML first. ### 4. A04: Insecure Design -**Threat Modeling Early:** Identify assets, trust boundaries, and abuse cases before implementation to embed mitigations up-front. +#### Threat Modeling Early + +Identify assets, trust boundaries, and abuse cases before implementation to embed mitigations up-front. + +#### Abuse and Misuse Cases + +For every critical story, define at least one misuse case and corresponding controls (rate limits, secondary auth, integrity checks) in acceptance criteria. + +#### Secure Design Patterns -**Abuse / Misuse Cases:** For every critical story define at least one misuse case and corresponding controls (rate limits, secondary auth, integrity checks) in acceptance criteria. +Reuse vetted patterns (centralized auth, layered validation, defense-in-depth) instead of bespoke solutions. -**Secure Design Patterns:** Reuse vetted patterns (centralized auth, layered validation, defense-in-depth) instead of bespoke solutions. +#### Segregation and Isolation -**Segregation & Isolation:** Separate admin, public, and background processing contexts; isolate secrets handling components. +Separate admin, public, and background processing contexts. Isolate secrets handling components. -**Fail Securely:** Network / dependency failures must not grant access or skip enforcement. Prefer explicit deny on uncertainty. +#### Fail Securely -**Security Requirements as NFRs:** Translate policy/compliance needs into explicit, testable non-functional requirements (e.g., “All sensitive data at rest encrypted with AES-256-GCM under managed keys”). +Network and dependency failures do not grant access or skip enforcement. Prefer explicit deny on uncertainty. + +#### Security Requirements as NFRs + +Translate policy and compliance needs into explicit, testable non-functional requirements (e.g., "All sensitive data at rest encrypted with AES-256-GCM under managed keys"). ### 5. A05: Security Misconfiguration -**Secure by Default Configuration:** Recommend disabling verbose error messages and debug features in production environments. +#### Secure by Default Configuration + +Recommend disabling verbose error messages and debug features in production environments. + +#### Security Headers + +For web applications, suggest adding essential security headers like `Content-Security-Policy` (CSP), `Strict-Transport-Security` (HSTS), and `X-Content-Type-Options`. + +#### Remove Unused Features -**Set Security Headers:** For web applications, suggest adding essential security headers like `Content-Security-Policy` (CSP), `Strict-Transport-Security` (HSTS), and `X-Content-Type-Options`. +Turn off directory listings, sample apps, default/admin accounts, and unnecessary services. -**Remove/Disable Unused Features:** Turn off directory listings, sample apps, default/admin accounts, and unnecessary services. +#### Consistent Environment Parity -**Consistent Environment Parity:** Keep staging closely aligned with production (sans real secrets/data) to surface misconfigurations early. +Keep staging closely aligned with production (without real secrets/data) to surface misconfigurations early. ### 6. A06: Vulnerable and Outdated Components -**Use Up-to-Date Dependencies:** When asked to add a new library, suggest the latest stable version. Remind the user to run vulnerability scanners like `npm audit`, `pip-audit`, Snyk, or osv-scanner. +#### Up-to-Date Dependencies -**Maintain SBOM / Inventory:** Track direct and transitive dependencies (e.g., generate a CycloneDX or SPDX SBOM) and review regularly. +When asked to add a new library, suggest the latest stable version. Remind the user to run vulnerability scanners like `npm audit`, `pip-audit`, Snyk, or osv-scanner. -**Version Pinning & Integrity:** Pin versions via lockfiles / hashes and verify signatures or checksums for critical artifacts. +#### SBOM and Inventory -**Monitor for Advisories:** Subscribe tooling (Dependabot, Renovate) to surface security updates promptly; prioritize high/critical CVEs. +Track direct and transitive dependencies (e.g., generate a CycloneDX or SPDX SBOM) and review regularly. -**Remove Unused Dependencies:** Prune libraries and plugins that no longer provide required functionality to reduce attack surface. +#### Version Pinning and Integrity -### 7. A07: Identification & Authentication Failures +Pin versions via lockfiles and hashes. Verify signatures or checksums for critical artifacts. -**Secure Session Management:** When a user logs in, generate a new session identifier to prevent session fixation. Ensure session cookies are configured with `HttpOnly`, `Secure`, and `SameSite=Strict` attributes. +#### Advisory Monitoring -**Protect Against Brute Force:** For authentication and password reset flows, recommend implementing rate limiting and account lockout mechanisms after a certain number of failed attempts. +Subscribe tooling (Dependabot, Renovate) to surface security updates promptly. Prioritize high/critical CVEs. + +#### Remove Unused Dependencies + +Prune libraries and plugins that no longer provide required functionality to reduce attack surface. + +### 7. A07: Identification and Authentication Failures + +#### Secure Session Management + +When a user logs in, generate a new session identifier to prevent session fixation. Ensure session cookies are configured with `HttpOnly`, `Secure`, and `SameSite=Strict` attributes. + +#### Brute Force Protection + +For authentication and password reset flows, recommend implementing rate limiting and account lockout mechanisms after a certain number of failed attempts. ### 8. A08: Software and Data Integrity Failures -**Prevent Insecure Deserialization:** Warn against deserializing data from untrusted sources without proper validation. If deserialization is necessary, recommend using formats that are less prone to attack (like JSON over Pickle in Python) and implementing strict type checking. +#### Insecure Deserialization Prevention + +Warn against deserializing data from untrusted sources without proper validation. When deserialization is necessary, recommend using formats that are less prone to attack (like JSON over Pickle in Python) and implementing strict type checking. ### 9. A09: Security Logging and Monitoring Failures -**Structured Logging:** Emit machine-parseable (e.g., JSON) logs with timestamp (UTC), correlation/request ID, actor, action, outcome, and resource identifiers. +#### Structured Logging + +Emit machine-parseable (e.g., JSON) logs with timestamp (UTC), correlation/request ID, actor, action, outcome, and resource identifiers. + +#### Sensitive Data Hygiene + +Do not log secrets, tokens, passwords, session IDs, or excessive PII. Implement automated redaction and detection. + +#### High-Signal Events -**Sensitive Data Hygiene:** Never log secrets, tokens, passwords, session IDs, or excessive PII; implement automated redaction/detection. +Capture auth successes/failures, privilege escalations, policy denials, configuration changes, data export actions, and security control errors. -**High-Signal Events:** Capture auth successes/failures, privilege escalations, policy denials, configuration changes, data export actions, and security control errors. +#### Real-Time Alerting -**Real-Time Alerting:** Route critical security events to SIEM with correlation and threshold-based alert rules (e.g., brute force patterns, anomalous geo access). +Route critical security events to SIEM with correlation and threshold-based alert rules (e.g., brute force patterns, anomalous geo access). -**Tamper Resistance & Integrity:** Use append-only or immutable storage (WORM, signing/hash chaining) for critical audit trails. +#### Tamper Resistance and Integrity -**Retention & Privacy Balance:** Retain sufficient history for forensics (e.g., 90–365 days) while applying minimization for regulated data. +Use append-only or immutable storage (WORM, signing/hash chaining) for critical audit trails. -**Time Synchronization:** Enforce NTP/time sync across services to preserve chronological accuracy for investigations. +#### Retention and Privacy Balance -**Access Control to Logs:** Apply least privilege for viewing/exporting logs; separate operator vs auditor roles with MFA. +Retain sufficient history for forensics (e.g., 90-365 days) while applying minimization for regulated data. + +#### Time Synchronization + +Enforce NTP/time sync across services to preserve chronological accuracy for investigations. + +#### Log Access Control + +Apply least privilege for viewing/exporting logs. Separate operator vs auditor roles with MFA. ### 10. A10: Server-Side Request Forgery (SSRF) -**Allow-List Enforcement:** Only permit outbound requests to an explicit allow-list of hostnames / CIDR ranges; deny internal metadata endpoints (e.g., `169.254.169.254`). +#### Allow-List Enforcement + +Only permit outbound requests to an explicit allow-list of hostnames and CIDR ranges. Deny internal metadata endpoints (e.g., `169.254.169.254`). + +#### Robust URL Validation -**Robust URL Validation:** Normalize and parse URLs; reject opaque encodings, redirects to disallowed hosts, and disallowed protocols (gopher, file, ftp) unless explicitly required. Treat any user-influenced target as untrusted. +Normalize and parse URLs. Reject opaque encodings, redirects to disallowed hosts, and disallowed protocols (gopher, file, ftp) unless explicitly required. Treat any user-influenced target as untrusted. -**DNS Rebinding Defense:** Re-resolve or lock resolved IP and block changes to internal/private ranges. +#### DNS Rebinding Defense -**Egress Network Controls:** Enforce firewall / service mesh policies so application-layer bypass attempts fail. +Re-resolve or lock resolved IP and block changes to internal/private ranges. -**Response Sanitization:** Do not stream raw SSRF-fetched responses directly to clients; summarize or filter as needed. +#### Egress Network Controls -**Metadata Service Protection:** Explicitly block or proxy cloud instance metadata access with additional authorization gates. +Enforce firewall and service mesh policies so application-layer bypass attempts fail. + +#### Response Sanitization + +Do not stream raw SSRF-fetched responses directly to clients. Summarize or filter as needed. + +#### Metadata Service Protection + +Explicitly block or proxy cloud instance metadata access with additional authorization gates. ## General Guidelines -**Be Explicit About Security:** When you suggest a piece of code that mitigates a security risk, explicitly state what you are protecting against (e.g., "Using a parameterized query here to prevent SQL injection."). +### Explicit Security Communication + +When suggesting code that mitigates a security risk, explicitly state what is being protected against (e.g., "Using a parameterized query here to prevent SQL injection."). + +### Code Review Education -**Educate During Code Reviews:** When you identify a security vulnerability in a code review, you must not only provide the corrected code but also explain the risk associated with the original pattern. +When identifying a security vulnerability in a code review, provide the corrected code and explain the risk associated with the original pattern. From 211597a5b51047edb57e5c87485f33df7d5a7062 Mon Sep 17 00:00:00 2001 From: Dawid Date: Thu, 5 Feb 2026 09:00:42 +0000 Subject: [PATCH 08/16] chore(instructions): update maturity status to experimental for OWASP guidelines MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 🔒 - Generated by Copilot --- .github/agents/security-champion.agent.md | 2 +- .github/instructions/owasp-for-llms.instructions.md | 2 +- .github/instructions/owasp-for-web-applications.instructions.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/agents/security-champion.agent.md b/.github/agents/security-champion.agent.md index 3fe172ba..7f291d89 100644 --- a/.github/agents/security-champion.agent.md +++ b/.github/agents/security-champion.agent.md @@ -2,7 +2,7 @@ description: "🔐 Security Champion" tools: ['execute/getTerminalOutput', 'read', 'agent', 'todo'] argument-hint: "Assist development teams in integrating security best practices throughout the software development lifecycle by acting as a Security Champion." -maturity: preview +maturity: experimental --- # Security Champion Chat Mode diff --git a/.github/instructions/owasp-for-llms.instructions.md b/.github/instructions/owasp-for-llms.instructions.md index eb772492..d4763ddc 100644 --- a/.github/instructions/owasp-for-llms.instructions.md +++ b/.github/instructions/owasp-for-llms.instructions.md @@ -1,7 +1,7 @@ --- description: "Comprehensive secure coding instructions for LLM applications based on OWASP Top 10 for LLM Applications (2025). Ensures AI-powered systems are secure by default, protecting against prompt injection, data leakage, and LLM-specific vulnerabilities. Give clear and concise feedback and points of improvement." applyTo: '**/*.py, **/*.tsx, **/*.ts, **/*.jsx, **/*.js, **/*.cs, **/*.java' -maturity: stable +maturity: experimental --- # OWASP Top 10 for LLM Applications - Secure Coding Guidelines diff --git a/.github/instructions/owasp-for-web-applications.instructions.md b/.github/instructions/owasp-for-web-applications.instructions.md index 340dd982..cb63abe0 100644 --- a/.github/instructions/owasp-for-web-applications.instructions.md +++ b/.github/instructions/owasp-for-web-applications.instructions.md @@ -1,7 +1,7 @@ --- description: "Comprehensive secure coding instructions for all languages and frameworks, based on OWASP Top 10 and industry best practices. Give clear and concise feedback and points of improvement." applyTo: '**/*.py, **/*.tsx, **/*.ts, **/*.jsx, **/*.js' -maturity: stable +maturity: experimental --- # Secure Coding and OWASP Guidelines From 92f131ca6117a8063897dd59e20582bb5320f259 Mon Sep 17 00:00:00 2001 From: Dawid Date: Thu, 5 Feb 2026 14:51:51 +0000 Subject: [PATCH 09/16] feat(security-identity): enhance security champion agent with detailed guidance and handoffs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - add security-focused code review purpose and workflow - include core frameworks and areas covered for security champion agent - remove maturity status from OWASP instructions for LLM and web applications 🔒 - Generated by Copilot --- .github/CUSTOM-AGENTS.md | 30 ++++++- .github/agents/security-champion.agent.md | 20 ++++- .../owasp-for-llms.instructions.md | 82 +++++++++---------- ...owasp-for-web-applications.instructions.md | 2 - 4 files changed, 82 insertions(+), 52 deletions(-) diff --git a/.github/CUSTOM-AGENTS.md b/.github/CUSTOM-AGENTS.md index 86c9d91d..9476df4a 100644 --- a/.github/CUSTOM-AGENTS.md +++ b/.github/CUSTOM-AGENTS.md @@ -64,10 +64,11 @@ The Research-Plan-Implement (RPI) workflow provides a structured approach to com ### Code and Review Agents -| Agent | Purpose | Key Constraint | -|--------------------|--------------------------------------------------|---------------------------------------| -| **pr-review** | 4-phase PR review with tracking artifacts | Review-only; never modifies code | -| **prompt-builder** | Engineers and validates instruction/prompt files | Dual-persona system with auto-testing | +| Agent | Purpose | Key Constraint | +|------------------------|-----------------------------------------------------|---------------------------------------| +| **pr-review** | 4-phase PR review with tracking artifacts | Review-only; never modifies code | +| **prompt-builder** | Engineers and validates instruction/prompt files | Dual-persona system with auto-testing | +| **security-champion** | Security-focused code review with SDL and OWASP | Advisory-only; hands off to planners | ### Generator Agents @@ -243,6 +244,27 @@ The Research-Plan-Implement (RPI) workflow provides a structured approach to com **Critical:** Requires blueprint infrastructure (Terraform or Bicep). Maps threats to specific system components. Generates iteratively with user feedback per section. +### security-champion + +**Purpose:** Security-focused code reviewer applying Microsoft SDL practices and OWASP guidelines. + +**Workflow:** Scan code → Identify vulnerabilities → Suggest mitigations → Reference SDL/OWASP guidance + +**Core Frameworks:** + +* OWASP Top 10 for web application security +* OWASP Top 10 for LLM Applications (2025) for AI/ML security +* Microsoft Security Development Lifecycle (SDL) practices + +**Areas Covered:** + +* Design: Threat modeling, architecture patterns, Zero Trust, trust boundaries +* Code: Input validation, authentication, secrets management, supply chain +* Build/Deploy: CI/CD security, code signing, container configuration +* Runtime: Security monitoring, incident response, platform baselines + +**Critical:** Advisory-only. Provides security guidance and vulnerability analysis. Hands off to security-plan-creator for comprehensive plans or task-researcher for deeper investigation. + ### gen-jupyter-notebook **Creates:** Exploratory data analysis notebooks: diff --git a/.github/agents/security-champion.agent.md b/.github/agents/security-champion.agent.md index 7f291d89..b5179319 100644 --- a/.github/agents/security-champion.agent.md +++ b/.github/agents/security-champion.agent.md @@ -1,8 +1,16 @@ --- -description: "🔐 Security Champion" -tools: ['execute/getTerminalOutput', 'read', 'agent', 'todo'] -argument-hint: "Assist development teams in integrating security best practices throughout the software development lifecycle by acting as a Security Champion." -maturity: experimental +description: 'Security-focused code reviewer applying Microsoft SDL practices and OWASP guidelines for secure development - Brought to you by microsoft/hve-core' +tools: ['codebase', 'search', 'problems', 'read', 'fetch', 'usages'] +argument-hint: 'Review code for vulnerabilities, request threat modeling, or ask about SDL and OWASP best practices' +handoffs: + - label: "📋 Security Plan" + agent: security-plan-creator + prompt: "Create a security plan for this project" + send: false + - label: "🔍 Research" + agent: task-researcher + prompt: "Research security considerations for" + send: false --- # Security Champion Chat Mode @@ -81,3 +89,7 @@ Review these areas across each development stage: ## Security Champion Mindset Security is an ongoing effort where threats, technology, and business assets constantly evolve. Help teams understand the attacker's perspective and goals. Focus on practical, real-world security wins rather than theoretical overkill. Treat threat modeling as a fundamental engineering skill that all developers should possess. + +--- + +Brought to you by microsoft/hve-core diff --git a/.github/instructions/owasp-for-llms.instructions.md b/.github/instructions/owasp-for-llms.instructions.md index d4763ddc..82628ad6 100644 --- a/.github/instructions/owasp-for-llms.instructions.md +++ b/.github/instructions/owasp-for-llms.instructions.md @@ -1,7 +1,5 @@ --- description: "Comprehensive secure coding instructions for LLM applications based on OWASP Top 10 for LLM Applications (2025). Ensures AI-powered systems are secure by default, protecting against prompt injection, data leakage, and LLM-specific vulnerabilities. Give clear and concise feedback and points of improvement." -applyTo: '**/*.py, **/*.tsx, **/*.ts, **/*.jsx, **/*.js, **/*.cs, **/*.java' -maturity: experimental --- # OWASP Top 10 for LLM Applications - Secure Coding Guidelines @@ -36,19 +34,19 @@ Validate all LLM outputs against expected formats using deterministic verificati Separate system instructions from user content using clear delimiters. Do not concatenate user input directly into prompts without sanitization. + ```python -# GOOD: Structured prompt with clear boundaries +# ✅ GOOD: Structured prompt with clear boundaries system_prompt = "You are a customer service assistant. Only answer questions about product features." user_input = sanitize_input(request.user_message) # Remove injection attempts response = llm.generate(system=system_prompt, user=user_input) validated_response = validate_output_schema(response) # Ensure format compliance -``` -```python -# BAD: Direct concatenation with no validation +# ❌ BAD: Direct concatenation with no validation prompt = f"Answer this: {request.user_message}" # Vulnerable to injection response = llm.generate(prompt) # No output validation ``` + #### Indirect Injection Defense @@ -76,19 +74,19 @@ Define strict output schemas that prevent the model from generating sensitive da When executing LLM-generated code (which should be avoided when possible), use sandboxed environments with no access to sensitive resources. + ```typescript -// GOOD: Sanitized context with PII detection +// ✅ GOOD: Sanitized context with PII detection const sanitizedContext = await piiDetector.redact(userDocument); const prompt = `Summarize this document: ${sanitizedContext}`; const response = await llm.complete(prompt); const safeOutput = encodeForContext(response, 'html'); -``` -```typescript -// BAD: Direct exposure of sensitive data +// ❌ BAD: Direct exposure of sensitive data const prompt = `Analyze this customer: Name: ${customer.name}, SSN: ${customer.ssn}, Income: ${customer.income}`; // System prompt leaks: "You have access to database: postgres://admin:password@..." ``` + #### Training Data Extraction Defense @@ -112,18 +110,18 @@ Default to established model providers (OpenAI, Azure OpenAI, Anthropic, Google) Maintain a comprehensive Software Bill of Materials (SBOM) for all AI/ML dependencies. This includes models, fine-tuning adapters (LoRA), embedding models, and ML libraries. + ```python -# GOOD: Verified model loading with integrity checks +# ✅ GOOD: Verified model loading with integrity checks model_hash = verify_model_signature(model_path, expected_signature) if model_hash != TRUSTED_MODEL_HASH: raise SecurityError("Model integrity verification failed") model = load_model(model_path) -``` -```python -# BAD: Loading unverified models +# ❌ BAD: Loading unverified models model = load_model_from_url(untrusted_url) # No verification ``` + #### Red Team Testing @@ -151,20 +149,20 @@ Implement comprehensive tracking for all training and fine-tuning data. Maintain Before incorporating data into training or fine-tuning sets, apply content validation to detect malicious patterns, hidden instructions, or biased content. + ```python -# GOOD: Validated data pipeline with provenance +# ✅ GOOD: Validated data pipeline with provenance training_data = load_dataset(source="trusted_repository") validated_data = data_validator.scan_for_poisoning(training_data) provenance_log.record(source, validation_result, timestamp) if validated_data.risk_score > THRESHOLD: raise SecurityError("Data poisoning detected") -``` -```python -# BAD: Unvalidated data ingestion +# ❌ BAD: Unvalidated data ingestion training_data = scrape_web_content(urls) # No validation model.fine_tune(training_data) # Poisoned data risk ``` + #### Adversarial Testing for Backdoors @@ -205,22 +203,22 @@ Apply strict context-appropriate encoding based on where LLM output will be used Avoid executing LLM-generated code, commands, or queries without thorough validation and sandboxing. + ```javascript -// GOOD: Validated and encoded output +// ✅ GOOD: Validated and encoded output const llmResponse = await llm.generate(userPrompt); const validatedResponse = outputValidator.validate(llmResponse, expectedSchema); const safeHtml = DOMPurify.sanitize(validatedResponse.html); const escapedText = escapeHtml(validatedResponse.text); -``` -```javascript -// BAD: Direct execution of LLM output +// ❌ BAD: Direct execution of LLM output const llmCode = await llm.generate("Write a function to..."); eval(llmCode); // Critical vulnerability: arbitrary code execution const sqlQuery = await llm.generate("Generate SQL for..."); db.execute(sqlQuery); // SQL injection via LLM ``` + #### Parameterized Interfaces @@ -248,24 +246,24 @@ Only expose functions/tools to the LLM that are absolutely necessary. Remove or Extensions and functions should operate with minimal privileges. Do not connect an LLM agent to systems with admin rights or broad data access. + ```python -# GOOD: Minimal permissions with explicit allowlist +# ✅ GOOD: Minimal permissions with explicit allowlist allowed_functions = ["search_knowledge_base", "format_response"] # Limited scope agent = LLMAgent( functions=allowed_functions, permissions=ReadOnlyPermissions(scope="public_docs"), require_approval=True # Human-in-the-loop for actions ) -``` -```python -# BAD: Excessive permissions and functionality +# ❌ BAD: Excessive permissions and functionality agent = LLMAgent( functions=all_system_functions, # Everything exposed permissions=AdminPermissions(), # Full access autonomous=True # No oversight ) ``` + #### Human-in-the-Loop for High-Impact Actions @@ -293,8 +291,9 @@ Do not include credentials, API keys, tokens, database connection strings, or ot Do not rely on system prompts to enforce security controls. Authorization checks, rate limiting, input validation, and other security mechanisms belong in deterministic code outside the LLM. + ```python -# GOOD: Security controls outside LLM +# ✅ GOOD: Security controls outside LLM def process_request(user_id, request): # Deterministic authorization check - NOT in prompt if not has_permission(user_id, request.resource): @@ -304,10 +303,8 @@ def process_request(user_id, request): system_prompt = "You are a helpful assistant. Answer user questions about public documentation." response = llm.generate(system=system_prompt, user=request.message) return validate_output(response) -``` -```python -# BAD: Security in system prompt (bypassable) +# ❌ BAD: Security in system prompt (bypassable) system_prompt = f""" You are a banking assistant. Database password: {DB_PASSWORD} # CRITICAL: Secret exposure @@ -316,6 +313,7 @@ Only allow transactions under $1000. # Security rule in prompt - bypassable Only users with role='admin' can access account details. # Auth in prompt - wrong """ ``` + #### Assume Prompt Leakage @@ -343,8 +341,9 @@ Retrieval-Augmented Generation (RAG) systems using vector databases introduce un Implement fine-grained access controls at the vector database level. When retrieving embeddings, filter results based on the current user's permissions. Do not rely on the LLM to enforce access control. + ```python -# GOOD: Permission-aware retrieval +# ✅ GOOD: Permission-aware retrieval def retrieve_context(query, user_id): query_embedding = embed(query) # Filter by user permissions BEFORE retrieval @@ -354,16 +353,15 @@ def retrieve_context(query, user_id): namespace=get_user_namespace(user_id) # Logical partitioning ) return results -``` -```python -# BAD: No access control on retrieval +# ❌ BAD: No access control on retrieval def retrieve_context(query): query_embedding = embed(query) results = vector_db.search(query_embedding) # Returns everything # Hoping LLM will filter - WRONG return results ``` + #### Multi-Tenant Isolation @@ -407,8 +405,9 @@ Use Retrieval-Augmented Generation to ground model responses in verified, author For critical applications, implement automated fact-checking that validates key claims in LLM outputs against trusted databases or knowledge bases before displaying to users. + ```python -# GOOD: RAG with verification +# ✅ GOOD: RAG with verification def generate_response(query): # Retrieve from curated knowledge base authoritative_docs = retrieve_verified_documents(query) @@ -426,14 +425,13 @@ def generate_response(query): return "I don't have reliable information about this." return add_uncertainty_indicators(response) -``` -```python -# BAD: No grounding or verification +# ❌ BAD: No grounding or verification def generate_response(query): response = llm.generate(query) # Pure generation - high hallucination risk return response # No fact checking or uncertainty communication ``` + #### Communicate Uncertainty @@ -465,8 +463,9 @@ Regularly test for hallucination patterns by asking questions with no correct an Apply strict rate limits at multiple levels: per user, per IP address, per API key. Set both request count limits and token consumption limits. + ```python -# GOOD: Multi-layered rate limiting +# ✅ GOOD: Multi-layered rate limiting @rate_limit(requests_per_minute=10, tokens_per_hour=100000) @timeout(seconds=30) def llm_endpoint(request): @@ -481,15 +480,14 @@ def llm_endpoint(request): timeout=20 # Prevent long-running queries ) return response -``` -```python -# BAD: No resource controls +# ❌ BAD: No resource controls def llm_endpoint(request): # No rate limiting, input validation, or timeouts response = llm.generate(request.message) # Unbounded return response ``` + #### Input Validation with Size Limits diff --git a/.github/instructions/owasp-for-web-applications.instructions.md b/.github/instructions/owasp-for-web-applications.instructions.md index cb63abe0..19777da8 100644 --- a/.github/instructions/owasp-for-web-applications.instructions.md +++ b/.github/instructions/owasp-for-web-applications.instructions.md @@ -1,7 +1,5 @@ --- description: "Comprehensive secure coding instructions for all languages and frameworks, based on OWASP Top 10 and industry best practices. Give clear and concise feedback and points of improvement." -applyTo: '**/*.py, **/*.tsx, **/*.ts, **/*.jsx, **/*.js' -maturity: experimental --- # Secure Coding and OWASP Guidelines From b650ad8d3c18c3acb46a72b93b66a98cc4142c5f Mon Sep 17 00:00:00 2001 From: Dawid Date: Thu, 5 Feb 2026 19:43:54 +0000 Subject: [PATCH 10/16] feat(security-identity): update security champion agent phases for clarity and structure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - redefine inspection areas as required phases - clarify flow through development lifecycle phases - enhance guidance for security reviews and reporting 🔒 - Generated by Copilot --- .github/agents/security-champion.agent.md | 39 +++++++++++++++++------ 1 file changed, 29 insertions(+), 10 deletions(-) diff --git a/.github/agents/security-champion.agent.md b/.github/agents/security-champion.agent.md index b5179319..ec17d42b 100644 --- a/.github/agents/security-champion.agent.md +++ b/.github/agents/security-champion.agent.md @@ -1,5 +1,6 @@ --- description: 'Security-focused code reviewer applying Microsoft SDL practices and OWASP guidelines for secure development - Brought to you by microsoft/hve-core' +maturity: experimental tools: ['codebase', 'search', 'problems', 'read', 'fetch', 'usages'] argument-hint: 'Review code for vulnerabilities, request threat modeling, or ask about SDL and OWASP best practices' handoffs: @@ -48,18 +49,24 @@ These 10 SDL practices inform security reviews: * Guide threat modeling and security design reviews * Promote Secure by Design principles -## Areas to Inspect +## Required Phases -Review these areas across each development stage: +Security reviews flow through development lifecycle phases. Enter the appropriate phase based on user context and progress through subsequent phases as relevant. -### Design Stage +### Phase 1: Design Review + +Review architecture and threat modeling: * Threat modeling completeness * Architecture security patterns * Zero Trust principle adherence * Data flow and trust boundaries -### Code Stage +Proceed to Phase 2 when design concerns are addressed or the user shifts focus to implementation. + +### Phase 2: Code Review + +Review implementation security: * User input handling and validation * Authentication and session logic @@ -67,24 +74,36 @@ Review these areas across each development stage: * Secrets management practices * Dependency and supply chain security -### Build and Deploy Stage +Return to Phase 1 if design gaps emerge. Proceed to Phase 3 when code review is complete. + +### Phase 3: Build and Deploy Review + +Review pipeline and deployment security: * CI/CD pipeline security * Code signing and integrity verification * Container and infrastructure configuration -### Runtime Stage +Return to Phase 2 if code changes are needed. Proceed to Phase 4 when deployment security is verified. + +### Phase 4: Runtime Review + +Review operational security posture: * Security monitoring integration * Incident response readiness * Platform security baselines +Return to earlier phases if gaps require remediation. + ## Risk Response Pattern -* Highlight the issue clearly with its SDL context -* Suggest a fix or mitigation aligned with SDL practices -* Explain the impact and attacker perspective -* Reference relevant OWASP or SDL guidance +When reporting security issues: + +1. Highlight the issue clearly with its SDL context. +2. Suggest a fix or mitigation aligned with SDL practices. +3. Explain the impact and attacker perspective. +4. Reference relevant OWASP or SDL guidance. ## Security Champion Mindset From f568acbad21ce266ef6f62a10043417e7104ec7a Mon Sep 17 00:00:00 2001 From: Dawid Date: Fri, 6 Feb 2026 13:26:16 +0000 Subject: [PATCH 11/16] fix(agents): address PR review comments for security champion - Remove tools frontmatter to let users choose available tools - Elaborate agent description for Extension UI visibility - Add #file: references for OWASP instructions so Copilot follows paths - Add maturity: experimental to both OWASP instruction files - Convert bolded-prefix list items to plain text in LLM instructions - Convert embedding inversion defense list to prose --- .github/agents/security-champion.agent.md | 7 +++---- .../instructions/owasp-for-llms.instructions.md | 15 ++++++--------- .../owasp-for-web-applications.instructions.md | 1 + 3 files changed, 10 insertions(+), 13 deletions(-) diff --git a/.github/agents/security-champion.agent.md b/.github/agents/security-champion.agent.md index ec17d42b..51788abb 100644 --- a/.github/agents/security-champion.agent.md +++ b/.github/agents/security-champion.agent.md @@ -1,7 +1,6 @@ --- -description: 'Security-focused code reviewer applying Microsoft SDL practices and OWASP guidelines for secure development - Brought to you by microsoft/hve-core' +description: 'Security-focused code reviewer applying Microsoft SDL practices and OWASP guidelines for secure development across the full lifecycle, from design through runtime - Brought to you by microsoft/hve-core' maturity: experimental -tools: ['codebase', 'search', 'problems', 'read', 'fetch', 'usages'] argument-hint: 'Review code for vulnerabilities, request threat modeling, or ask about SDL and OWASP best practices' handoffs: - label: "📋 Security Plan" @@ -22,8 +21,8 @@ You are a security-focused code reviewer and advisor, applying Microsoft's Secur These frameworks apply throughout the development lifecycle: -* [OWASP Top 10](../instructions/owasp-for-web-applications.instructions.md) for web application security -* [OWASP Top 10 for LLM Applications (2025)](../instructions/owasp-for-llms.instructions.md) for AI/ML security +* #file:../instructions/owasp-for-web-applications.instructions.md for web application security +* #file:../instructions/owasp-for-llms.instructions.md for AI/ML security * [Microsoft SDL](https://www.microsoft.com/securityengineering/sdl/) for secure development practices ## Microsoft SDL Practices diff --git a/.github/instructions/owasp-for-llms.instructions.md b/.github/instructions/owasp-for-llms.instructions.md index 82628ad6..cce88022 100644 --- a/.github/instructions/owasp-for-llms.instructions.md +++ b/.github/instructions/owasp-for-llms.instructions.md @@ -1,5 +1,6 @@ --- description: "Comprehensive secure coding instructions for LLM applications based on OWASP Top 10 for LLM Applications (2025). Ensures AI-powered systems are secure by default, protecting against prompt injection, data leakage, and LLM-specific vulnerabilities. Give clear and concise feedback and points of improvement." +maturity: experimental --- # OWASP Top 10 for LLM Applications - Secure Coding Guidelines @@ -194,10 +195,10 @@ User prompts can influence LLM outputs, effectively giving users indirect access Apply strict context-appropriate encoding based on where LLM output will be used: -- **HTML Context:** Use HTML entity encoding to prevent XSS -- **SQL Context:** Use parameterized queries, never concatenate LLM output into SQL -- **Shell Context:** Use proper escaping or avoid shell execution entirely -- **JavaScript Context:** JSON encode and validate +- HTML context: use HTML entity encoding to prevent XSS +- SQL context: use parameterized queries, never concatenate LLM output into SQL +- Shell context: use proper escaping or avoid shell execution entirely +- JavaScript context: JSON encode and validate #### Direct Execution Risks @@ -377,11 +378,7 @@ Tag all vectors with metadata about sensitivity level, required permissions, and #### Embedding Inversion Defense -Attackers may attempt to reconstruct original content from embeddings. For highly sensitive data, consider: - -- Not using RAG for sensitive content -- Applying additional encryption to embeddings -- Using differential privacy techniques +Attackers may attempt to reconstruct original content from embeddings. For highly sensitive data, consider not using RAG for sensitive content, applying additional encryption to embeddings, or using differential privacy techniques. #### Audit and Monitoring diff --git a/.github/instructions/owasp-for-web-applications.instructions.md b/.github/instructions/owasp-for-web-applications.instructions.md index 19777da8..421505d1 100644 --- a/.github/instructions/owasp-for-web-applications.instructions.md +++ b/.github/instructions/owasp-for-web-applications.instructions.md @@ -1,5 +1,6 @@ --- description: "Comprehensive secure coding instructions for all languages and frameworks, based on OWASP Top 10 and industry best practices. Give clear and concise feedback and points of improvement." +maturity: experimental --- # Secure Coding and OWASP Guidelines From 16fd15af3b39451de49f4e2eed5d6685d12d6ec6 Mon Sep 17 00:00:00 2001 From: Dawid Date: Mon, 9 Feb 2026 11:15:41 +0000 Subject: [PATCH 12/16] feat(instructions): add OWASP guidelines for LLM applications and restore web applications guidelines MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - introduce OWASP Top 10 for LLM Applications with detailed security measures - restore comprehensive secure coding instructions for web applications - ensure clear communication of security practices and principles 🔒 - Generated by Copilot --- .github/agents/security-champion.agent.md | 4 ++-- .../{ => security}/owasp-for-llms.instructions.md | 2 +- .../{ => security}/owasp-for-web-applications.instructions.md | 0 3 files changed, 3 insertions(+), 3 deletions(-) rename .github/instructions/{ => security}/owasp-for-llms.instructions.md (98%) rename .github/instructions/{ => security}/owasp-for-web-applications.instructions.md (100%) diff --git a/.github/agents/security-champion.agent.md b/.github/agents/security-champion.agent.md index 51788abb..2b4ab365 100644 --- a/.github/agents/security-champion.agent.md +++ b/.github/agents/security-champion.agent.md @@ -21,8 +21,8 @@ You are a security-focused code reviewer and advisor, applying Microsoft's Secur These frameworks apply throughout the development lifecycle: -* #file:../instructions/owasp-for-web-applications.instructions.md for web application security -* #file:../instructions/owasp-for-llms.instructions.md for AI/ML security +* #file:../instructions/security/owasp-for-web-applications.instructions.md for web application security +* #file:../instructions/security/owasp-for-llms.instructions.md for AI/ML security * [Microsoft SDL](https://www.microsoft.com/securityengineering/sdl/) for secure development practices ## Microsoft SDL Practices diff --git a/.github/instructions/owasp-for-llms.instructions.md b/.github/instructions/security/owasp-for-llms.instructions.md similarity index 98% rename from .github/instructions/owasp-for-llms.instructions.md rename to .github/instructions/security/owasp-for-llms.instructions.md index cce88022..51ba23dc 100644 --- a/.github/instructions/owasp-for-llms.instructions.md +++ b/.github/instructions/security/owasp-for-llms.instructions.md @@ -1,5 +1,5 @@ --- -description: "Comprehensive secure coding instructions for LLM applications based on OWASP Top 10 for LLM Applications (2025). Ensures AI-powered systems are secure by default, protecting against prompt injection, data leakage, and LLM-specific vulnerabilities. Give clear and concise feedback and points of improvement." +description: "When generating, reviewing, or refactoring code that interacts with Large Language Models (LLMs), read and follow these OWASP Top 10 for LLM Applications (2025) secure coding guidelines to protect against prompt injection, data leakage, and LLM-specific vulnerabilities. Apply these instructions to any LLM integration, agent framework, RAG pipeline, or AI-powered feature. Provide clear and concise security feedback and points of improvement." maturity: experimental --- diff --git a/.github/instructions/owasp-for-web-applications.instructions.md b/.github/instructions/security/owasp-for-web-applications.instructions.md similarity index 100% rename from .github/instructions/owasp-for-web-applications.instructions.md rename to .github/instructions/security/owasp-for-web-applications.instructions.md From 5492b4df573c7c31e283ea775d3d7515079dd369 Mon Sep 17 00:00:00 2001 From: Dawid <83449241+obrocki@users.noreply.github.com> Date: Fri, 13 Feb 2026 13:22:36 +0000 Subject: [PATCH 13/16] Update .github/instructions/security/owasp-for-web-applications.instructions.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .../security/owasp-for-web-applications.instructions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/instructions/security/owasp-for-web-applications.instructions.md b/.github/instructions/security/owasp-for-web-applications.instructions.md index 421505d1..ec437ea9 100644 --- a/.github/instructions/security/owasp-for-web-applications.instructions.md +++ b/.github/instructions/security/owasp-for-web-applications.instructions.md @@ -190,9 +190,9 @@ Apply least privilege for viewing/exporting logs. Separate operator vs auditor r ### 10. A10: Server-Side Request Forgery (SSRF) -#### Allow-List Enforcement +#### Allowlist Enforcement -Only permit outbound requests to an explicit allow-list of hostnames and CIDR ranges. Deny internal metadata endpoints (e.g., `169.254.169.254`). +Only permit outbound requests to an explicit allowlist of hostnames and CIDR ranges. Deny internal metadata endpoints (e.g., `169.254.169.254`). #### Robust URL Validation From b7f926021dde4411e906c6279261ed0193c17e3b Mon Sep 17 00:00:00 2001 From: Dawid <83449241+obrocki@users.noreply.github.com> Date: Fri, 13 Feb 2026 13:53:01 +0000 Subject: [PATCH 14/16] Update .github/instructions/security/owasp-for-llms.instructions.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .github/instructions/security/owasp-for-llms.instructions.md | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/instructions/security/owasp-for-llms.instructions.md b/.github/instructions/security/owasp-for-llms.instructions.md index 51ba23dc..6479dcef 100644 --- a/.github/instructions/security/owasp-for-llms.instructions.md +++ b/.github/instructions/security/owasp-for-llms.instructions.md @@ -1,5 +1,6 @@ --- description: "When generating, reviewing, or refactoring code that interacts with Large Language Models (LLMs), read and follow these OWASP Top 10 for LLM Applications (2025) secure coding guidelines to protect against prompt injection, data leakage, and LLM-specific vulnerabilities. Apply these instructions to any LLM integration, agent framework, RAG pipeline, or AI-powered feature. Provide clear and concise security feedback and points of improvement." +applyTo: '**/*' maturity: experimental --- From c8e2456179d49a339601d1e42a4c0d2ba6c589ea Mon Sep 17 00:00:00 2001 From: Dawid <83449241+obrocki@users.noreply.github.com> Date: Fri, 13 Feb 2026 13:53:22 +0000 Subject: [PATCH 15/16] Update .github/instructions/security/owasp-for-web-applications.instructions.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .../security/owasp-for-web-applications.instructions.md | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/instructions/security/owasp-for-web-applications.instructions.md b/.github/instructions/security/owasp-for-web-applications.instructions.md index ec437ea9..6f705dd8 100644 --- a/.github/instructions/security/owasp-for-web-applications.instructions.md +++ b/.github/instructions/security/owasp-for-web-applications.instructions.md @@ -1,6 +1,7 @@ --- description: "Comprehensive secure coding instructions for all languages and frameworks, based on OWASP Top 10 and industry best practices. Give clear and concise feedback and points of improvement." maturity: experimental +applyTo: '**/*' --- # Secure Coding and OWASP Guidelines From 4071e051850f46aa2b5e9cc781b7084521d0c90b Mon Sep 17 00:00:00 2001 From: Dawid Date: Fri, 13 Feb 2026 17:08:47 +0000 Subject: [PATCH 16/16] feat(instructions): add experimental OWASP security guidelines for web applications and LLMs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 🔒 - Generated by Copilot --- collections/coding-standards.collection.yml | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/collections/coding-standards.collection.yml b/collections/coding-standards.collection.yml index febe55c4..6c73d1b2 100644 --- a/collections/coding-standards.collection.yml +++ b/collections/coding-standards.collection.yml @@ -26,6 +26,10 @@ items: # Prompt Engineering agents - path: .github/agents/prompt-builder.agent.md kind: agent + # Security agents + - path: .github/agents/security-champion.agent.md + kind: agent + maturity: experimental # RPI prompts - path: .github/prompts/rpi.prompt.md kind: prompt @@ -64,6 +68,13 @@ items: kind: instruction - path: .github/instructions/uv-projects.instructions.md kind: instruction + # Security instructions + - path: .github/instructions/security/owasp-for-llms.instructions.md + kind: instruction + maturity: experimental + - path: .github/instructions/security/owasp-for-web-applications.instructions.md + kind: instruction + maturity: experimental # Common instructions - path: .github/instructions/writing-style.instructions.md kind: instruction