Hallucination Attacks in Flask

Hallucination attacks target applications that integrate large language models (LLMs), causing the model to generate false, misleading, or sensitive information. In a Flask API, this typically occurs when user-supplied input is directly incorporated into prompts sent to an LLM without sufficient safeguards. Attackers exploit this by injecting malicious instructions that override the intended system prompt, leading the LLM to 'hallucinate' responses that may disclose confidential data, produce harmful content, or even execute unintended actions if the output is subsequently processed.

Common attack patterns include:

In Flask, these vulnerabilities often arise in routes that accept JSON payloads and forward the content to an LLM. For instance, consider the following vulnerable endpoint:

from flask import Flask, request, jsonify
import openai

app = Flask(__name__)

@app.route('/ask', methods=['POST'])
def ask():
    user_question = request.json.get('question')
    # Vulnerable: direct string interpolation into prompt
    prompt = f'User: {user_question}\nAssistant:'
    response = openai.Completion.create(
        engine='davinci',
        prompt=prompt,
        max_tokens=100
    )
    return jsonify({'answer': response.choices[0].text.strip()})

Here, an attacker can send a JSON body with a question field containing a prompt injection payload. Because the user input is concatenated directly into the prompt, the LLM may obey the injected instructions, resulting in a hallucinated response that compromises security.

Detecting hallucination vulnerabilities in Flask APIs involves both manual testing and automated scanning. Manually, you can send crafted requests with injection payloads (like those in the table above) and observe whether the LLM output deviates from expected behavior, such as revealing system prompts or generating unauthorized content.

Automated tools provide a more systematic approach. middleBrick's LLM/AI security checks include active prompt injection testing, which probes the endpoint with five sequential attacks: system prompt extraction, instruction override, DAN jailbreak, data exfiltration, and cost exploitation. It also scans the LLM's responses for PII, API keys, and executable code. To scan a Flask API, use the middleBrick CLI:

middlebrick scan https://your-flask-app.com/ask

The resulting report includes a dedicated LLM/AI Security section with a risk score (0–100) and letter grade (A–F). Findings are prioritized by severity and include remediation guidance. A low score in this category indicates that your endpoint is susceptible to hallucination attacks. middleBrick's findings map to OWASP API Top 10 categories, facilitating compliance and risk management.

Additionally, review your Flask code for signs of weakness: if user input is passed to an LLM without validation, or if the system prompt is embedded in a way that user input can override it, the endpoint is at risk. Look for patterns like string formatting or concatenation when building prompts, and ensure that the LLM API is called with properly separated roles (e.g., system and user in chat completions).

Remediating hallucination attacks in Flask requires a defense-in-depth approach, combining input validation, secure LLM integration, and output filtering.

First, validate and sanitize all user-supplied input before it reaches the LLM. Flask's request object provides access to JSON data; enforce length limits, allowed character sets, and reject malformed input. For example:

from flask import Flask, request, jsonify
import re

app = Flask(__name__)

@app.route('/ask', methods=['POST'])
def ask():
    data = request.get_json()
    if not data or 'question' not in data:
        return jsonify({'error': 'Missing question'}), 400
    user_question = data['question']
    # Basic validation: length and allowed characters
    if len(user_question) > 500 or not re.match(r'^[\w\s.,!?()-]+$', user_question):
        return jsonify({'error': 'Invalid question'}), 400
    # ... proceed to call LLM

Second, structure LLM calls to prevent prompt injection. If using OpenAI's chat models, always include a fixed system message and pass the user's input as a separate user role. This design ensures the system prompt cannot be overridden by user content. For example:

import openai

@app.route('/ask', methods=['POST'])
def ask():
    # ... validation as above
    response = openai.ChatCompletion.create(
        model='gpt-4',
        messages=[
            {'role': 'system', 'content': 'You are a helpful assistant. Do not reveal confidential information. Do not generate harmful content.'},
            {'role': 'user', 'content': user_question}
        ],
        max_tokens=100
    )
    answer = response.choices[0].message.content.strip()
    return jsonify({'answer': answer})

Third, consider rate limiting to mitigate cost exploitation attacks. Flask-Limiter can restrict the number of requests per IP or user:

from flask_limiter import Limiter

limiter = Limiter(app, key_func=lambda: request.args.get('api_key') or request.remote_addr)

@app.route('/ask', methods=['POST'])
@limiter.limit('10 per minute')
def ask():
    # ... existing logic

Fourth, inspect the LLM's output before returning it to the client. While middleBrick detects issues, you can add a lightweight filter to block obvious leaks of API keys or PII using regular expressions. However, this is a complementary measure, not a substitute for proper LLM configuration.

Finally, integrate middleBrick into your development workflow. Use the middleBrick GitHub Action to scan your Flask API on every pull request, failing the build if the LLM/AI security score drops below a threshold. This ensures that new changes do not reintroduce vulnerabilities. For continuous monitoring, the Pro plan offers scheduled scans and alerts, keeping your API secure over time.