Hallucination Attacks in Django with Cockroachdb

Hallucination attacks in Django with CockroachDB occur when an application exposes model fields or query hints that mislead downstream models or client-side logic into generating false but plausible outputs. In Django, this commonly surfaces when developers use CharField or TextField to store user-facing summaries or auto-generated descriptions and then allow models to be queried or ordered in ways that imply a factual guarantee. CockroachDB, as a distributed SQL database, does not cause hallucinations by itself, but its SQL semantics and compatibility with Django’s ORM can amplify risks when queries rely on incomplete joins, window functions, or serialized JSON fields that are interpreted as authoritative.

For example, consider a Django model that stores product facts in a JSONB column on CockroachDB and exposes a computed field via a database view or a raw SQL annotation. If the view merges multiple rows using STRING_AGG or ARRAY_AGG without clear provenance, an LLM-facing layer might treat the concatenated result as a single fact. An attacker can supply crafted inputs that cause row duplication or ordering shifts, leading the application to present aggregated snippets as definitive evidence. This is especially relevant when Django’s QuerySet chaining is used to build dynamic prompts, where .filter() and .annotate() expressions may silently omit rows due to CockroachDB’s strict consistency checks across distributed ranges, creating gaps that an LLM can fill with convincing fabrications.

Another vector involves Django’s F expressions and Case/When annotations evaluated on CockroachDB. If business logic uses conditional annotations to derive a status or score, and that status is later presented as a ground-truth label, an adversary can probe boundary conditions (e.g., NULL handling or transaction isolation) to force a model to hallucinate a canonical state. Because CockroachDB serializable isolation can delay visibility of concurrent writes, a Django view that reads immediately after a write may present a partially merged dataset, which an LLM interface might misrepresent as a complete and consistent record.

To detect these risks, middleBrick scans Django endpoints that interact with CockroachDB, examining query patterns, JSON schema usage, and annotation chains for implicit assertions of correctness. The scanner checks whether raw SQL or ORM constructs implicitly claim factual completeness, and flags areas where output could be misused for prompt injection or data exfiltration. Findings include concrete remediation guidance to ensure that any data presented to LLM interfaces is explicitly framed as partial or probabilistic, rather than authoritative.

Remediation focuses on making data lineage explicit, avoiding authoritative aggregation, and ensuring that any content passed to LLM interfaces is clearly marked as incomplete or contextual. Below are concrete Django patterns that work safely with CockroachDB.

1. Use explicit source tracking for JSONB fields

When storing structured facts in JSONB, include metadata that identifies origin and confidence. Define a model with a schema that records provenance:

from django.db import models

class Fact(models.Model):
    entity_id = models.UUIDField()
    payload = models.JSONField()  # {"value": "...", "source": "api_v1", "confidence": 0.7}
    created_at = models.DateTimeField(auto_now_add=True)

    class Meta:
        indexes = [
            models.Index(fields=['entity_id', '-created_at']),
        ]

When querying, always project the source and confidence instead of presenting payload__value as a standalone fact:

from django.db.models import F, Value
from django.db.models.functions import Coalesce

qs = Fact.objects.filter(entity_id=some_id).annotate(
    display_value=Coalesce(F('payload__value'), Value('')),
    provenance=Coalesce(F('payload__source'), Value('unknown')),
).values('display_value', 'provenance')

2. Avoid authoritative aggregation in views

If you use CockroachDB SQL functions via raw SQL or Func expressions, wrap results with disclaimers and avoid presenting concatenated lists as definitive. For example, instead of exposing STRING_AGG(tag, ', ') AS tags as a canonical list, return an array and a flag indicating multiplicity:

from django.db import connection

def get_entity_tags_safe(entity_id):
    with connection.cursor() as cursor:
        cursor.execute("""
            SELECT array_agg(tag ORDER BY tag) AS tags, count(*) AS cnt
            FROM entity_tags
            WHERE entity_id = %s
        """, [entity_id])
        row = cursor.fetchone()
    return {
        'tags': row[0] if row else [],
        'count': row[1] if row else 0,
        'note': 'Tags are aggregated; order may vary across nodes',
    }

3. Handle transaction visibility explicitly

When a Django view writes and then reads within a short window, force a refresh or accept potential staleness rather than implying certainty. Use set_session to adjust isolation when necessary, and annotate reads with a warning:

from django.db import transaction

with transaction.atomic():
    Fact.objects.create(entity_id=uuid4(), payload={'value': 'test', 'source': 'manual', 'confidence': 0.9})

# Explicitly acknowledge possible lag
with transaction.atomic():
    latest = Fact.objects.select_for_update().filter(entity_id=uuid4()).first()
    status = 'fresh' if latest else 'possibly_stale'

4. Validate and sanitize before LLM consumption

Before passing data to an LLM endpoint, ensure that any dynamic prompt includes context about completeness. For instance, if generating a summary from multiple rows, include a prefix that clarifies scope:

def build_prompt(facts):
    fact_lines = [f['display_value'] for f in facts if f.get('confidence', 0) > 0.5]
    prompt = (
        'The following facts are provided with their provenance. '
        'Do not infer missing relations.\n'
    ) + '\n'.join(f'[source: {f["provenance"]}] {f["display_value"]}' for f in facts)
    return prompt

By combining schema discipline, explicit provenance, and cautious aggregation, teams can reduce the surface for hallucination attacks while still leveraging CockroachDB’s scalability from Django.