Memory Leak in Django with Dynamodb

A memory leak in a Django application that uses DynamoDB typically arises not from DynamoDB itself, which is a managed NoSQL service, but from how Django code manages resources and references when interacting with DynamoDB responses and persistent connections. When Django views or services repeatedly query DynamoDB and retain references to large response objects, ORM-like wrappers, or cursors without releasing them, memory usage grows over time. This is especially true when developers cache entire DynamoDB Item dictionaries or use lazy-loading patterns that keep references alive longer than necessary.

Another contributing factor is the use of long-lived HTTP sessions or connection pools to DynamoDB without proper cleanup. If your Django app uses a custom DynamoDB client instantiated at module level and reused across requests, unclosed paginators or iterators (e.g., from scan or query with pagination) can hold onto pages of results in memory. In addition, middleware or signals that attach DynamoDB-derived data to request or session objects can inadvertently create retention chains that prevent garbage collection, leading to increased memory consumption that middleBrick may flag as a Data Exposure or Unsafe Consumption finding due to unpredictable resource usage.

Patterns that commonly trigger this issue include: accumulating query results in a global list for “batch processing,” failing to consume or close generator-based responses, and storing DynamoDB Stream records in in-memory queues without bounds. Because middleBrick performs black-box scanning and OpenAPI analysis, it can detect abnormal response size distributions and missing resource cleanup hints in integration code, correlating findings across the Authentication, Data Exposure, and Unsafe Consumption checks to highlight risky runtime behaviors before they impact availability.

To mitigate memory leaks, ensure every DynamoDB interaction is scoped and cleaned up promptly. Use context managers or explicit close patterns for paginators, avoid retaining full response objects when only a subset of attributes is needed, and prefer lightweight data structures. Below are concrete examples using the AWS SDK for Python (Boto3) within a Django service.

Example 1: Safe paginated query with explicit resource cleanup

import boto3
from django.conf import settings

def fetch_active_users():
    client = boto3.client(
        'dynamodb',
        region_name=settings.AWS_REGION,
        aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
        aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY,
    )
    paginator = client.get_paginator('scan')
    # Use the paginator as a context manager to ensure cleanup
    with paginator.paginate(TableName='users', FilterExpression='attribute_exists(status)') as pages:
        results = []
        for page in pages:
            # Extract only needed fields and avoid retaining the full page dict
            for item in page.get('Items', []):
                results.append({
                    'user_id': item['user_id']['S'],
                    'email': item['email']['S'],
                })
        return results
    # Exiting the 'with' block ensures page resources are released

Example 2: Stream consumer with bounded queue and explicit checkpointing

import boto3
from collections import deque
from django.dispatch import receiver
from myapp.signals import process_dynamodb_record

MAX_QUEUE_SIZE = 100
record_queue = deque(maxlen=MAX_QUEUE_SIZE)

def consume_stream():
    client = boto3.client('dynamodb', region_name='us-east-1')
    # Use a limited queue to avoid unbounded memory growth
    response = client.get_shard_iterator(
        StreamArn='arn:aws:dynamodb:us-east-1:123456789011:table/my-table/stream/2023-01-01T00:00:00',
        ShardId='shardId-00000001541642345678936877852536',
        ShardIteratorType='TRIM_HORIZON',
    )
    shard_iterator = response['ShardIterator']
    while shard_iterator:
        resp = client.get_records(ShardIterator=shard_iterator, Limit=100)
        for rec in resp['Records']:
            # Process and immediately emit, avoiding long retention
            process_dynamodb_record.send(record=rec)
        shard_iterator = resp.get('NextShardIterator')
        # Clear resp to release memory
        del resp

Example 3: Scoped client usage within a Django view to avoid cross-request retention

from django.http import JsonResponse
import boto3

def user_detail(request, user_id):
    # Create a short-lived client per request or use a connection pool with limits
    client = boto3.resource('dynamodb', region_name='eu-west-1')
    table = client.Table('users')
    response = table.get_item(Key={'user_id': user_id})
    item = response.get('Item')
    # Do not attach the full response or raw item to request/session
    return JsonResponse({'user_id': item.get('user_id'), 'email': item.get('email')})

General practices

• Avoid global or module-level accumulators for DynamoDB results; if needed, use bounded structures and periodic flushes.
• Explicitly close or use context managers for paginators and iterators.
• Profile memory in staging using tools compatible with your stack to validate that remediation reduces growth; middleBrick can be run periodically to monitor for regressions in Data Exposure and Unsafe Consumption categories.