Memory Leak in Feathersjs with Dynamodb

A memory leak in a FeathersJS service that uses DynamoDB typically arises from unbounded accumulation of references or incomplete asynchronous resource handling, rather than from DynamoDB itself. FeathersJS manages service lifecycles and hooks, and if developers store request-specific data in module-level caches or global objects, or if they fail to properly clean up EventEmitter listeners and pending promises, memory usage can grow over time. When DynamoDB operations—such as scans, queries, or batch gets—are invoked repeatedly within hooks or services without limiting concurrency or backpressure, the in-flight requests and their response buffers may persist longer than necessary, especially under sustained traffic or misconfigured retry strategies.

DynamoDB’s SDK for JavaScript uses streams and asynchronous pagination. If a service uses pagination (e.g., scan with ExclusiveStartKey) inside a hook or a custom method and does not properly destroy or finalize the stream, accumulated pages can remain referenced, preventing garbage collection. Similarly, if responses are cached in-process (for example, in a Map or array keyed by request identifiers) without TTL or eviction policies, memory grows linearly with request volume. Misconfigured connection or retry settings can also lead to an accumulation of pending HTTP agents and socket handles, indirectly increasing memory footprint.

Consider a FeathersJS hook that performs a paginated scan to enrich data before passing control to the next hook. If the hook stores each page’s items in a closure variable or attaches them to the context object without cleanup, each request contributes to a larger heap. Under continuous traffic, the Node.js process may show a steady increase in RSS and heap usage, longer GC pauses, and eventually degraded performance. In a cloud environment where functions may be reused (e.g., AWS Lambda with provisioned concurrency), failure to reset module-level state between invocations can turn a small leak into a significant problem.

FeathersJS’s channel mechanisms can also contribute when broadcasting events that include large DynamoDB payloads; if channels retain references to sockets or subscribers fail to unsubscribe, memory can accumulate. Because DynamoDB responses can be large (especially with ProjectionExpression that returns many attributes), retaining these objects in memory multiplies the impact. Properly managing the lifecycle of hooks, streams, and context objects is essential to prevent memory growth in this stack.

To detect such issues, use runtime monitoring alongside middleBrick’s LLM/AI Security and Data Exposure checks, which can highlight unusual response sizes or unauthentinated endpoints that may exacerbate memory pressure. While middleBrick does not fix leaks, its findings can guide remediation by identifying risky patterns in API behavior and configuration.

Address memory leaks by ensuring streams are destroyed, pagination loops terminate, and caches are bounded. In FeathersJS, prefer configuration-based pagination limits, avoid attaching large payloads to the context, and explicitly clean up resources in hooks destroy or after hooks.

Example 1: Safe paginated scan with cleanup

const { DynamoDB } = require('aws-sdk');
const dynamodb = new DynamoDB.DocumentClient();

class PaginatedItemsService {
  constructor(options) {
    this.options = options || {};
  }

  async find(params) {
    const limit = Math.min(params.pagination?.limit || 10, 100); // enforce a server-side cap
    let lastEvaluatedKey = null;
    let accumulated = [];

    do {
      const input = {
        TableName: process.env.DYNAMODB_TABLE,
        Limit: limit,
        ExclusiveStartKey: lastEvaluatedKey,
        Select: 'SPECIFIC_ATTRIBUTES',
        ProjectionExpression: 'id,content'
      };

      const data = await dynamodb.scan(input).promise();
      accumulated = accumulated.concat(data.Items || []);
      lastEvaluatedKey = data.LastEvaluatedKey || null;

      // Destroy stream references explicitly if using stream API
      if (data.$response && data.$response.request && data.$response.request.httpRequest) {
        const req = data.$response.request.httpRequest;
        if (req.stream) {
          req.stream.destroy();
        }
      }
    } while (lastEvaluatedKey && accumulated.length < limit * 5); // safety cap

    return accumulated.slice(0, limit);
  }
}

// In your FeathersJS service file
const itemsService = new PaginatedItemsService();
module.exports = function (app) {
  const options = {
    name: 'items',
    paginate: { default: 10, max: 50 }
  };
  app.use('/items', Object.assign(itemsService, options));
};

Example 2: Hook-level cleanup and avoiding context bloat

module.exports = function () {
  return async context => {
    // Avoid storing large payloads on context
    const { data } = context;
    const enriched = [];

    for (const item of data) {
      const params = {
        TableName: process.env.DYNAMODB_TABLE,
        Key: { id: item.id }
      };
      const result = await app.service('metadata').find({ query: { id: item.id } });
      // Process only necessary fields
      enriched.push({
        id: item.id,
        title: item.title,
        meta: result.data[0]?.summary || null
      });
    }

    // Replace payload with lightweight shape
    context.result = enriched;
    // Explicitly nullify large intermediate references
    context.params = context.params || {};
    context.params._cleanup = { items: null };
    return context;
  };
};

Example 3: Bounded in-memory cache with TTL

class BoundedCache {
  constructor(limit = 1000, ttlMs = 300000) {
    this.limit = limit;
    this.ttl = ttlMs;
    this.store = new Map();
    this.timestamps = new Map();
  }

  get(key) {
    if (!this.store.has(key)) return undefined;
    if (Date.now() - this.timestamps.get(key) > this.ttl) {
      this.store.delete(key);
      this.timestamps.delete(key);
      return undefined;
    }
    return this.store.get(key);
  }

  set(key, value) {
    if (this.store.size >= this.limit) {
      const oldestKey = this.timestamps.entries().next().value?.[0];
      if (oldestKey) {
        this.store.delete(oldestKey);
        this.timestamps.delete(oldestKey);
      }
    }
    this.store.set(key, value);
    this.timestamps.set(key, Date.now());
  }
}

const responseCache = new BoundedCache(500, 60000);

// Use within a FeathersJS hook cautiously; ensure cache key includes tenant/query context
module.exports = {
  responseCache
};

Operational practices

• Set pagination caps on both client and server; avoid open-ended scans in production hooks.
• Use ProjectionExpression to limit returned attributes and reduce payload size.
• Monitor Node.js heap usage and GC behavior in staging; correlate with request rates.
• Ensure EventEmitter listeners are removed in service teardown or hook destroy functions.

These steps reduce in-memory retention of DynamoDB responses and related objects, mitigating memory growth while preserving functionality.