Memory Leak on Docker

In a Docker environment a memory leak usually appears as a steady increase in the resident memory of a container or of the Docker daemon itself, eventually triggering an OOM kill or degrading host performance. The leak can originate from three places:

• Application code inside the container – e.g., a Node.js service that creates timers or buffers without clearing them, or a Go program that holds onto HTTP response bodies.
• Container runtime components – containerd, runc, or the Docker daemon may fail to release memory after certain operations. A real‑world example is CVE-2022-23648, a memory leak in containerd when pulling malformed image layers.
• Docker Engine API usage – scripts that repeatedly call endpoints such as POST /containers/create or POST /images/pull without cleaning up the created containers or images can cause the daemon to retain allocated buffers.

Consider a simple Go utility that inspects containers but forgets to close the HTTP response body:

package main
import (
	"net/http"
)
func main() {
	resp, _ := http.Get("http://localhost:2376/containers/json")
	// resp.Body is never closed → leak per request
	_ = resp
}

When this binary runs inside a container and is invoked repeatedly (e.g., via a cron job), each call consumes a few kilobytes that are never freed, leading to a measurable upward trend in docker stats output.

Detecting a memory leak in Docker starts with observing container resource usage and then confirming whether the API surface behaves as expected.

• docker stats <container-id> shows live memory consumption; a constantly rising line indicates a leak.
• docker events can reveal a high frequency of container create/delete actions that are not being cleaned up.
• Prometheus + cAdvisor expose container_memory_usage_bytes; alerts on a sustained upward slope help catch leaks early.

middleBrick can be pointed at the Docker Engine API (typically exposed on tcp://host:2376 or via a Unix socket through a proxy) to test for unrestricted resource consumption patterns that often accompany leaks. For example:

middlebrick scan https://docker-host:2376

The scanner runs the 12 parallel checks, including the “Rate Limiting” and “Input Validation” tests. If the API allows unauthenticated, repeated POST /containers/create calls without enforcement, middleBrick will flag it as a potential vector for a memory‑exhaustion attack, providing a severity rating and remediation guidance.

In CI/CD, the GitHub Action can be added to a workflow that builds a Docker image and then runs:

- name: Scan Docker API
  uses: middlebrick/action@v1
  with:
    target: https://staging-docker.example.com:2376
    fail-below: B

If the score drops below the configured threshold (e.g., B), the action fails the build, alerting the team before the image is promoted to production.

Remediation focuses on limiting the impact of a leak and eliminating the root cause where possible.

• Apply hard memory limits – use --memory and --memory-swap on docker run or mem_limit in Compose to cap the container’s RSS. When the limit is reached, the container is OOM‑killed, protecting the host.

  docker run -d --memory=512m --memory-swap=1g myapp

• Restrict PID and file‑descriptor counts – --pids-limit and --ulimit nofile=1024:2048 prevent a leak from spawning endless processes or opening excessive sockets.
• Automate cleanup – add a watchdog script or use Docker’s restart policies to remove stopped containers:

  docker run --restart=unless-stopped -d myapp
  # Periodic cleanup
  0 * * * * root docker container prune -f

• Fix the application – ensure resources are closed. In Go, defer the close; in Node.js, clear intervals and timers:

  // Node.js example
  const interval = setInterval(() => {
    // work
  }, 1000);
  process.on('SIGTERM', () => {
    clearInterval(interval);
    process.exit(0);
  });
  

• Update runtime components – keep containerd and the Docker engine patched. For the containerd leak (CVE‑2022‑23648), upgrading to containerd ≥ 1.6.8 eliminates the issue.
• Leverage multi‑stage builds and distroless images – smaller attack surface reduces the amount of native libraries that could leak.

  FROM golang:1.22-alpine AS builder
  WORKDIR /app
  COPY . .
  RUN go build -o main .
  
  FROM gcr.io/distroless/base-debian12
  COPY --from=builder /app/main /main
  ENTRYPOINT ["/main"]
  

By combining runtime limits, vigilant monitoring, and proper resource handling in the application code, the risk of a memory‑leak‑induced denial of service is reduced to a manageable level.