Denial Of Service with Mutual Tls

In a mutual TLS (mTLS) handshake both parties present certificates, which forces the server to perform expensive cryptographic operations: verifying the client certificate chain, checking revocation status, and validating signatures. An attacker can abuse this by flooding the server with malformed or low‑cost ClientHello messages that still trigger full certificate verification.

• ClientHello flood – The attacker opens many TCP connections and sends a ClientHello that advertises a weak or absent client certificate. The server must allocate memory, parse the extensions, and attempt certificate verification for each connection, consuming CPU and file descriptors.
• Renegotiation abuse – If the server allows renegotiation, an attacker can request a new handshake after an initial legitimate mTLS exchange, forcing the server to repeat certificate verification without tearing down the underlying TCP stream.
• Session ticket exhaustion – mTLS implementations often issue session tickets to resume handshakes. By presenting unique, never‑seen client certificates in each ClientHello, the attacker prevents ticket reuse, causing the server to generate a new ticket (and perform signing operations) for every connection.
• Certificate chain validation overload – Supplying a client certificate with a long, deep chain (many intermediate CAs) increases the verification work per handshake.

These patterns map to OWASP API Security Top 10 2023 category API4: Lack of Resources & Rate Limiting, because the server’s resources (CPU, memory, file descriptors) are exhausted without any application‑level throttling.

middleBrick performs unauthenticated black‑box scanning of the API endpoint’s TLS surface. To detect mTLS‑focused DoS weaknesses it:

1. Establishes a TCP connection to the target host and port.
2. Sends a rapid burst (e.g., 100 connections per second) of ClientHello messages that either omit the client certificate extension or present a self‑signed certificate that fails validation.
3. Measures the server’s response time and tracks whether connections are accepted, reset, or left hanging.
4. Checks for mitigations such as rate limiting on new TLS handshakes, session ticket caching, and disabled renegotiation.
5. Reports findings with severity based on observed latency growth and connection drop rates.

For example, a scan might return:

Finding	Severity	Description
Unrestricted mTLS handshake rate	High	The server accepted >80 ClientHello floods per second without throttling, leading to rising CPU usage.
Session ticket caching disabled	Medium	Each unique client certificate forced a new ticket sign operation, increasing cryptographic load.
TLS renegotiation allowed	Low	The server permitted post‑handshake renegotiation, enabling an attacker to trigger extra verification steps.

These results appear in the middleBrick dashboard under the "Denial of Service" category, with remediation guidance linked to the specific missing control.

Mitigations focus on limiting the cost of each handshake and ensuring the server cannot be overwhelmed by excessive verification work. Below are concrete, language‑specific examples that use only the native TLS libraries.

Node.js (tls module)

const tls = require('tls');
const fs = require('fs');

const options = {
  key: fs.readFileSync('server-key.pem'),
  cert: fs.readFileSync('server-cert.pem'),
  ca: fs.readFileSync('ca.pem'),            // trust store for client certs
  requestCert: true,                       // ask for client cert
  rejectUnauthorized: true,                // abort if client cert invalid
  // --- DoS mitigations ---
  minVersion: 'TLSv1.2',                   // avoid weak versions
  maxVersion: 'TLSv1.3',                   // limit to modern versions
  // Disable renegotiation (Node.js disables by default in recent versions)
  // Enable session ticket caching (default is on)
  sessionIdContext: 'myapp',               // isolate sessions
  // Limit concurrent connections via net.Server (see below)
};

const server = tls.createServer(options, (socket) => {
  socket.write('welcome\n');
  socket.pipe(socket);
});

// Connection‑level throttling (optional but effective)
server.maxConnections = 200;               // reject new TCP SYN after limit
server.connections = 0;
server.on('connection', (socket) => {
  server.connections++;
  socket.on('close', () => { server.connections--; });
});

server.listen(8443, () => {
  console.log('mTLS server listening on 8443');
});

Go (crypto/tls)

package main

import (
	"crypto/tls"
	"net"
	"log"
)

func main() {
	cert, err := tls.LoadX509KeyPair("server-cert.pem", "server-key.pem")
	if err != nil {
		log.Fatalf("load keys: %v", err)
	}
	
	config := &tls.Config{
		Certificates: []tls.Certificate{cert},
		ClientAuth:   tls.RequireAndVerifyClientCert,
		ClientCAs:    loadCAPool(), // helper to read ca.pem
		MinVersion:   tls.VersionTLS12,
		MaxVersion:   tls.VersionTLS13,
		// Session ticket caching is enabled by default; provide a custom cache if needed
		SessionTicketsDisabled: false,
		// Disable renegotiation (Go 1.19+ disables by default; explicitly set)
		Renegotiation:          tls.RenegotiateNever,
	}

	ln, err := tls.Listen("tcp", ":8443", config)
	if err != nil {
		log.Fatalf("listen: %v", err)
	}
	defer ln.Close()

	// Simple connection counter for basic rate limiting
	var connCount int
	for {
		conn, err := ln.Accept()
		if err != nil {
			if netErr, ok := err.(net.Error); ok && netErr.Temporary() {
				continue
			}
			log.Printf("accept error: %v", err)
			break
		}
		go func(c net.Conn) {
			// per‑connection goroutine; enforce a soft limit
			if connCount > 500 {
				c.Close()
				return
			}
			connCount++
			defer func() { connCount-- }()
			// handle the mTLS connection (read/write)
			buf := make([]byte, 1024)
			for {
				_, err := c.Read(buf)
				if err != nil {
					break
				}
				c.Write(buf) // echo
			}
		}(conn)
	}
}

func loadCAPool() *tls.CertPool {
	pool := tls.NewCertPool()
	ca, err := os.ReadFile("ca.pem")
	if err != nil {
		log.Fatalf("read ca: %v", err)
	}
	if ok := pool.AppendCertsFromPEM(ca); !ok {
		log.Fatalf("failed to parse ca")
	}
	return pool
}

Key points:

• Set minVersion/MaxVersion to avoid weak TLS versions that are cheaper to attack.
• Enable SessionTicketsDisabled: false (default) so resumed handshakes skip certificate verification.
• Disable renegotiation (RenegotiateNever in Go, default off in recent Node.js).
• Apply connection‑level limits (maxConnections in Node.js, a simple counter in Go) to throttle new TCP/TLS handshakes.
• Use a reasonable client CA pool; reject unauthenticated certificates early to avoid wasted chain verification.

After applying these controls, a middleBrick rescan should show the handshake rate finding downgraded to "Low" or "Info", confirming that the server now resists mTLS‑focused DoS attempts.