Unicode Normalization in APIs

Unicode normalization is a process that converts text into a standardized form to ensure consistent representation of characters across different systems. Unicode allows multiple ways to represent the same visual character, which can lead to security vulnerabilities when applications don't properly normalize input.

Consider the character 'é' (e-acute). It can be represented in two ways: as a single precomposed character (U+00E9) or as a base character 'e' (U+0065) combined with a combining acute accent (U+0301). Both produce the same visual output, but they're different byte sequences.

The Unicode standard defines four normalization forms:

• NFD (Normalization Form D): Canonical Decomposition - breaks characters into base characters plus combining marks
• NFC (Normalization Form C): Canonical Decomposition + Canonical Composition - recomposes characters where possible
• NFKD (Normalization Form KD): Compatibility Decomposition - also decomposes compatibility characters
• NFKC (Normalization Form KC): Compatibility Decomposition + Canonical Composition
• NFKD (Normalization Form KD): Compatibility Decomposition - also decomposes compatibility characters
• NFKC (Normalization Form KC): Compatibility Decomposition + Canonical Composition

When APIs don't normalize input before processing, attackers can exploit these variations to bypass security controls, create duplicate accounts, or access unauthorized resources.

APIs that handle user input without proper normalization are vulnerable to several attack patterns. The most common is bypassing authentication and authorization checks.

Consider a web application that stores usernames in a database. If the application doesn't normalize usernames before checking authentication, an attacker could create an account with a username like 'admiń' (using combining characters). Visually, this appears identical to 'admin', but the underlying byte representation differs. When the legitimate admin tries to log in with 'admin', the system might not recognize the difference, potentially allowing unauthorized access.

Another scenario involves URL path traversal. An API endpoint that checks file paths without normalization might be vulnerable to attacks like '/etc/passwd' versus '/etc/́passwd'. The combining character could bypass simple string comparisons designed to prevent directory traversal.

Rate limiting can also be bypassed. If an API enforces rate limits based on unnormalized IP addresses or user identifiers, an attacker could make requests using different Unicode representations to circumvent limits.

Database queries are particularly vulnerable. SQL injection attacks can sometimes bypass input sanitization if the database engine and application handle Unicode normalization differently. An attacker might use Unicode variations to evade simple pattern matching in prepared statements.

Detecting Unicode normalization vulnerabilities requires systematic testing with various Unicode representations. Here's what to look for:

First, test authentication endpoints with Unicode variations of known usernames. Try logging in with precomposed and decomposed forms of usernames to see if the system treats them as equivalent. Check if the application normalizes input before comparing credentials.

For authorization checks, test resource access with Unicode variations in identifiers. If an API uses user IDs or resource names in URLs, try accessing the same resource with different Unicode representations to see if authorization checks pass inconsistently.

middleBrick's API security scanner automatically tests for Unicode normalization issues by submitting requests with various Unicode representations of common characters. The scanner checks if the API treats visually identical Unicode strings differently in authentication, authorization, and rate limiting contexts.

middleBrick specifically looks for:

• Inconsistent handling of precomposed vs decomposed characters
• Authorization bypasses using Unicode variations
• Rate limiting circumvention through Unicode manipulation
• Database query inconsistencies with Unicode input
• Path traversal vulnerabilities using combining characters

The scanner provides a risk score and detailed findings, showing exactly which Unicode variations bypass security controls and how to fix them.

The primary defense against Unicode normalization vulnerabilities is to normalize all user input before processing. Here's how to implement this in different programming environments:

JavaScript (Node.js):

// Normalize to NFC form before processing
const normalizedInput = input.normalize('NFC');

// For case-insensitive comparisons
const normalizedLower = input.normalize('NFC').toLowerCase();

Python:

import unicodedata

# Normalize to NFC form
normalized_input = unicodedata.normalize('NFC', user_input)

# For case-insensitive comparisons
normalized_lower = unicodedata.normalize('NFC', user_input).casefold()

Java:

// Normalize to NFC form
String normalized = Normalizer.normalize(input, Normalizer.Form.NFC);

// For case-insensitive comparisons
String normalizedLower = Normalizer.normalize(input, Normalizer.Form.NFC).toLowerCase();

Database considerations: Ensure your database collation and connection settings handle Unicode consistently. Use Unicode-aware collations like utf8mb4_unicode_ci in MySQL or equivalent in other databases.

API design best practices:

• Always normalize input before authentication checks
• Normalize identifiers used in authorization decisions
• Implement consistent Unicode handling across all API endpoints
• Validate that normalization doesn't alter the semantic meaning of input
• Test with various Unicode representations during development

Additional security measures include input validation that checks for suspicious Unicode patterns, rate limiting based on normalized identifiers, and logging that records both the original and normalized forms of input for audit purposes.

While specific public incidents of Unicode normalization vulnerabilities in APIs are rare in CVE databases, the underlying principle has been exploited in numerous attacks. The classic example is the 'homograph attack' where visually similar Unicode characters are used to create phishing URLs or impersonate legitimate services.

In 2017, a vulnerability in Ruby on Rails' strong parameters allowed attackers to bypass mass assignment protection using Unicode variations. While not directly related to API normalization, it demonstrated how Unicode can be used to circumvent security controls.

Directory traversal attacks using Unicode have been documented since the early 2000s. Attackers use combining characters and full-width characters to bypass simple path validation that doesn't properly normalize input before comparison.

Modern API security scanners like middleBrick help prevent these issues by automatically testing for Unicode normalization vulnerabilities during development and in production. The scanner's findings map to OWASP API Security Top 10 risks, specifically addressing broken object level authorization and authentication flaws that can result from improper Unicode handling.

Organizations that fail to address Unicode normalization vulnerabilities risk account takeover, data exposure, and unauthorized access to sensitive resources. The impact can be severe, especially in systems where usernames, email addresses, or resource identifiers are used for both authentication and authorization decisions.