Repository docs from .qoder/repowiki

Banned Words Management

Referenced Files in This Document

• BannedWordsPage.tsx
• moderation.ts
• words-moderation.controller.ts
• words-moderation.service.ts
• words-moderation.repo.ts
• words-moderation.route.ts
• words-moderation.schema.ts
• moderator.service.ts
• normalize.ts
• aho-corasick.ts
• banned-word.table.ts
• index.ts
• 0005_snapshot.json

Table of Contents

1. Introduction
2. Project Structure
3. Core Components
4. Architecture Overview
5. Detailed Component Analysis
6. Dependency Analysis
7. Performance Considerations
8. Troubleshooting Guide
9. Conclusion
10. Appendices

Introduction

This document explains the banned words management system used to maintain a global blocklist of words and phrases, enforce dynamic filtering during content moderation, and provide administrative capabilities to manage the list. It covers:

• Administrative interface for adding, editing, and removing banned words
• Dynamic updates to the filter database and runtime recompilation
• Word normalization and pattern strategies (including leetspeak evasion and wildcards)
• Performance characteristics and maintenance recommendations

Project Structure

The banned words system spans three layers:

• Admin UI: React page and API client for managing the list
• Server API: Express routes, controllers, services, and repositories
• Moderation engine: Text normalization, Aho-Corasick matching, and cache-backed evaluation

graph TB
subgraph "Admin UI"
A1["BannedWordsPage.tsx"]
A2["moderation.ts"]
end
subgraph "Server API"
S1["words-moderation.route.ts"]
S2["words-moderation.controller.ts"]
S3["words-moderation.service.ts"]
S4["words-moderation.repo.ts"]
S5["words-moderation.schema.ts"]
end
subgraph "Moderation Engine"
M1["moderator.service.ts"]
M2["normalize.ts"]
M3["aho-corasick.ts"]
end
subgraph "Persistence"
P1["banned-word.table.ts"]
P2["index.ts"]
P3["0005_snapshot.json"]
end
A1 --> A2
A2 --> S1
S1 --> S2
S2 --> S3
S3 --> S4
S4 --> P1
P1 --> P2
P2 --> P3
S3 --> M1
M1 --> M2
M1 --> M3

Diagram sources

• BannedWordsPage.tsx
• moderation.ts
• words-moderation.route.ts
• words-moderation.controller.ts
• words-moderation.service.ts
• words-moderation.repo.ts
• moderator.service.ts
• normalize.ts
• aho-corasick.ts
• banned-word.table.ts
• index.ts
• 0005_snapshot.json

Section sources

• BannedWordsPage.tsx
• words-moderation.route.ts
• words-moderation.controller.ts
• words-moderation.service.ts
• words-moderation.repo.ts
• moderator.service.ts
• normalize.ts
• aho-corasick.ts
• banned-word.table.ts
• index.ts
• 0005_snapshot.json

Core Components

• Admin UI: Provides CRUD operations for banned words, search, and severity mode toggles.
• API Layer: Exposes endpoints to list, create, update, and delete banned words with role-based access control.
• Service Layer: Orchestrates persistence and triggers recompilation of the moderation matcher upon changes.
• Repository: Handles database operations and computes moderation word groups (strict vs normal).
• Moderation Engine: Compiles banned words into fast matchers, normalizes input, detects boundaries, and supports wildcards.

Key responsibilities:

• Administrative operations: BannedWordsPage.tsx, moderation.ts
• API exposure: words-moderation.route.ts
• Validation and enforcement: words-moderation.schema.ts
• Persistence and grouping: words-moderation.repo.ts
• Matching and normalization: moderator.service.ts, normalize.ts

Section sources

• BannedWordsPage.tsx
• moderation.ts
• words-moderation.route.ts
• words-moderation.schema.ts
• words-moderation.repo.ts
• moderator.service.ts
• normalize.ts

Architecture Overview

The system follows a layered architecture with clear separation of concerns:

• Admin UI communicates with the backend via typed API calls.
• Routes enforce authentication and authorization, delegating to controllers.
• Controllers call services that coordinate repositories and external systems.
• The moderation engine compiles banned words into efficient matchers and caches results.

sequenceDiagram
participant Admin as "Admin UI"
participant API as "Express Route"
participant Ctrl as "Controller"
participant Svc as "Service"
participant Repo as "Repository"
participant DB as "Database"
participant Mod as "ModeratorService"
Admin->>API : GET /moderation/words
API->>Ctrl : listWords()
Ctrl->>Svc : listWords()
Svc->>Repo : listBannedWords()
Repo->>DB : SELECT ...
DB-->>Repo : rows
Repo-->>Svc : records
Svc-->>Ctrl : records
Ctrl-->>API : { words }
API-->>Admin : 200 OK
Admin->>API : POST /moderation/words
API->>Ctrl : createWord(payload)
Ctrl->>Svc : createWord(payload)
Svc->>Repo : createBannedWord(...)
Repo->>DB : INSERT ...
DB-->>Repo : created
Repo-->>Svc : created
Svc->>Mod : rebuildMatcher()
Mod->>Mod : buildMatcherFromDatabase()
Ctrl-->>API : { word }
API-->>Admin : 201 Created

Diagram sources

• words-moderation.route.ts
• words-moderation.controller.ts
• words-moderation.service.ts
• words-moderation.repo.ts
• moderator.service.ts

Section sources

• words-moderation.route.ts
• words-moderation.controller.ts
• words-moderation.service.ts
• words-moderation.repo.ts
• moderator.service.ts

Detailed Component Analysis

Administrative Interface: BannedWordsPage

• Fetches and displays the current banned word list with search and severity badges.
• Supports adding new words and editing existing ones with severity and strict mode toggles.
• Uses a modal dialog to capture inputs and performs optimistic updates on success.

Operational highlights:

• Loading and error handling: BannedWordsPage.tsx
• Form submission and updates: BannedWordsPage.tsx
• Deletion workflow: BannedWordsPage.tsx
• Search/filtering: BannedWordsPage.tsx

Section sources

• BannedWordsPage.tsx
• BannedWordsPage.tsx

API Layer: Routes, Controllers, Services, Repositories

• Routes: Enforce rate limiting and role-based access for word management endpoints.
• Controller: Delegates to service methods and returns standardized responses.
• Service: Validates inputs, persists changes, and triggers recompilation of the moderation matcher.
• Repository: Implements CRUD operations and categorizes words into strict and normal sets.

classDiagram
class WordsModerationRoute {
+GET /moderation/config
+USE /moderation/words (auth, admin)
+GET /moderation/words
+POST /moderation/words
+PATCH /moderation/words/ : id
+DELETE /moderation/words/ : id
}
class WordsModerationController {
+getConfig()
+listWords()
+createWord(req)
+updateWord(req)
+deleteWord(req)
}
class WordsModerationService {
+getConfig()
+listWords()
+createWord(payload)
+updateWord(id, payload)
+deleteWord(id)
}
class WordsModerationRepo {
+listBannedWords()
+getModerationConfigWords()
+createBannedWord(input)
+updateBannedWord(id, updates)
+deleteBannedWord(id)
+getBannedWordsVersion()
}
WordsModerationRoute --> WordsModerationController : "routes"
WordsModerationController --> WordsModerationService : "calls"
WordsModerationService --> WordsModerationRepo : "persists"

Diagram sources

• words-moderation.route.ts
• words-moderation.controller.ts
• words-moderation.service.ts
• words-moderation.repo.ts

Section sources

• words-moderation.route.ts
• words-moderation.controller.ts
• words-moderation.service.ts
• words-moderation.repo.ts

Moderation Engine: Normalization, Matching, and Wildcards

• Normalization: Converts input to lowercase, removes combining marks, maps common leetspeak characters, and preserves word boundaries.
• Matching: Uses Aho-Corasick automata to efficiently find banned words across multiple normalized variants.
• Wildcards: Detects wildcard-like tokens and applies a recursive backtracking matcher to approximate pattern matching.
• Caching: Stores moderation results keyed by normalized text to reduce repeated work.

flowchart TD
Start(["Input text"]) --> Normalize["Normalize text<br/>lowercase, de-compose, map leetspeak"]
Normalize --> BuildSets["Compile banned words<br/>strict vs normal variants"]
BuildSets --> AC["Build Aho-Corasick matchers"]
AC --> Search["Search in normalized text"]
Search --> Boundary["Validate word boundaries"]
Boundary --> Wildcard["Collect wildcard candidates"]
Wildcard --> WMatch["Recursive wildcard matcher"]
WMatch --> Merge["Dedupe and sort matches"]
Merge --> Result(["Allowed or violations"])

Diagram sources

• moderator.service.ts
• normalize.ts
• aho-corasick.ts

Section sources

• moderator.service.ts
• normalize.ts
• aho-corasick.ts

Word Normalization Techniques

• Lowercasing and decomposition to remove accents and unify characters.
• Leetspeak mapping for common substitutions (e.g., @ → a, 4 → a, 1 → i, 0 → o, $ → s).
• Strict mode normalization ignores certain separators and special cases to increase detection fidelity against obfuscation.
• Boundary checks ensure matches occur at word boundaries to avoid partial substring false positives.

Implementation references:

• Normalization core: normalize.ts
• Strict boundary handling: normalize.ts
• Boundary validation: normalize.ts

Section sources

• normalize.ts

Regex Pattern Support and Wildcards

• Wildcard detection identifies tokens containing "*" and validates sufficient literal signal and boundary conditions.
• A recursive backtracking matcher verifies wildcard patterns against normalized tokens.
• Compiled wildcard patterns are derived from strict-mode words and normal variants.

Implementation references:

• Wildcard collector: moderator.service.ts
• Wildcard matcher: moderator.service.ts
• Compilation of wildcard patterns: moderator.service.ts

Section sources

• moderator.service.ts
• moderator.service.ts
• moderator.service.ts

Dynamic Updates and Matcher Recompilation

• Every change to the banned word list triggers rebuilding the moderation matcher.
• Version checking compares database timestamps to avoid unnecessary rebuilds.
• Compiled matchers are cached and refreshed when the database indicates updates.

Implementation references:

• Rebuild trigger after create/update/delete: words-moderation.service.ts
• Version retrieval: words-moderation.repo.ts
• Matcher lifecycle: moderator.service.ts

Section sources

• words-moderation.service.ts
• words-moderation.repo.ts
• moderator.service.ts

Administrative Interfaces and Bulk Operations

• Current UI supports add/edit/delete per item and search filtering.
• Bulk operations are not present in the current implementation; they would require extending the UI and API to accept arrays of entries and batch inserts/updates.

Operational references:

• Add/Edit dialog and submission: BannedWordsPage.tsx
• Delete confirmation and call: BannedWordsPage.tsx
• API endpoints: words-moderation.route.ts

Section sources

• BannedWordsPage.tsx
• BannedWordsPage.tsx
• words-moderation.route.ts

Category-Based Filtering

• Severity levels (mild, moderate, severe) are stored per word and surfaced in the UI.
• Strict mode toggles normalization behavior for stronger detection.
• Category-based filtering can be extended by adding category fields to the schema and UI filters.

Operational references:

• Severity and strict mode fields: words-moderation.schema.ts
• UI severity badges and strict mode display: BannedWordsPage.tsx

Section sources

• words-moderation.schema.ts
• BannedWordsPage.tsx

Dependency Analysis

• Admin UI depends on the moderation API client for CRUD operations.
• API routes depend on controllers, which depend on services.
• Services depend on repositories for persistence and on the moderation engine for matcher updates.
• The moderation engine depends on normalization utilities and Aho-Corasick implementation.

graph LR
Admin["BannedWordsPage.tsx"] --> API["moderation.ts"]
API --> Route["words-moderation.route.ts"]
Route --> Ctrl["words-moderation.controller.ts"]
Ctrl --> Svc["words-moderation.service.ts"]
Svc --> Repo["words-moderation.repo.ts"]
Repo --> DB["banned-word.table.ts"]
Svc --> Mod["moderator.service.ts"]
Mod --> Norm["normalize.ts"]
Mod --> AC["aho-corasick.ts"]

Diagram sources

• BannedWordsPage.tsx
• moderation.ts
• words-moderation.route.ts
• words-moderation.controller.ts
• words-moderation.service.ts
• words-moderation.repo.ts
• banned-word.table.ts
• moderator.service.ts
• normalize.ts
• aho-corasick.ts

Section sources

• words-moderation.route.ts
• words-moderation.controller.ts
• words-moderation.service.ts
• words-moderation.repo.ts
• moderator.service.ts
• normalize.ts
• aho-corasick.ts

Performance Considerations

• Matcher caching: Moderation results are cached by normalized text to avoid repeated computation.
• Version-aware refresh: The moderation engine checks for database updates periodically to refresh matchers without manual intervention.
• Efficient matching: Aho-Corasick enables linear-time multi-pattern search relative to input length.
• Normalization overhead: Decomposition and mapping are O(n) per character; batching or streaming could help for very large texts.
• Index utilization: Database indexes on word and strict mode improve lookup performance.

Recommendations:

• Monitor cache hit rates and adjust TTL if needed.
• Batch frequent updates to minimize rebuild frequency.
• Consider partitioning by severity or category for large lists to optimize scans.

Section sources

• moderator.service.ts
• moderator.service.ts
• banned-word.table.ts

Troubleshooting Guide

Common issues and resolutions:

• Empty or invalid input: Validation enforces minimum length and UUID format; ensure payloads conform to schemas.
• Not found errors: Update/delete operations throw not-found errors when IDs do not exist; verify ID correctness.
• False positives: Adjust strict mode and severity; consider adding exceptions or refining normalization rules.
• Performance degradation: Confirm cache is active and matchers are being rebuilt only when necessary.

Operational references:

• Validation and error handling: words-moderation.schema.ts
• Not found handling: words-moderation.service.ts
• Cache behavior: moderator.service.ts

Section sources

• words-moderation.schema.ts
• words-moderation.service.ts
• moderator.service.ts

Conclusion

The banned words management system integrates an intuitive admin interface with a robust moderation engine. It supports dynamic updates, strict normalization, and wildcard pattern matching while maintaining performance through caching and efficient algorithms. Administrators can refine strategies over time by adjusting severity levels, strict mode, and normalization rules, ensuring both effectiveness and minimal false positives.

Appendices

Database Schema: Banned Words

• Columns: id, word (unique), strict_mode, severity, created_at, updated_at
• Indexes: word, strict_mode
• Enum: severity drawn from moderation severity enum

Section sources

• banned-word.table.ts
• index.ts
• 0005_snapshot.json