shell/docs/cleanup-log-ansi.md

Log ANSI Cleanup Utility

This utility script removes ANSI color codes from log files to improve readability and reduce file sizes when viewing logs in editors or processing them with other tools.

Features

• Auto-discovery: Automatically finds log files in common locations
• Backup creation: Creates backup files before cleaning (recommended)
• Pattern filtering: Process only files matching specific patterns
• Recursive processing: Can process directories recursively
• Dry run mode: Preview changes without making them
• Comprehensive detection: Detects various ANSI escape sequence formats

Installation

# Make the script executable
chmod +x cleanup-log-ansi.sh

# Optional: Create a symbolic link for system-wide access
sudo ln -s "$(pwd)/cleanup-log-ansi.sh" /usr/local/bin/cleanup-log-ansi

Usage

Basic Usage

# Clean a single log file with backup
./cleanup-log-ansi.sh --backup /path/to/logfile.log

# Clean all log files in a directory
./cleanup-log-ansi.sh --recursive --backup /path/to/logs/

# Auto-discover and clean all common log files
./cleanup-log-ansi.sh --auto-discover --backup

Advanced Usage

# Dry run to see what would be cleaned
./cleanup-log-ansi.sh --dry-run --auto-discover

# Clean only .log files in a directory tree
./cleanup-log-ansi.sh --recursive --filter "*.log" --backup /var/log/

# Clean without creating backups (use with caution)
./cleanup-log-ansi.sh --no-backup /path/to/logfile.log

# Verbose output
./cleanup-log-ansi.sh --verbose --auto-discover --backup

Options

Option	Description
-h, --help	Show help message
-r, --recursive	Process directories recursively
-b, --backup	Create backup files before cleaning (recommended)
-n, --no-backup	Don't create backup files
-v, --verbose	Show detailed output
-d, --dry-run	Show what would be done without making changes
-f, --filter PATTERN	Only process files matching pattern (e.g., '*.log')
-a, --auto-discover	Find common log files automatically

Auto-Discovery Locations

The script automatically searches for log files in these locations:

• ./logs/
• ./crontab/logs/
• ./plex/logs/
• ../logs/
• ~/.local/share/logs/
• /var/log/ (if readable)

File Extensions

Auto-discovery looks for files with these extensions:

• .log
• .out
• .err

ANSI Code Detection

The script detects and removes these ANSI escape sequence patterns:

• \033[0;31m (literal backslash sequences)
• \x1b[0;31m (hexadecimal escape sequences)
• \033[0;31m (octal escape sequences)

Examples

Clean Crontab Logs

# Clean all crontab management logs
./cleanup-log-ansi.sh --backup crontab/logs/

# Example output:
# INFO: Starting ANSI cleanup utility
# INFO: Backup created: crontab/logs/crontab-management.log.ansi-backup
# SUCCESS: Cleaned: crontab/logs/crontab-management.log (24K → 21K)

Clean Plex Logs

# Clean specific Plex database recovery logs
./cleanup-log-ansi.sh --backup plex/logs/database-recovery-*.log

# Clean all Plex logs recursively
./cleanup-log-ansi.sh --recursive --backup plex/logs/

System-wide Cleanup

# Find and clean all log files (with dry run first)
./cleanup-log-ansi.sh --dry-run --auto-discover

# If satisfied with the preview, run for real
./cleanup-log-ansi.sh --auto-discover --backup

Backup Management

When using --backup, backup files are created with the .ansi-backup suffix:

# Original file: app.log
# Backup file:   app.log.ansi-backup

# Remove all backup files when satisfied
rm -f *.ansi-backup **/*.ansi-backup

Integration with Crontab

You can add this script to your crontab for regular cleanup:

# Clean logs weekly (Sundays at 3 AM)
0 3 * * 0 /path/to/cleanup-log-ansi.sh --auto-discover --backup --quiet 2>&1 | logger -t log-cleanup

# Clean specific directory daily
0 2 * * * /path/to/cleanup-log-ansi.sh --recursive --no-backup /var/log/myapp/ 2>&1 | logger -t log-cleanup

Safety Features

1. Backup Creation: Always create backups by default (use --backup)
2. File Validation: Checks file existence and readability before processing
3. Integrity Verification: Ensures cleaned files aren't corrupted
4. Dry Run Mode: Preview changes before applying them
5. Error Handling: Graceful error handling with meaningful messages

Common Use Cases

After Log Rotation

# Clean old log files after rotation
./cleanup-log-ansi.sh --filter "*.log.*" --backup /var/log/

CI/CD Integration

# Clean build logs in CI pipeline
./cleanup-log-ansi.sh --no-backup --recursive build/logs/

Development Environment

# Clean development logs regularly
./cleanup-log-ansi.sh --auto-discover --backup --verbose

Troubleshooting

Permission Denied

# If you get permission errors for system logs
sudo ./cleanup-log-ansi.sh --backup /var/log/

No Files Found

# Check what files would be processed
./cleanup-log-ansi.sh --dry-run --verbose --auto-discover

# Specify paths manually if auto-discovery fails
./cleanup-log-ansi.sh --backup /specific/path/to/logs/

Large Files

For very large log files, the script processes them efficiently in a single pass, but you may want to:

1. Use --dry-run first to estimate processing time
2. Process files individually for better control
3. Ensure sufficient disk space for backups

Performance

• Memory Efficient: Uses streaming processing via sed
• Fast: Single-pass processing of files
• Safe: Creates backups before modification
• Scalable: Can handle hundreds of log files

Related Scripts

• crontab-backup-system.sh: Now has improved logging (ANSI codes properly stripped)
• backup-*.sh: Various backup scripts that generate logs
• validate-*.sh: Validation scripts with colored output

Version History

• v1.0: Initial release with basic ANSI cleanup functionality
• v1.1: Added auto-discovery and recursive processing
• v1.2: Improved pattern matching and backup handling
• v1.3: Added comprehensive safety features and verbose output