mirror of
https://github.com/acedanger/shell.git
synced 2025-12-06 01:10:12 -08:00
clean up log files with ANSI color codes
This commit is contained in:
Peter Wood
221
docs/cleanup-log-ansi.md
Normal file
221
docs/cleanup-log-ansi.md
Normal file
@@ -0,0 +1,221 @@
|
||||
# Log ANSI Cleanup Utility
|
||||
|
||||
This utility script removes ANSI color codes from log files to improve readability and reduce file sizes when viewing logs in editors or processing them with other tools.
|
||||
|
||||
## Features
|
||||
|
||||
- **Auto-discovery**: Automatically finds log files in common locations
|
||||
- **Backup creation**: Creates backup files before cleaning (recommended)
|
||||
- **Pattern filtering**: Process only files matching specific patterns
|
||||
- **Recursive processing**: Can process directories recursively
|
||||
- **Dry run mode**: Preview changes without making them
|
||||
- **Comprehensive detection**: Detects various ANSI escape sequence formats
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# Make the script executable
|
||||
chmod +x cleanup-log-ansi.sh
|
||||
|
||||
# Optional: Create a symbolic link for system-wide access
|
||||
sudo ln -s "$(pwd)/cleanup-log-ansi.sh" /usr/local/bin/cleanup-log-ansi
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Clean a single log file with backup
|
||||
./cleanup-log-ansi.sh --backup /path/to/logfile.log
|
||||
|
||||
# Clean all log files in a directory
|
||||
./cleanup-log-ansi.sh --recursive --backup /path/to/logs/
|
||||
|
||||
# Auto-discover and clean all common log files
|
||||
./cleanup-log-ansi.sh --auto-discover --backup
|
||||
```
|
||||
|
||||
### Advanced Usage
|
||||
|
||||
```bash
|
||||
# Dry run to see what would be cleaned
|
||||
./cleanup-log-ansi.sh --dry-run --auto-discover
|
||||
|
||||
# Clean only .log files in a directory tree
|
||||
./cleanup-log-ansi.sh --recursive --filter "*.log" --backup /var/log/
|
||||
|
||||
# Clean without creating backups (use with caution)
|
||||
./cleanup-log-ansi.sh --no-backup /path/to/logfile.log
|
||||
|
||||
# Verbose output
|
||||
./cleanup-log-ansi.sh --verbose --auto-discover --backup
|
||||
```
|
||||
|
||||
## Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `-h, --help` | Show help message |
|
||||
| `-r, --recursive` | Process directories recursively |
|
||||
| `-b, --backup` | Create backup files before cleaning (recommended) |
|
||||
| `-n, --no-backup` | Don't create backup files |
|
||||
| `-v, --verbose` | Show detailed output |
|
||||
| `-d, --dry-run` | Show what would be done without making changes |
|
||||
| `-f, --filter PATTERN` | Only process files matching pattern (e.g., '*.log') |
|
||||
| `-a, --auto-discover` | Find common log files automatically |
|
||||
|
||||
## Auto-Discovery Locations
|
||||
|
||||
The script automatically searches for log files in these locations:
|
||||
|
||||
- `./logs/`
|
||||
- `./crontab/logs/`
|
||||
- `./plex/logs/`
|
||||
- `../logs/`
|
||||
- `~/.local/share/logs/`
|
||||
- `/var/log/` (if readable)
|
||||
|
||||
## File Extensions
|
||||
|
||||
Auto-discovery looks for files with these extensions:
|
||||
- `.log`
|
||||
- `.out`
|
||||
- `.err`
|
||||
|
||||
## ANSI Code Detection
|
||||
|
||||
The script detects and removes these ANSI escape sequence patterns:
|
||||
- `\033[0;31m` (literal backslash sequences)
|
||||
- `\x1b[0;31m` (hexadecimal escape sequences)
|
||||
- `\033[0;31m` (octal escape sequences)
|
||||
|
||||
## Examples
|
||||
|
||||
### Clean Crontab Logs
|
||||
|
||||
```bash
|
||||
# Clean all crontab management logs
|
||||
./cleanup-log-ansi.sh --backup crontab/logs/
|
||||
|
||||
# Example output:
|
||||
# INFO: Starting ANSI cleanup utility
|
||||
# INFO: Backup created: crontab/logs/crontab-management.log.ansi-backup
|
||||
# SUCCESS: Cleaned: crontab/logs/crontab-management.log (24K → 21K)
|
||||
```
|
||||
|
||||
### Clean Plex Logs
|
||||
|
||||
```bash
|
||||
# Clean specific Plex database recovery logs
|
||||
./cleanup-log-ansi.sh --backup plex/logs/database-recovery-*.log
|
||||
|
||||
# Clean all Plex logs recursively
|
||||
./cleanup-log-ansi.sh --recursive --backup plex/logs/
|
||||
```
|
||||
|
||||
### System-wide Cleanup
|
||||
|
||||
```bash
|
||||
# Find and clean all log files (with dry run first)
|
||||
./cleanup-log-ansi.sh --dry-run --auto-discover
|
||||
|
||||
# If satisfied with the preview, run for real
|
||||
./cleanup-log-ansi.sh --auto-discover --backup
|
||||
```
|
||||
|
||||
## Backup Management
|
||||
|
||||
When using `--backup`, backup files are created with the `.ansi-backup` suffix:
|
||||
|
||||
```bash
|
||||
# Original file: app.log
|
||||
# Backup file: app.log.ansi-backup
|
||||
|
||||
# Remove all backup files when satisfied
|
||||
rm -f *.ansi-backup **/*.ansi-backup
|
||||
```
|
||||
|
||||
## Integration with Crontab
|
||||
|
||||
You can add this script to your crontab for regular cleanup:
|
||||
|
||||
```bash
|
||||
# Clean logs weekly (Sundays at 3 AM)
|
||||
0 3 * * 0 /path/to/cleanup-log-ansi.sh --auto-discover --backup --quiet 2>&1 | logger -t log-cleanup
|
||||
|
||||
# Clean specific directory daily
|
||||
0 2 * * * /path/to/cleanup-log-ansi.sh --recursive --no-backup /var/log/myapp/ 2>&1 | logger -t log-cleanup
|
||||
```
|
||||
|
||||
## Safety Features
|
||||
|
||||
1. **Backup Creation**: Always create backups by default (use `--backup`)
|
||||
2. **File Validation**: Checks file existence and readability before processing
|
||||
3. **Integrity Verification**: Ensures cleaned files aren't corrupted
|
||||
4. **Dry Run Mode**: Preview changes before applying them
|
||||
5. **Error Handling**: Graceful error handling with meaningful messages
|
||||
|
||||
## Common Use Cases
|
||||
|
||||
### After Log Rotation
|
||||
```bash
|
||||
# Clean old log files after rotation
|
||||
./cleanup-log-ansi.sh --filter "*.log.*" --backup /var/log/
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
```bash
|
||||
# Clean build logs in CI pipeline
|
||||
./cleanup-log-ansi.sh --no-backup --recursive build/logs/
|
||||
```
|
||||
|
||||
### Development Environment
|
||||
```bash
|
||||
# Clean development logs regularly
|
||||
./cleanup-log-ansi.sh --auto-discover --backup --verbose
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Permission Denied
|
||||
```bash
|
||||
# If you get permission errors for system logs
|
||||
sudo ./cleanup-log-ansi.sh --backup /var/log/
|
||||
```
|
||||
|
||||
### No Files Found
|
||||
```bash
|
||||
# Check what files would be processed
|
||||
./cleanup-log-ansi.sh --dry-run --verbose --auto-discover
|
||||
|
||||
# Specify paths manually if auto-discovery fails
|
||||
./cleanup-log-ansi.sh --backup /specific/path/to/logs/
|
||||
```
|
||||
|
||||
### Large Files
|
||||
For very large log files, the script processes them efficiently in a single pass, but you may want to:
|
||||
|
||||
1. Use `--dry-run` first to estimate processing time
|
||||
2. Process files individually for better control
|
||||
3. Ensure sufficient disk space for backups
|
||||
|
||||
## Performance
|
||||
|
||||
- **Memory Efficient**: Uses streaming processing via `sed`
|
||||
- **Fast**: Single-pass processing of files
|
||||
- **Safe**: Creates backups before modification
|
||||
- **Scalable**: Can handle hundreds of log files
|
||||
|
||||
## Related Scripts
|
||||
|
||||
- `crontab-backup-system.sh`: Now has improved logging (ANSI codes properly stripped)
|
||||
- `backup-*.sh`: Various backup scripts that generate logs
|
||||
- `validate-*.sh`: Validation scripts with colored output
|
||||
|
||||
## Version History
|
||||
|
||||
- **v1.0**: Initial release with basic ANSI cleanup functionality
|
||||
- **v1.1**: Added auto-discovery and recursive processing
|
||||
- **v1.2**: Improved pattern matching and backup handling
|
||||
- **v1.3**: Added comprehensive safety features and verbose output
|
||||
Reference in New Issue
Block a user