Implement enhanced crontab management system with automated backups and logging

- Added enhanced crontab entries with comprehensive logging to syslog.
- Created a management script (`manage-enhanced-crontab.sh`) for installing and verifying crontab entries.
- Introduced backup scripts for crontab management, including automated timestamped backups and cleanup.
- Developed documentation for enhanced crontab and multi-system management.
- Established a directory structure for managing crontab backups across multiple systems.
- Implemented error handling and logging for backup operations.
- Added health monitoring and reporting features for backup processes.

docs/enhanced-crontab-system.md

@@ -0,0 +1,226 @@
+# Enhanced Crontab and Backup Monitoring System
+
+## Overview
+
+This enhanced system provides comprehensive crontab management with automatic backups, system logging, and advanced monitoring capabilities for your backup operations.
+
+## Components
+
+### 1. Enhanced Crontab Entries (`enhanced-crontab.txt`)
+
+- **Move backups** (01:00): Transfers Docker backups with logging
+- **Plex backup** (04:15): Daily Plex database backup with auto-repair
+- **Plex validation** (07:00): Validates and fixes backup integrity
+- **Immich backup** (05:00 Mon): Weekly Immich database backup move
+- **Weekly report** (08:00 Sun): Comprehensive backup status report
+
+### 2. Crontab Management (`manage-enhanced-crontab.sh`)
+
+- Install enhanced crontab entries with validation
+- Verify all backup scripts exist and are executable
+- Setup automated backups and log rotation
+- Integration with backup and monitoring systems
+
+### 3. Crontab Backup System (`crontab-backup-system.sh`)
+
+- Automated timestamped crontab backups
+- Backup comparison and restoration
+- Syntax validation
+- Automated cleanup of old backups
+- Daily automated backup scheduling
+
+### 4. Backup Log Monitor (`backup-log-monitor.sh`)
+
+- Real-time log monitoring with color coding
+- Error analysis and reporting
+- System health checks
+- Comprehensive backup reports
+- Service status overview
+
+## Quick Start
+
+### 1. Install the Enhanced System
+
+```bash
+# Make scripts executable
+chmod +x manage-enhanced-crontab.sh crontab-backup-system.sh backup-log-monitor.sh
+
+# Install enhanced crontab with all features
+sudo ./manage-enhanced-crontab.sh install
+```
+
+### 2. Monitor Your Backups
+
+```bash
+# Real-time monitoring
+./backup-log-monitor.sh monitor
+
+# Check system health
+./backup-log-monitor.sh health
+
+# View recent activity
+./backup-log-monitor.sh recent 24
+
+# Generate weekly report
+./backup-log-monitor.sh report 7
+```
+
+### 3. Manage Crontab Backups
+
+```bash
+# Create manual backup
+./crontab-backup-system.sh backup manual
+
+# List all backups
+./crontab-backup-system.sh list
+
+# Compare current with backup
+./crontab-backup-system.sh compare current
+
+# System status
+./crontab-backup-system.sh status
+```
+
+## Features
+
+### Enhanced Logging
+
+All backup operations now log to syslog with specific tags:
+
+- `plex-backup`: Plex database backup operations
+- `backup-move`: Docker backup file transfers
+- `plex-validation`: Backup integrity checks
+- `immich-backup`: Immich database operations
+- `plex-report`: Weekly reporting
+- `crontab-backup`: Crontab backup operations
+
+### Automatic Backups
+
+- **Crontab backups**: Daily automated backups at midnight
+- **Cleanup**: Automatic removal of backups older than 30 days
+- **Validation**: Syntax checking before applying changes
+- **Recovery**: Easy restoration from any backup point
+
+### Health Monitoring
+
+- Script existence and permissions
+- Backup directory availability
+- Recent activity tracking
+- Error rate monitoring
+- Overall system health scoring
+
+### Error Handling
+
+- Graceful failure handling in cron jobs
+- Detailed error logging and reporting
+- Exit code tracking for debugging
+- Comprehensive error summaries
+
+## Log Analysis
+
+### View Real-time Logs
+
+```bash
+# All backup services
+sudo journalctl -f -t plex-backup -t backup-move -t plex-validation -t immich-backup -t plex-report
+
+# Specific service
+sudo journalctl -f -t plex-backup
+
+# With our monitor (recommended)
+./backup-log-monitor.sh monitor
+```
+
+### Historical Analysis
+
+```bash
+# Last 24 hours
+sudo journalctl --since '24 hours ago' -t plex-backup
+
+# Last week with errors only
+sudo journalctl --since '1 week ago' --priority=err -t plex-backup
+
+# Using our tools
+./backup-log-monitor.sh recent 24 plex-backup
+./backup-log-monitor.sh errors 7
+```
+
+## Backup Recovery
+
+### Restore Crontab from Backup
+
+```bash
+# List available backups
+./crontab-backup-system.sh list
+
+# Restore specific backup
+./crontab-backup-system.sh restore crontab-manual-20250526_120000.backup
+
+# Compare before restoring
+./crontab-backup-system.sh compare current crontab-manual-20250526_120000.backup
+```
+
+### Emergency Recovery
+
+If you need to quickly restore the original crontab:
+
+```bash
+# The system automatically creates pre-install backups
+./crontab-backup-system.sh list | grep pre-install
+./crontab-backup-system.sh restore [backup-filename]
+```
+
+## Maintenance
+
+### Regular Tasks
+
+```bash
+# Weekly health check
+./backup-log-monitor.sh health
+
+# Monthly backup cleanup
+./crontab-backup-system.sh cleanup 30
+
+# Quarterly comprehensive report
+./backup-log-monitor.sh report 90
+```
+
+### Troubleshooting
+
+```bash
+# Verify all components
+./manage-enhanced-crontab.sh verify
+
+# Check system status
+./manage-enhanced-crontab.sh status
+
+# View configuration
+./manage-enhanced-crontab.sh show
+
+# Monitor for issues
+./backup-log-monitor.sh monitor
+```
+
+## Integration Notes
+
+- All scripts follow the established shell repository coding standards
+- Logging uses consistent tags and priorities
+- Error handling preserves backup integrity
+- Color-coded output for better readability
+- Comprehensive documentation and help systems
+
+## Security Considerations
+
+- Scripts validate input and handle errors gracefully
+- Backup files include metadata for tracking
+- Permissions are properly managed
+- Sensitive operations require sudo confirmation
+- All operations are logged for audit trails
+
+## Next Steps
+
+1. **Install the system**: Run `sudo ./manage-enhanced-crontab.sh install`
+2. **Test monitoring**: Use `./backup-log-monitor.sh monitor` during next backup
+3. **Review reports**: Generate weekly reports to establish baseline
+4. **Set up alerts**: Consider integrating with your notification system
+5. **Document customizations**: Add any local modifications to this guide

docs/multi-system-crontab-management.md

@@ -0,0 +1,280 @@
+# Multi-System Crontab Management Guide
+
+## Overview
+
+The enhanced crontab backup system now supports managing crontab backups across multiple systems using a hostname-based directory structure. This enables centralized backup management for distributed environments.
+
+## System Architecture
+
+### Directory Structure
+
+```
+crontab-backups/
+├── europa/              # Current system (example)
+│   ├── current-crontab.backup
+│   └── archive/
+│       ├── europa-crontab-initial-20250526_101354.backup
+│       └── europa-crontab-pre-install-20250526_100622.backup
+├── io/                  # Remote system backups
+│   ├── current-crontab.backup
+│   └── archive/
+│       └── io-crontab-sample-20250526_101558.backup
+└── racknerd/            # Another remote system
+    ├── current-crontab.backup
+    └── archive/
+        └── racknerd-crontab-sample-20250526_101558.backup
+```
+
+### File Naming Convention
+
+- Format: `{hostname}-crontab-{type}-{timestamp}.backup`
+- Examples:
+  - `europa-crontab-manual-20250526_101354.backup`
+  - `io-crontab-pre-upgrade-20250526_120000.backup`
+  - `racknerd-crontab-auto-20250526_000001.backup`
+
+## Features
+
+### 🔄 Multi-System Support
+
+- **Hostname-based organization**: Each system gets its own directory
+- **Cross-system operations**: View, compare, and restore backups from any system
+- **Centralized management**: Manage all systems from a single location
+
+### 📊 System Status Monitoring
+
+```bash
+# View status for current system
+./crontab-backup-system.sh status
+
+# View status for specific system
+./crontab-backup-system.sh status io
+
+# View status for all systems
+./crontab-backup-system.sh status all
+```
+
+### 🗂️ Backup Operations
+
+```bash
+# Create backup on current system
+./crontab-backup-system.sh backup pre-upgrade
+
+# List backups for specific system
+./crontab-backup-system.sh list io
+
+# List all systems with backups
+./crontab-backup-system.sh list-systems
+
+# Import backup from external source
+./crontab-backup-system.sh import /path/to/backup.txt io manual
+```
+
+### 🔍 Cross-System Comparison
+
+```bash
+# Compare current crontab with backup from another system
+./crontab-backup-system.sh compare current io-crontab-sample-20250526_101558.backup
+
+# Compare two backups from different systems
+./crontab-backup-system.sh compare europa-crontab-manual-20250526_101354.backup racknerd-crontab-sample-20250526_101558.backup
+```
+
+### 🧹 Cleanup Management
+
+```bash
+# Clean up backups older than 30 days for current system
+./crontab-backup-system.sh cleanup 30
+
+# Clean up backups for specific system
+./crontab-backup-system.sh cleanup 7 io
+
+# Clean up backups for all systems
+./crontab-backup-system.sh cleanup 30 all
+```
+
+## Enhanced Logging Integration
+
+All backup operations now integrate with system logging:
+
+### Syslog Integration
+
+- **Tag-based logging**: Each operation uses specific syslog tags
+- **Priority levels**: Different priorities for info, warnings, and errors
+- **Performance monitoring**: Execution time tracking for all operations
+
+### Example Enhanced Crontab Entries
+
+```bash
+# Plex backup with comprehensive logging
+15 4 * * * /home/acedanger/shell/backup-plex.sh 2>&1 | logger -t plex-backup -p user.info
+
+# Backup move operation with error handling
+0 1 * * * /home/acedanger/shell/move-backups.sh 2>&1 | logger -t backup-move -p user.info
+
+# Validation with performance tracking
+0 7 * * * /home/acedanger/shell/validate-plex-backups.sh --fix 2>&1 | logger -t plex-validation -p user.info
+```
+
+### Log Monitoring
+
+```bash
+# View all backup-related logs
+journalctl -t plex-backup -t backup-move -t plex-validation -f
+
+# View logs for specific operation
+journalctl -t plex-backup --since "1 hour ago"
+
+# Monitor backup performance
+./backup-log-monitor.sh --real-time
+```
+
+## Migration from Legacy Structure
+
+The system automatically detects and migrates legacy backup structures:
+
+### Automatic Migration
+
+- **Legacy detection**: Automatically detects old `crontab-backups/archive/` structure
+- **Hostname prefix**: Adds hostname prefix to existing backup files
+- **Backward compatibility**: Preserves all existing backup data
+- **Safe migration**: Original files remain untouched until manual cleanup
+
+### Manual Migration
+
+```bash
+# Force migration of legacy backups
+./crontab-backup-system.sh migrate
+```
+
+## Production Deployment
+
+### System Setup
+
+1. **Deploy script**: Copy `crontab-backup-system.sh` to each system
+2. **Configure permissions**: Ensure proper read/write access to backup directories
+3. **Setup automation**: Configure automated daily backups
+
+### Automated Backup Setup
+
+```bash
+# Setup automated daily backups on each system
+./crontab-backup-system.sh setup-auto
+```
+
+This adds the following entry to the system crontab:
+
+```bash
+0 0 * * * /path/to/crontab-backup-system.sh backup auto --auto-cleanup 2>&1 | logger -t crontab-backup -p user.info
+```
+
+### Cross-System Synchronization
+
+For distributed environments, consider setting up backup synchronization:
+
+```bash
+# Example rsync command to sync backups from remote systems
+rsync -avz europa:/home/acedanger/shell/crontab-backups/europa/ /home/acedanger/shell/crontab-backups/europa/
+rsync -avz io:/home/acedanger/shell/crontab-backups/io/ /home/acedanger/shell/crontab-backups/io/
+```
+
+## Security Considerations
+
+### File Permissions
+
+- **Backup directories**: Restrict access to authorized users only
+- **Log files**: Ensure proper log rotation and access controls
+- **Remote access**: Use secure methods (SSH, rsync) for cross-system operations
+
+### Backup Integrity
+
+- **Validation**: Regular syntax validation of backup files
+- **Checksums**: Consider adding checksum verification for critical backups
+- **Retention**: Implement appropriate backup retention policies
+
+## Advanced Use Cases
+
+### Disaster Recovery
+
+```bash
+# Restore from specific system backup during emergency
+./crontab-backup-system.sh restore io-crontab-pre-incident-20250526_101354.backup
+
+# Compare pre-incident and post-incident configurations
+./crontab-backup-system.sh compare io-crontab-pre-incident-20250526_101354.backup current
+```
+
+### Configuration Management
+
+```bash
+# Standardize crontab across multiple systems
+./crontab-backup-system.sh compare europa-crontab-standard-20250526_101354.backup io-crontab-current-20250526_120000.backup
+
+# Validate configurations before deployment
+./crontab-backup-system.sh validate new-crontab-config.txt
+```
+
+### Compliance and Auditing
+
+- **Change tracking**: Complete history of all crontab changes across systems
+- **Audit trails**: System logs provide comprehensive audit information
+- **Compliance reporting**: Generate reports showing backup frequency and success rates
+
+## Monitoring and Alerting
+
+### Health Checks
+
+```bash
+# Check backup system health
+./crontab-backup-system.sh status all
+
+# Monitor recent backup activity
+./backup-log-monitor.sh --health-check
+```
+
+### Alert Integration
+
+Consider integrating with monitoring systems:
+
+- **Backup failures**: Alert when backups fail or are missing
+- **Old backups**: Alert when systems haven't been backed up recently
+- **Disk space**: Monitor backup directory disk usage
+
+## Best Practices
+
+1. **Regular Testing**: Periodically test backup restoration procedures
+2. **Documentation**: Keep records of system configurations and backup schedules
+3. **Automation**: Use automated cleanup to prevent disk space issues
+4. **Monitoring**: Implement comprehensive monitoring and alerting
+5. **Security**: Regularly review and update access controls
+
+## Support and Troubleshooting
+
+### Common Issues
+
+- **Permission errors**: Ensure proper file permissions on backup directories
+- **Missing backups**: Check automated backup cron entries
+- **Syntax errors**: Use validation feature before deploying new crontabs
+
+### Debug Mode
+
+Enable verbose logging for troubleshooting:
+
+```bash
+# Add debug logging to any command
+./crontab-backup-system.sh status all 2>&1 | tee debug.log
+```
+
+### Log Analysis
+
+```bash
+# Analyze backup patterns
+grep "SUCCESS" logs/crontab-management.log | tail -20
+
+# Check for errors
+grep "ERROR" logs/crontab-management.log | tail -10
+```
+
+---
+
+*This multi-system crontab management solution provides robust, scalable backup management for distributed environments while maintaining simplicity and reliability.*