Implement enhanced crontab management system with automated backups and logging

- Added enhanced crontab entries with comprehensive logging to syslog.
- Created a management script (`manage-enhanced-crontab.sh`) for installing and verifying crontab entries.
- Introduced backup scripts for crontab management, including automated timestamped backups and cleanup.
- Developed documentation for enhanced crontab and multi-system management.
- Established a directory structure for managing crontab backups across multiple systems.
- Implemented error handling and logging for backup operations.
- Added health monitoring and reporting features for backup processes.

docs/multi-system-crontab-management.md

@@ -0,0 +1,280 @@
+# Multi-System Crontab Management Guide
+
+## Overview
+
+The enhanced crontab backup system now supports managing crontab backups across multiple systems using a hostname-based directory structure. This enables centralized backup management for distributed environments.
+
+## System Architecture
+
+### Directory Structure
+
+```
+crontab-backups/
+├── europa/              # Current system (example)
+│   ├── current-crontab.backup
+│   └── archive/
+│       ├── europa-crontab-initial-20250526_101354.backup
+│       └── europa-crontab-pre-install-20250526_100622.backup
+├── io/                  # Remote system backups
+│   ├── current-crontab.backup
+│   └── archive/
+│       └── io-crontab-sample-20250526_101558.backup
+└── racknerd/            # Another remote system
+    ├── current-crontab.backup
+    └── archive/
+        └── racknerd-crontab-sample-20250526_101558.backup
+```
+
+### File Naming Convention
+
+- Format: `{hostname}-crontab-{type}-{timestamp}.backup`
+- Examples:
+  - `europa-crontab-manual-20250526_101354.backup`
+  - `io-crontab-pre-upgrade-20250526_120000.backup`
+  - `racknerd-crontab-auto-20250526_000001.backup`
+
+## Features
+
+### 🔄 Multi-System Support
+
+- **Hostname-based organization**: Each system gets its own directory
+- **Cross-system operations**: View, compare, and restore backups from any system
+- **Centralized management**: Manage all systems from a single location
+
+### 📊 System Status Monitoring
+
+```bash
+# View status for current system
+./crontab-backup-system.sh status
+
+# View status for specific system
+./crontab-backup-system.sh status io
+
+# View status for all systems
+./crontab-backup-system.sh status all
+```
+
+### 🗂️ Backup Operations
+
+```bash
+# Create backup on current system
+./crontab-backup-system.sh backup pre-upgrade
+
+# List backups for specific system
+./crontab-backup-system.sh list io
+
+# List all systems with backups
+./crontab-backup-system.sh list-systems
+
+# Import backup from external source
+./crontab-backup-system.sh import /path/to/backup.txt io manual
+```
+
+### 🔍 Cross-System Comparison
+
+```bash
+# Compare current crontab with backup from another system
+./crontab-backup-system.sh compare current io-crontab-sample-20250526_101558.backup
+
+# Compare two backups from different systems
+./crontab-backup-system.sh compare europa-crontab-manual-20250526_101354.backup racknerd-crontab-sample-20250526_101558.backup
+```
+
+### 🧹 Cleanup Management
+
+```bash
+# Clean up backups older than 30 days for current system
+./crontab-backup-system.sh cleanup 30
+
+# Clean up backups for specific system
+./crontab-backup-system.sh cleanup 7 io
+
+# Clean up backups for all systems
+./crontab-backup-system.sh cleanup 30 all
+```
+
+## Enhanced Logging Integration
+
+All backup operations now integrate with system logging:
+
+### Syslog Integration
+
+- **Tag-based logging**: Each operation uses specific syslog tags
+- **Priority levels**: Different priorities for info, warnings, and errors
+- **Performance monitoring**: Execution time tracking for all operations
+
+### Example Enhanced Crontab Entries
+
+```bash
+# Plex backup with comprehensive logging
+15 4 * * * /home/acedanger/shell/backup-plex.sh 2>&1 | logger -t plex-backup -p user.info
+
+# Backup move operation with error handling
+0 1 * * * /home/acedanger/shell/move-backups.sh 2>&1 | logger -t backup-move -p user.info
+
+# Validation with performance tracking
+0 7 * * * /home/acedanger/shell/validate-plex-backups.sh --fix 2>&1 | logger -t plex-validation -p user.info
+```
+
+### Log Monitoring
+
+```bash
+# View all backup-related logs
+journalctl -t plex-backup -t backup-move -t plex-validation -f
+
+# View logs for specific operation
+journalctl -t plex-backup --since "1 hour ago"
+
+# Monitor backup performance
+./backup-log-monitor.sh --real-time
+```
+
+## Migration from Legacy Structure
+
+The system automatically detects and migrates legacy backup structures:
+
+### Automatic Migration
+
+- **Legacy detection**: Automatically detects old `crontab-backups/archive/` structure
+- **Hostname prefix**: Adds hostname prefix to existing backup files
+- **Backward compatibility**: Preserves all existing backup data
+- **Safe migration**: Original files remain untouched until manual cleanup
+
+### Manual Migration
+
+```bash
+# Force migration of legacy backups
+./crontab-backup-system.sh migrate
+```
+
+## Production Deployment
+
+### System Setup
+
+1. **Deploy script**: Copy `crontab-backup-system.sh` to each system
+2. **Configure permissions**: Ensure proper read/write access to backup directories
+3. **Setup automation**: Configure automated daily backups
+
+### Automated Backup Setup
+
+```bash
+# Setup automated daily backups on each system
+./crontab-backup-system.sh setup-auto
+```
+
+This adds the following entry to the system crontab:
+
+```bash
+0 0 * * * /path/to/crontab-backup-system.sh backup auto --auto-cleanup 2>&1 | logger -t crontab-backup -p user.info
+```
+
+### Cross-System Synchronization
+
+For distributed environments, consider setting up backup synchronization:
+
+```bash
+# Example rsync command to sync backups from remote systems
+rsync -avz europa:/home/acedanger/shell/crontab-backups/europa/ /home/acedanger/shell/crontab-backups/europa/
+rsync -avz io:/home/acedanger/shell/crontab-backups/io/ /home/acedanger/shell/crontab-backups/io/
+```
+
+## Security Considerations
+
+### File Permissions
+
+- **Backup directories**: Restrict access to authorized users only
+- **Log files**: Ensure proper log rotation and access controls
+- **Remote access**: Use secure methods (SSH, rsync) for cross-system operations
+
+### Backup Integrity
+
+- **Validation**: Regular syntax validation of backup files
+- **Checksums**: Consider adding checksum verification for critical backups
+- **Retention**: Implement appropriate backup retention policies
+
+## Advanced Use Cases
+
+### Disaster Recovery
+
+```bash
+# Restore from specific system backup during emergency
+./crontab-backup-system.sh restore io-crontab-pre-incident-20250526_101354.backup
+
+# Compare pre-incident and post-incident configurations
+./crontab-backup-system.sh compare io-crontab-pre-incident-20250526_101354.backup current
+```
+
+### Configuration Management
+
+```bash
+# Standardize crontab across multiple systems
+./crontab-backup-system.sh compare europa-crontab-standard-20250526_101354.backup io-crontab-current-20250526_120000.backup
+
+# Validate configurations before deployment
+./crontab-backup-system.sh validate new-crontab-config.txt
+```
+
+### Compliance and Auditing
+
+- **Change tracking**: Complete history of all crontab changes across systems
+- **Audit trails**: System logs provide comprehensive audit information
+- **Compliance reporting**: Generate reports showing backup frequency and success rates
+
+## Monitoring and Alerting
+
+### Health Checks
+
+```bash
+# Check backup system health
+./crontab-backup-system.sh status all
+
+# Monitor recent backup activity
+./backup-log-monitor.sh --health-check
+```
+
+### Alert Integration
+
+Consider integrating with monitoring systems:
+
+- **Backup failures**: Alert when backups fail or are missing
+- **Old backups**: Alert when systems haven't been backed up recently
+- **Disk space**: Monitor backup directory disk usage
+
+## Best Practices
+
+1. **Regular Testing**: Periodically test backup restoration procedures
+2. **Documentation**: Keep records of system configurations and backup schedules
+3. **Automation**: Use automated cleanup to prevent disk space issues
+4. **Monitoring**: Implement comprehensive monitoring and alerting
+5. **Security**: Regularly review and update access controls
+
+## Support and Troubleshooting
+
+### Common Issues
+
+- **Permission errors**: Ensure proper file permissions on backup directories
+- **Missing backups**: Check automated backup cron entries
+- **Syntax errors**: Use validation feature before deploying new crontabs
+
+### Debug Mode
+
+Enable verbose logging for troubleshooting:
+
+```bash
+# Add debug logging to any command
+./crontab-backup-system.sh status all 2>&1 | tee debug.log
+```
+
+### Log Analysis
+
+```bash
+# Analyze backup patterns
+grep "SUCCESS" logs/crontab-management.log | tail -20
+
+# Check for errors
+grep "ERROR" logs/crontab-management.log | tail -10
+```
+
+---
+
+*This multi-system crontab management solution provides robust, scalable backup management for distributed environments while maintaining simplicity and reliability.*