Commit local changes before merging with remote

2025-12-06 02:20:11 -08:00 · 2025-05-29 11:25:02 -04:00
parent 868b340fb5
commit be4f6a8d8c
75 changed files with 14107 additions and 562 deletions
--- a/crontab/README.md
+++ b/crontab/README.md
@@ -0,0 +1,376 @@
+# Crontab Management and Backup Scripts
+
+This directory contains all scripts and documentation related to crontab management, backup, monitoring, and multi-system administration.
+
+## Scripts Overview
+
+### Core Management Scripts
+
+#### `crontab-backup-system.sh`
+
+**Comprehensive crontab backup and recovery system**
+
+- **Multi-system support** with hostname-based organization
+- **Automated backup creation** with timestamped archives
+- **Syntax validation** and integrity checking
+- **Cross-system comparison** and restoration
+- **Legacy migration** from old backup structures
+- **Automated cleanup** with configurable retention
+
+**Usage:**
+
+```bash
+./crontab-backup-system.sh backup manual            # Create manual backup
+./crontab-backup-system.sh list                     # List backups for current system
+./crontab-backup-system.sh list io                  # List backups for specific system
+./crontab-backup-system.sh status all               # View all systems status
+./crontab-backup-system.sh restore backup-file      # Restore from backup
+./crontab-backup-system.sh compare current old.backup # Compare configurations
+```
+
+#### `manage-enhanced-crontab.sh`
+
+**Enhanced crontab installation and management**
+
+- **System-specific crontab installation** with validation
+- **Pre/post-install backups** for safety
+- **Script verification** and executable checking
+- **Logrotate setup** for log management
+- **Integration** with backup and monitoring systems
+
+**Usage:**
+
+```bash
+./manage-enhanced-crontab.sh install    # Install enhanced crontab system
+./manage-enhanced-crontab.sh verify     # Verify all scripts exist
+./manage-enhanced-crontab.sh status     # Check system health
+./manage-enhanced-crontab.sh backup     # Backup current crontab only
+```
+
+#### `fix-crontab-merging.sh`
+
+**Crontab merge conflict resolution**
+
+- **Conflict detection** and resolution
+- **Safe merging** of crontab entries
+- **Backup preservation** during merge operations
+- **Interactive conflict resolution**
+
+### System-Specific Configuration Files
+
+#### `crontab-europa.txt`
+
+Production crontab configuration for the Europa system:
+
+```bash
+# Move docker backups (01:00)
+0 1 * * * /home/acedanger/shell/move-backups.sh 2>&1 | logger -t backup-move -p user.info
+
+# Plex backup (04:15)
+15 4 * * * /home/acedanger/shell/plex/backup-plex.sh 2>&1 | logger -t plex-backup -p user.info
+
+# Validate plex backups (07:00)
+0 7 * * * /home/acedanger/shell/validate-plex-backups.sh --fix 2>&1 | logger -t plex-validation -p user.info
+```
+
+#### `crontab-io.txt`
+
+Crontab configuration for the IO system with media-focused operations.
+
+#### `crontab-racknerd.txt`
+
+Crontab configuration for the RackNerd system with backup synchronization.
+
+#### `crontab.txt.bak`
+
+Backup of legacy crontab configuration for reference.
+
+## Directory Structure
+
+```
+crontab-backups/
+├── europa/              # Current system (example)
+│   ├── current-crontab.backup
+│   └── archive/
+│       ├── europa-crontab-initial-20250526_101354.backup
+│       └── europa-crontab-pre-install-20250526_100622.backup
+├── io/                  # Remote system backups
+│   ├── current-crontab.backup
+│   └── archive/
+│       └── io-crontab-sample-20250526_101558.backup
+└── racknerd/            # Another remote system
+    ├── current-crontab.backup
+    └── archive/
+        └── racknerd-crontab-sample-20250526_101558.backup
+```
+
+## Enhanced Features
+
+### Multi-System Support
+
+- **Hostname-based organization**: Each system gets its own directory
+- **Cross-system operations**: View, compare, and restore backups from any system
+- **Centralized management**: Manage all systems from a single location
+- **Legacy migration**: Automatic migration from old backup structures
+
+### System Logging Integration
+
+All crontab operations integrate with system logging using specific tags:
+
+- `crontab-backup`: Crontab backup operations
+- `plex-backup`: Plex database backup operations
+- `backup-move`: Docker backup file transfers
+- `plex-validation`: Backup integrity checks
+- `backup-cleanup`: Cleanup operations
+
+### Automated Operations
+
+- **Daily automated backups** at midnight
+- **Syntax validation** before installing changes
+- **Automatic cleanup** of old backups (30-day retention)
+- **Health monitoring** and status reporting
+- **Error handling** with comprehensive logging
+
+## Configuration
+
+### Environment Variables
+
+Key configuration parameters in the backup system:
+
+```bash
+# Retention settings
+BACKUP_RETENTION_DAYS=30      # Keep backups for 30 days
+AUTO_CLEANUP=true             # Enable automatic cleanup
+
+# Directory settings
+BACKUP_ROOT="./crontab-backups"
+LOG_DIR="./logs"
+
+# System identification
+HOSTNAME=$(hostname)          # Automatic system identification
+```
+
+### File Naming Convention
+
+Backup files follow the pattern: `{hostname}-crontab-{type}-{timestamp}.backup`
+
+Examples:
+
+- `europa-crontab-manual-20250526_101354.backup`
+- `io-crontab-pre-upgrade-20250526_120000.backup`
+- `racknerd-crontab-auto-20250526_000001.backup`
+
+## Automation and Scheduling
+
+### Automated Backup Setup
+
+```bash
+# Setup automated daily backups
+./crontab-backup-system.sh setup-auto
+```
+
+This adds a daily backup entry to the system crontab:
+
+```bash
+0 0 * * * /path/to/crontab-backup-system.sh backup auto --auto-cleanup 2>&1 | logger -t crontab-backup -p user.info
+```
+
+### Cross-System Synchronization
+
+For distributed environments, set up backup synchronization:
+
+```bash
+# Sync backups from remote systems
+rsync -avz europa:/home/acedanger/shell/crontab-backups/europa/ ./crontab-backups/europa/
+rsync -avz io:/home/acedanger/shell/crontab-backups/io/ ./crontab-backups/io/
+```
+
+## Monitoring and Health Checks
+
+### System Status Monitoring
+
+```bash
+# View status for current system
+./crontab-backup-system.sh status
+
+# View status for specific system
+./crontab-backup-system.sh status io
+
+# View status for all systems
+./crontab-backup-system.sh status all
+```
+
+### Log Monitoring
+
+```bash
+# View all backup-related logs
+journalctl -t crontab-backup -t plex-backup -t backup-move -f
+
+# View logs for specific operation
+journalctl -t plex-backup --since "1 hour ago"
+
+# Check for errors in the last 24 hours
+journalctl --since '24 hours ago' --priority=err -t plex-backup
+```
+
+### Health Check Commands
+
+```bash
+# Check backup system health
+./crontab-backup-system.sh status all
+
+# Verify all scripts exist and are executable
+./manage-enhanced-crontab.sh verify
+
+# View recent backup activity
+grep "SUCCESS" logs/crontab-management.log | tail -20
+```
+
+## Troubleshooting
+
+### Common Issues
+
+#### Permission Errors
+
+Ensure proper file permissions on backup directories:
+
+```bash
+chmod 755 crontab-backups/
+chmod 644 crontab-backups/*/*.backup
+```
+
+#### Missing Backups
+
+Check automated backup cron entries:
+
+```bash
+sudo crontab -l | grep crontab-backup
+```
+
+#### Syntax Errors
+
+Validate crontab syntax before applying:
+
+```bash
+./crontab-backup-system.sh validate new-crontab.txt
+```
+
+### Debug Mode
+
+Enable verbose logging for troubleshooting:
+
+```bash
+./crontab-backup-system.sh status all 2>&1 | tee debug.log
+```
+
+### Recovery Procedures
+
+#### Emergency Crontab Restoration
+
+```bash
+# List available backups
+./crontab-backup-system.sh list
+
+# Compare current with backup
+./crontab-backup-system.sh compare current backup-file.backup
+
+# Restore from specific backup
+./crontab-backup-system.sh restore backup-file.backup
+```
+
+#### Disaster Recovery
+
+```bash
+# Restore from specific system backup during emergency
+./crontab-backup-system.sh restore io-crontab-pre-incident-20250526_101354.backup
+
+# Compare pre-incident and post-incident configurations
+./crontab-backup-system.sh compare io-crontab-pre-incident-20250526_101354.backup current
+```
+
+## Advanced Use Cases
+
+### Configuration Management
+
+```bash
+# Standardize crontab across multiple systems
+./crontab-backup-system.sh compare europa-crontab-standard-20250526_101354.backup io-crontab-current-20250526_120000.backup
+
+# Validate configurations before deployment
+./crontab-backup-system.sh validate new-crontab-config.txt
+```
+
+### Compliance and Auditing
+
+- **Change tracking**: Complete history of all crontab changes across systems
+- **Audit trails**: System logs provide comprehensive audit information
+- **Compliance reporting**: Generate reports showing backup frequency and success rates
+
+### Cross-System Comparison
+
+```bash
+# Compare current crontab with backup from another system
+./crontab-backup-system.sh compare current io-crontab-sample-20250526_101558.backup
+
+# Compare two backups from different systems
+./crontab-backup-system.sh compare europa-crontab-manual-20250526_101354.backup racknerd-crontab-sample-20250526_101558.backup
+```
+
+## Security Considerations
+
+### File Permissions
+
+- **Backup directories**: Restrict access to authorized users only
+- **Log files**: Ensure proper log rotation and access controls
+- **Remote access**: Use secure methods (SSH, rsync) for cross-system operations
+
+### Backup Integrity
+
+- **Validation**: Regular syntax validation of backup files
+- **Checksums**: Consider adding checksum verification for critical backups
+- **Retention**: Implement appropriate backup retention policies
+
+### Access Control
+
+- Scripts require appropriate sudo permissions for crontab operations
+- Backup locations should have restricted access
+- Log files contain operational data only (no sensitive information)
+
+## Documentation
+
+### Detailed Documentation
+
+- **[enhanced-crontab-system.md](./enhanced-crontab-system.md)**: Comprehensive crontab system documentation
+- **[multi-system-crontab-management.md](./multi-system-crontab-management.md)**: Multi-system management guide
+- **[crontab-merging-issue-resolution.md](./crontab-merging-issue-resolution.md)**: Conflict resolution procedures
+
+### Integration Notes
+
+- All scripts follow repository coding standards
+- Consistent logging and error handling
+- Color-coded output for readability
+- Comprehensive help systems
+- System logging integration
+
+## Best Practices
+
+1. **Regular Testing**: Periodically test backup restoration procedures
+2. **Documentation**: Keep records of system configurations and backup schedules
+3. **Automation**: Use automated cleanup to prevent disk space issues
+4. **Monitoring**: Implement comprehensive monitoring and alerting
+5. **Security**: Regularly review and update access controls
+
+## Migration Notes
+
+When migrating from legacy crontab systems:
+
+1. **Backup current configuration**: Create manual backup before changes
+2. **Test new system**: Use validation and dry-run features
+3. **Gradual migration**: Migrate one system at a time
+4. **Monitor performance**: Check logs for any issues post-migration
+
+The system automatically detects and migrates legacy backup structures while preserving all existing data.
+
+---
+
+*This crontab management system provides robust, scalable backup management for distributed environments while maintaining simplicity and reliability.*