./crontab-backup-system.sh backup manual            # Create manual backup
./crontab-backup-system.sh list                     # List backups for current system
./crontab-backup-system.sh list io                  # List backups for specific system
./crontab-backup-system.sh status all               # View all systems status
./crontab-backup-system.sh restore backup-file      # Restore from backup
./crontab-backup-system.sh compare current old.backup # Compare configurations

`manage-enhanced-crontab.sh`

Enhanced crontab installation and management

System-specific crontab installation with validation
Pre/post-install backups for safety
Script verification and executable checking
Logrotate setup for log management
Integration with backup and monitoring systems

Usage:

./manage-enhanced-crontab.sh install    # Install enhanced crontab system
./manage-enhanced-crontab.sh verify     # Verify all scripts exist
./manage-enhanced-crontab.sh status     # Check system health
./manage-enhanced-crontab.sh backup     # Backup current crontab only

`fix-crontab-merging.sh`

Crontab merge conflict resolution

Conflict detection and resolution
Safe merging of crontab entries
Backup preservation during merge operations
Interactive conflict resolution

System-Specific Configuration Files

`crontab-europa.txt`

Production crontab configuration for the Europa system:

# Move docker backups (01:00)
0 1 * * * /home/acedanger/shell/move-backups.sh 2>&1 | logger -t backup-move -p user.info

# Backup environment files (02:00)
0 2 * * * { echo "Starting .env files backup"; /home/acedanger/shell/backup-env-files.sh; echo ".env backup completed with exit code: $?"; } 2>&1 | logger -t env-backup -p user.info

# Crontab backup system with auto-cleanup (04:00)
0 4 * * * /home/acedanger/shell/crontab/crontab-backup-system.sh backup auto --auto-cleanup 2>&1 | logger -t crontab-backup -p user.info

# Move Immich database backups (05:00, Mondays only)
0 5 * * 1 { echo "Starting Immich database backup move"; if mv /mnt/share/media/immich/uploads/backups/immich-db-backup* /mnt/share/media/backups/immich 2>/dev/null; then echo "Immich backup move completed successfully"; else echo "No Immich backup files found or move failed"; fi; } 2>&1 | logger -t immich-backup -p user.info

# Validate environment backups (08:30, Sundays only)
30 8 * * 0 { echo "Starting .env backup validation"; /home/acedanger/shell/validate-env-backups.sh; echo ".env validation completed with exit code: $?"; } 2>&1 | logger -t env-validation -p user.info

`crontab-io.txt`

Crontab configuration for the IO system with media-focused operations.

`crontab-racknerd.txt`

Crontab configuration for the RackNerd system with backup synchronization.

`crontab.txt.bak`

Backup of legacy crontab configuration for reference.

Directory Structure

crontab-backups/
├── europa/              # Current system (example)
│   ├── current-crontab.backup
│   └── archive/
│       ├── europa-crontab-initial-20250526_101354.backup
│       └── europa-crontab-pre-install-20250526_100622.backup
├── io/                  # Remote system backups
│   ├── current-crontab.backup
│   └── archive/
│       └── io-crontab-sample-20250526_101558.backup
└── racknerd/            # Another remote system
    ├── current-crontab.backup
    └── archive/
        └── racknerd-crontab-sample-20250526_101558.backup

Enhanced Features

Multi-System Support

Hostname-based organization: Each system gets its own directory
Cross-system operations: View, compare, and restore backups from any system
Centralized management: Manage all systems from a single location
Legacy migration: Automatic migration from old backup structures

System Logging Integration

All crontab operations integrate with system logging using specific tags:

crontab-backup: Crontab backup operations
plex-backup: Plex database backup operations
backup-move: Docker backup file transfers
plex-validation: Backup integrity checks
backup-cleanup: Cleanup operations

Automated Operations

Daily automated backups at midnight
Syntax validation before installing changes
Automatic cleanup of old backups (30-day retention)
Health monitoring and status reporting
Error handling with comprehensive logging

Configuration

Environment Variables

Key configuration parameters in the backup system:

# Retention settings
BACKUP_RETENTION_DAYS=30      # Keep backups for 30 days
AUTO_CLEANUP=true             # Enable automatic cleanup

# Directory settings
BACKUP_ROOT="./crontab-backups"
LOG_DIR="./logs"

# System identification
HOSTNAME=$(hostname)          # Automatic system identification

File Naming Convention

Backup files follow the pattern: {hostname}-crontab-{type}-{timestamp}.backup

Examples:

europa-crontab-manual-20250526_101354.backup
io-crontab-pre-upgrade-20250526_120000.backup
racknerd-crontab-auto-20250526_000001.backup

Automation and Scheduling

Automated Backup Setup

# Setup automated daily backups
./crontab-backup-system.sh setup-auto

This adds a daily backup entry to the system crontab:

0 0 * * * /path/to/crontab-backup-system.sh backup auto --auto-cleanup 2>&1 | logger -t crontab-backup -p user.info

Cross-System Synchronization

For distributed environments, set up backup synchronization:

# Sync backups from remote systems
rsync -avz europa:/home/acedanger/shell/crontab-backups/europa/ ./crontab-backups/europa/
rsync -avz io:/home/acedanger/shell/crontab-backups/io/ ./crontab-backups/io/

Monitoring and Health Checks

System Status Monitoring

# View status for current system
./crontab-backup-system.sh status

# View status for specific system
./crontab-backup-system.sh status io

# View status for all systems
./crontab-backup-system.sh status all

Log Monitoring

# View all backup-related logs
journalctl -t crontab-backup -t plex-backup -t backup-move -f

# View logs for specific operation
journalctl -t plex-backup --since "1 hour ago"

# Check for errors in the last 24 hours
journalctl --since '24 hours ago' --priority=err -t plex-backup

Health Check Commands

# Check backup system health
./crontab-backup-system.sh status all

# Verify all scripts exist and are executable
./manage-enhanced-crontab.sh verify

# View recent backup activity
grep "SUCCESS" logs/crontab-management.log | tail -20

Troubleshooting

Common Issues

Permission Errors

Ensure proper file permissions on backup directories:

chmod 755 crontab-backups/
chmod 644 crontab-backups/*/*.backup

Missing Backups

Check automated backup cron entries:

sudo crontab -l | grep crontab-backup

Syntax Errors

Validate crontab syntax before applying:

./crontab-backup-system.sh validate new-crontab.txt

Debug Mode

Enable verbose logging for troubleshooting:

./crontab-backup-system.sh status all 2>&1 | tee debug.log

Recovery Procedures

Emergency Crontab Restoration

# List available backups
./crontab-backup-system.sh list

# Compare current with backup
./crontab-backup-system.sh compare current backup-file.backup

# Restore from specific backup
./crontab-backup-system.sh restore backup-file.backup

Disaster Recovery

# Restore from specific system backup during emergency
./crontab-backup-system.sh restore io-crontab-pre-incident-20250526_101354.backup

# Compare pre-incident and post-incident configurations
./crontab-backup-system.sh compare io-crontab-pre-incident-20250526_101354.backup current

Advanced Use Cases

Configuration Management

# Standardize crontab across multiple systems
./crontab-backup-system.sh compare europa-crontab-standard-20250526_101354.backup io-crontab-current-20250526_120000.backup

# Validate configurations before deployment
./crontab-backup-system.sh validate new-crontab-config.txt

Compliance and Auditing

Change tracking: Complete history of all crontab changes across systems
Audit trails: System logs provide comprehensive audit information
Compliance reporting: Generate reports showing backup frequency and success rates

Cross-System Comparison

# Compare current crontab with backup from another system
./crontab-backup-system.sh compare current io-crontab-sample-20250526_101558.backup

# Compare two backups from different systems
./crontab-backup-system.sh compare europa-crontab-manual-20250526_101354.backup racknerd-crontab-sample-20250526_101558.backup

Security Considerations

File Permissions

Backup directories: Restrict access to authorized users only
Log files: Ensure proper log rotation and access controls
Remote access: Use secure methods (SSH, rsync) for cross-system operations

Backup Integrity

Validation: Regular syntax validation of backup files
Checksums: Consider adding checksum verification for critical backups
Retention: Implement appropriate backup retention policies

Access Control

Scripts require appropriate sudo permissions for crontab operations
Backup locations should have restricted access
Log files contain operational data only (no sensitive information)

Documentation

Detailed Documentation

enhanced-crontab-system.md: Comprehensive crontab system documentation
multi-system-crontab-management.md: Multi-system management guide
crontab-merging-issue-resolution.md: Conflict resolution procedures

Integration Notes

All scripts follow repository coding standards
Consistent logging and error handling
Color-coded output for readability
Comprehensive help systems
System logging integration

Best Practices

Regular Testing: Periodically test backup restoration procedures
Documentation: Keep records of system configurations and backup schedules
Automation: Use automated cleanup to prevent disk space issues
Monitoring: Implement comprehensive monitoring and alerting
Security: Regularly review and update access controls

Migration Notes

When migrating from legacy crontab systems:

Backup current configuration: Create manual backup before changes
Test new system: Use validation and dry-run features
Gradual migration: Migrate one system at a time
Monitor performance: Check logs for any issues post-migration

The system automatically detects and migrates legacy backup structures while preserving all existing data.

This crontab management system provides robust, scalable backup management for distributed environments while maintaining simplicity and reliability.