shell/crontab/README.md

Crontab Management and Backup Scripts

This directory contains all scripts and documentation related to crontab management, backup, monitoring, and multi-system administration.

Scripts Overview

Core Management Scripts

crontab-backup-system.sh

Comprehensive crontab backup and recovery system

• Multi-system support with hostname-based organization
• Automated backup creation with timestamped archives
• Syntax validation and integrity checking
• Cross-system comparison and restoration
• Legacy migration from old backup structures
• Automated cleanup with configurable retention

Usage:

./crontab-backup-system.sh backup manual            # Create manual backup
./crontab-backup-system.sh list                     # List backups for current system
./crontab-backup-system.sh list io                  # List backups for specific system
./crontab-backup-system.sh status all               # View all systems status
./crontab-backup-system.sh restore backup-file      # Restore from backup
./crontab-backup-system.sh compare current old.backup # Compare configurations

manage-enhanced-crontab.sh

Enhanced crontab installation and management

• System-specific crontab installation with validation
• Pre/post-install backups for safety
• Script verification and executable checking
• Logrotate setup for log management
• Integration with backup and monitoring systems

Usage:

./manage-enhanced-crontab.sh install    # Install enhanced crontab system
./manage-enhanced-crontab.sh verify     # Verify all scripts exist
./manage-enhanced-crontab.sh status     # Check system health
./manage-enhanced-crontab.sh backup     # Backup current crontab only

fix-crontab-merging.sh

Crontab merge conflict resolution

• Conflict detection and resolution
• Safe merging of crontab entries
• Backup preservation during merge operations
• Interactive conflict resolution

System-Specific Configuration Files

crontab-europa.txt

Production crontab configuration for the Europa system:

# Move docker backups (01:00)
0 1 * * * /home/acedanger/shell/move-backups.sh 2>&1 | logger -t backup-move -p user.info

# Plex backup (04:15)
15 4 * * * /home/acedanger/shell/plex/backup-plex.sh 2>&1 | logger -t plex-backup -p user.info

# Validate plex backups (07:00)
0 7 * * * /home/acedanger/shell/validate-plex-backups.sh --fix 2>&1 | logger -t plex-validation -p user.info

crontab-io.txt

Crontab configuration for the IO system with media-focused operations.

crontab-racknerd.txt

Crontab configuration for the RackNerd system with backup synchronization.

crontab.txt.bak

Backup of legacy crontab configuration for reference.

Directory Structure

crontab-backups/
├── europa/              # Current system (example)
│   ├── current-crontab.backup
│   └── archive/
│       ├── europa-crontab-initial-20250526_101354.backup
│       └── europa-crontab-pre-install-20250526_100622.backup
├── io/                  # Remote system backups
│   ├── current-crontab.backup
│   └── archive/
│       └── io-crontab-sample-20250526_101558.backup
└── racknerd/            # Another remote system
    ├── current-crontab.backup
    └── archive/
        └── racknerd-crontab-sample-20250526_101558.backup

Enhanced Features

Multi-System Support

• Hostname-based organization: Each system gets its own directory
• Cross-system operations: View, compare, and restore backups from any system
• Centralized management: Manage all systems from a single location
• Legacy migration: Automatic migration from old backup structures

System Logging Integration

All crontab operations integrate with system logging using specific tags:

• crontab-backup: Crontab backup operations
• plex-backup: Plex database backup operations
• backup-move: Docker backup file transfers
• plex-validation: Backup integrity checks
• backup-cleanup: Cleanup operations

Automated Operations

• Daily automated backups at midnight
• Syntax validation before installing changes
• Automatic cleanup of old backups (30-day retention)
• Health monitoring and status reporting
• Error handling with comprehensive logging

Configuration

Environment Variables

Key configuration parameters in the backup system:

# Retention settings
BACKUP_RETENTION_DAYS=30      # Keep backups for 30 days
AUTO_CLEANUP=true             # Enable automatic cleanup

# Directory settings
BACKUP_ROOT="./crontab-backups"
LOG_DIR="./logs"

# System identification
HOSTNAME=$(hostname)          # Automatic system identification

File Naming Convention

Backup files follow the pattern: {hostname}-crontab-{type}-{timestamp}.backup

Examples:

• europa-crontab-manual-20250526_101354.backup
• io-crontab-pre-upgrade-20250526_120000.backup
• racknerd-crontab-auto-20250526_000001.backup

Automation and Scheduling

Automated Backup Setup

# Setup automated daily backups
./crontab-backup-system.sh setup-auto

This adds a daily backup entry to the system crontab:

0 0 * * * /path/to/crontab-backup-system.sh backup auto --auto-cleanup 2>&1 | logger -t crontab-backup -p user.info

Cross-System Synchronization

For distributed environments, set up backup synchronization:

# Sync backups from remote systems
rsync -avz europa:/home/acedanger/shell/crontab-backups/europa/ ./crontab-backups/europa/
rsync -avz io:/home/acedanger/shell/crontab-backups/io/ ./crontab-backups/io/

Monitoring and Health Checks

System Status Monitoring

# View status for current system
./crontab-backup-system.sh status

# View status for specific system
./crontab-backup-system.sh status io

# View status for all systems
./crontab-backup-system.sh status all

Log Monitoring

# View all backup-related logs
journalctl -t crontab-backup -t plex-backup -t backup-move -f

# View logs for specific operation
journalctl -t plex-backup --since "1 hour ago"

# Check for errors in the last 24 hours
journalctl --since '24 hours ago' --priority=err -t plex-backup

Health Check Commands

# Check backup system health
./crontab-backup-system.sh status all

# Verify all scripts exist and are executable
./manage-enhanced-crontab.sh verify

# View recent backup activity
grep "SUCCESS" logs/crontab-management.log | tail -20

Troubleshooting

Common Issues

Permission Errors

Ensure proper file permissions on backup directories:

chmod 755 crontab-backups/
chmod 644 crontab-backups/*/*.backup

Missing Backups

Check automated backup cron entries:

sudo crontab -l | grep crontab-backup

Syntax Errors

Validate crontab syntax before applying:

./crontab-backup-system.sh validate new-crontab.txt

Debug Mode

Enable verbose logging for troubleshooting:

./crontab-backup-system.sh status all 2>&1 | tee debug.log

Recovery Procedures

Emergency Crontab Restoration

# List available backups
./crontab-backup-system.sh list

# Compare current with backup
./crontab-backup-system.sh compare current backup-file.backup

# Restore from specific backup
./crontab-backup-system.sh restore backup-file.backup

Disaster Recovery

# Restore from specific system backup during emergency
./crontab-backup-system.sh restore io-crontab-pre-incident-20250526_101354.backup

# Compare pre-incident and post-incident configurations
./crontab-backup-system.sh compare io-crontab-pre-incident-20250526_101354.backup current

Advanced Use Cases

Configuration Management

# Standardize crontab across multiple systems
./crontab-backup-system.sh compare europa-crontab-standard-20250526_101354.backup io-crontab-current-20250526_120000.backup

# Validate configurations before deployment
./crontab-backup-system.sh validate new-crontab-config.txt

Compliance and Auditing

• Change tracking: Complete history of all crontab changes across systems
• Audit trails: System logs provide comprehensive audit information
• Compliance reporting: Generate reports showing backup frequency and success rates

Cross-System Comparison

# Compare current crontab with backup from another system
./crontab-backup-system.sh compare current io-crontab-sample-20250526_101558.backup

# Compare two backups from different systems
./crontab-backup-system.sh compare europa-crontab-manual-20250526_101354.backup racknerd-crontab-sample-20250526_101558.backup

Security Considerations

File Permissions

• Backup directories: Restrict access to authorized users only
• Log files: Ensure proper log rotation and access controls
• Remote access: Use secure methods (SSH, rsync) for cross-system operations

Backup Integrity

• Validation: Regular syntax validation of backup files
• Checksums: Consider adding checksum verification for critical backups
• Retention: Implement appropriate backup retention policies

Access Control

• Scripts require appropriate sudo permissions for crontab operations
• Backup locations should have restricted access
• Log files contain operational data only (no sensitive information)

Documentation

Detailed Documentation

• enhanced-crontab-system.md: Comprehensive crontab system documentation
• multi-system-crontab-management.md: Multi-system management guide
• crontab-merging-issue-resolution.md: Conflict resolution procedures

Integration Notes

• All scripts follow repository coding standards
• Consistent logging and error handling
• Color-coded output for readability
• Comprehensive help systems
• System logging integration

Best Practices

1. Regular Testing: Periodically test backup restoration procedures
2. Documentation: Keep records of system configurations and backup schedules
3. Automation: Use automated cleanup to prevent disk space issues
4. Monitoring: Implement comprehensive monitoring and alerting
5. Security: Regularly review and update access controls

Migration Notes

When migrating from legacy crontab systems:

1. Backup current configuration: Create manual backup before changes
2. Test new system: Use validation and dry-run features
3. Gradual migration: Migrate one system at a time
4. Monitor performance: Check logs for any issues post-migration

The system automatically detects and migrates legacy backup structures while preserving all existing data.

---

This crontab management system provides robust, scalable backup management for distributed environments while maintaining simplicity and reliability.