shell/crontab/README.md

# Crontab Management and Backup Scripts

This directory contains all scripts and documentation related to crontab management, backup, monitoring, and multi-system administration.

## Scripts Overview

### Core Management Scripts

#### `crontab-backup-system.sh`

**Comprehensive crontab backup and recovery system**

- **Multi-system support** with hostname-based organization
- **Automated backup creation** with timestamped archives
- **Syntax validation** and integrity checking
- **Cross-system comparison** and restoration
- **Legacy migration** from old backup structures
- **Automated cleanup** with configurable retention

**Usage:**

```bash
./crontab-backup-system.sh backup manual            # Create manual backup
./crontab-backup-system.sh list                     # List backups for current system
./crontab-backup-system.sh list io                  # List backups for specific system
./crontab-backup-system.sh status all               # View all systems status
./crontab-backup-system.sh restore backup-file      # Restore from backup
./crontab-backup-system.sh compare current old.backup # Compare configurations
```

#### `manage-enhanced-crontab.sh`

**Enhanced crontab installation and management**

- **System-specific crontab installation** with validation
- **Pre/post-install backups** for safety
- **Script verification** and executable checking
- **Logrotate setup** for log management
- **Integration** with backup and monitoring systems

**Usage:**

```bash
./manage-enhanced-crontab.sh install    # Install enhanced crontab system
./manage-enhanced-crontab.sh verify     # Verify all scripts exist
./manage-enhanced-crontab.sh status     # Check system health
./manage-enhanced-crontab.sh backup     # Backup current crontab only
```

#### `fix-crontab-merging.sh`

**Crontab merge conflict resolution**

- **Conflict detection** and resolution
- **Safe merging** of crontab entries
- **Backup preservation** during merge operations
- **Interactive conflict resolution**

### System-Specific Configuration Files

#### `crontab-europa.txt`

Production crontab configuration for the Europa system:

```bash
# Move docker backups (01:00)
0 1 * * * /home/acedanger/shell/move-backups.sh 2>&1 | logger -t backup-move -p user.info

# Backup environment files (02:00)
0 2 * * * { echo "Starting .env files backup"; /home/acedanger/shell/backup-env-files.sh; echo ".env backup completed with exit code: $?"; } 2>&1 | logger -t env-backup -p user.info

# Crontab backup system with auto-cleanup (04:00)
0 4 * * * /home/acedanger/shell/crontab/crontab-backup-system.sh backup auto --auto-cleanup 2>&1 | logger -t crontab-backup -p user.info

# Move Immich database backups (05:00, Mondays only)
0 5 * * 1 { echo "Starting Immich database backup move"; if mv /mnt/share/media/immich/uploads/backups/immich-db-backup* /mnt/share/media/backups/immich 2>/dev/null; then echo "Immich backup move completed successfully"; else echo "No Immich backup files found or move failed"; fi; } 2>&1 | logger -t immich-backup -p user.info

# Validate environment backups (08:30, Sundays only)
30 8 * * 0 { echo "Starting .env backup validation"; /home/acedanger/shell/validate-env-backups.sh; echo ".env validation completed with exit code: $?"; } 2>&1 | logger -t env-validation -p user.info
```

#### `crontab-io.txt`

Crontab configuration for the IO system with media-focused operations.

#### `crontab-racknerd.txt`

Crontab configuration for the RackNerd system with backup synchronization.

#### `crontab.txt.bak`

Backup of legacy crontab configuration for reference.

## Directory Structure

```
crontab-backups/
├── europa/              # Current system (example)
│   ├── current-crontab.backup
│   └── archive/
│       ├── europa-crontab-initial-20250526_101354.backup
│       └── europa-crontab-pre-install-20250526_100622.backup
├── io/                  # Remote system backups
│   ├── current-crontab.backup
│   └── archive/
│       └── io-crontab-sample-20250526_101558.backup
└── racknerd/            # Another remote system
    ├── current-crontab.backup
    └── archive/
        └── racknerd-crontab-sample-20250526_101558.backup
```

## Enhanced Features

### Multi-System Support

- **Hostname-based organization**: Each system gets its own directory
- **Cross-system operations**: View, compare, and restore backups from any system
- **Centralized management**: Manage all systems from a single location
- **Legacy migration**: Automatic migration from old backup structures

### System Logging Integration

All crontab operations integrate with system logging using specific tags:

- `crontab-backup`: Crontab backup operations
- `plex-backup`: Plex database backup operations
- `backup-move`: Docker backup file transfers
- `plex-validation`: Backup integrity checks
- `backup-cleanup`: Cleanup operations

### Automated Operations

- **Daily automated backups** at midnight
- **Syntax validation** before installing changes
- **Automatic cleanup** of old backups (30-day retention)
- **Health monitoring** and status reporting
- **Error handling** with comprehensive logging

## Configuration

### Environment Variables

Key configuration parameters in the backup system:

```bash
# Retention settings
BACKUP_RETENTION_DAYS=30      # Keep backups for 30 days
AUTO_CLEANUP=true             # Enable automatic cleanup

# Directory settings
BACKUP_ROOT="./crontab-backups"
LOG_DIR="./logs"

# System identification
HOSTNAME=$(hostname)          # Automatic system identification
```

### File Naming Convention

Backup files follow the pattern: `{hostname}-crontab-{type}-{timestamp}.backup`

Examples:

- `europa-crontab-manual-20250526_101354.backup`
- `io-crontab-pre-upgrade-20250526_120000.backup`
- `racknerd-crontab-auto-20250526_000001.backup`

## Automation and Scheduling

### Automated Backup Setup

```bash
# Setup automated daily backups
./crontab-backup-system.sh setup-auto
```

This adds a daily backup entry to the system crontab:

```bash
0 0 * * * /path/to/crontab-backup-system.sh backup auto --auto-cleanup 2>&1 | logger -t crontab-backup -p user.info
```

### Cross-System Synchronization

For distributed environments, set up backup synchronization:

```bash
# Sync backups from remote systems
rsync -avz europa:/home/acedanger/shell/crontab-backups/europa/ ./crontab-backups/europa/
rsync -avz io:/home/acedanger/shell/crontab-backups/io/ ./crontab-backups/io/
```

## Monitoring and Health Checks

### System Status Monitoring

```bash
# View status for current system
./crontab-backup-system.sh status

# View status for specific system
./crontab-backup-system.sh status io

# View status for all systems
./crontab-backup-system.sh status all
```

### Log Monitoring

```bash
# View all backup-related logs
journalctl -t crontab-backup -t plex-backup -t backup-move -f

# View logs for specific operation
journalctl -t plex-backup --since "1 hour ago"

# Check for errors in the last 24 hours
journalctl --since '24 hours ago' --priority=err -t plex-backup
```

### Health Check Commands

```bash
# Check backup system health
./crontab-backup-system.sh status all

# Verify all scripts exist and are executable
./manage-enhanced-crontab.sh verify

# View recent backup activity
grep "SUCCESS" logs/crontab-management.log | tail -20
```

## Troubleshooting

### Common Issues

#### Permission Errors

Ensure proper file permissions on backup directories:

```bash
chmod 755 crontab-backups/
chmod 644 crontab-backups/*/*.backup
```

#### Missing Backups

Check automated backup cron entries:

```bash
sudo crontab -l | grep crontab-backup
```

#### Syntax Errors

Validate crontab syntax before applying:

```bash
./crontab-backup-system.sh validate new-crontab.txt
```

### Debug Mode

Enable verbose logging for troubleshooting:

```bash
./crontab-backup-system.sh status all 2>&1 | tee debug.log
```

### Recovery Procedures

#### Emergency Crontab Restoration

```bash
# List available backups
./crontab-backup-system.sh list

# Compare current with backup
./crontab-backup-system.sh compare current backup-file.backup

# Restore from specific backup
./crontab-backup-system.sh restore backup-file.backup
```

#### Disaster Recovery

```bash
# Restore from specific system backup during emergency
./crontab-backup-system.sh restore io-crontab-pre-incident-20250526_101354.backup

# Compare pre-incident and post-incident configurations
./crontab-backup-system.sh compare io-crontab-pre-incident-20250526_101354.backup current
```

## Advanced Use Cases

### Configuration Management

```bash
# Standardize crontab across multiple systems
./crontab-backup-system.sh compare europa-crontab-standard-20250526_101354.backup io-crontab-current-20250526_120000.backup

# Validate configurations before deployment
./crontab-backup-system.sh validate new-crontab-config.txt
```

### Compliance and Auditing

- **Change tracking**: Complete history of all crontab changes across systems
- **Audit trails**: System logs provide comprehensive audit information
- **Compliance reporting**: Generate reports showing backup frequency and success rates

### Cross-System Comparison

```bash
# Compare current crontab with backup from another system
./crontab-backup-system.sh compare current io-crontab-sample-20250526_101558.backup

# Compare two backups from different systems
./crontab-backup-system.sh compare europa-crontab-manual-20250526_101354.backup racknerd-crontab-sample-20250526_101558.backup
```

## Security Considerations

### File Permissions

- **Backup directories**: Restrict access to authorized users only
- **Log files**: Ensure proper log rotation and access controls
- **Remote access**: Use secure methods (SSH, rsync) for cross-system operations

### Backup Integrity

- **Validation**: Regular syntax validation of backup files
- **Checksums**: Consider adding checksum verification for critical backups
- **Retention**: Implement appropriate backup retention policies

### Access Control

- Scripts require appropriate sudo permissions for crontab operations
- Backup locations should have restricted access
- Log files contain operational data only (no sensitive information)

## Documentation

### Detailed Documentation

- **[enhanced-crontab-system.md](./enhanced-crontab-system.md)**: Comprehensive crontab system documentation
- **[multi-system-crontab-management.md](./multi-system-crontab-management.md)**: Multi-system management guide
- **[crontab-merging-issue-resolution.md](./crontab-merging-issue-resolution.md)**: Conflict resolution procedures

### Integration Notes

- All scripts follow repository coding standards
- Consistent logging and error handling
- Color-coded output for readability
- Comprehensive help systems
- System logging integration

## Best Practices

1. **Regular Testing**: Periodically test backup restoration procedures
2. **Documentation**: Keep records of system configurations and backup schedules
3. **Automation**: Use automated cleanup to prevent disk space issues
4. **Monitoring**: Implement comprehensive monitoring and alerting
5. **Security**: Regularly review and update access controls

## Migration Notes

When migrating from legacy crontab systems:

1. **Backup current configuration**: Create manual backup before changes
2. **Test new system**: Use validation and dry-run features
3. **Gradual migration**: Migrate one system at a time
4. **Monitor performance**: Check logs for any issues post-migration

The system automatically detects and migrates legacy backup structures while preserving all existing data.

---

*This crontab management system provides robust, scalable backup management for distributed environments while maintaining simplicity and reliability.*