peterwood/shell
1
0
Fork 0
mirror of https://github.com/acedanger/shell.git synced 2025-12-06 00:00:13 -08:00
Code Issues 19 Packages Projects Releases Wiki Activity

Commit local changes before merging with remote

Browse Source
This commit is contained in:
Peter Wood
2025-05-29 11:25:02 -04:00
parent 868b340fb5
commit be4f6a8d8c
75 changed files with 14107 additions and 562 deletions
Download Patch File Download Diff File Expand all files Collapse all files

376
crontab/README.md Normal file
View File

@@ -0,0 +1,376 @@
# Crontab Management and Backup Scripts
This directory contains all scripts and documentation related to crontab management, backup, monitoring, and multi-system administration.
## Scripts Overview
### Core Management Scripts
#### `crontab-backup-system.sh`
**Comprehensive crontab backup and recovery system**
- **Multi-system support** with hostname-based organization
- **Automated backup creation** with timestamped archives
- **Syntax validation** and integrity checking
- **Cross-system comparison** and restoration
- **Legacy migration** from old backup structures
- **Automated cleanup** with configurable retention
**Usage:**
```bash
./crontab-backup-system.sh backup manual # Create manual backup
./crontab-backup-system.sh list # List backups for current system
./crontab-backup-system.sh list io # List backups for specific system
./crontab-backup-system.sh status all # View all systems status
./crontab-backup-system.sh restore backup-file # Restore from backup
./crontab-backup-system.sh compare current old.backup # Compare configurations
```
#### `manage-enhanced-crontab.sh`
**Enhanced crontab installation and management**
- **System-specific crontab installation** with validation
- **Pre/post-install backups** for safety
- **Script verification** and executable checking
- **Logrotate setup** for log management
- **Integration** with backup and monitoring systems
**Usage:**
```bash
./manage-enhanced-crontab.sh install # Install enhanced crontab system
./manage-enhanced-crontab.sh verify # Verify all scripts exist
./manage-enhanced-crontab.sh status # Check system health
./manage-enhanced-crontab.sh backup # Backup current crontab only
```
#### `fix-crontab-merging.sh`
**Crontab merge conflict resolution**
- **Conflict detection** and resolution
- **Safe merging** of crontab entries
- **Backup preservation** during merge operations
- **Interactive conflict resolution**
### System-Specific Configuration Files
#### `crontab-europa.txt`
Production crontab configuration for the Europa system:
```bash
# Move docker backups (01:00)
0 1 * * * /home/acedanger/shell/move-backups.sh 2>&1 | logger -t backup-move -p user.info
# Plex backup (04:15)
15 4 * * * /home/acedanger/shell/plex/backup-plex.sh 2>&1 | logger -t plex-backup -p user.info
# Validate plex backups (07:00)
0 7 * * * /home/acedanger/shell/validate-plex-backups.sh --fix 2>&1 | logger -t plex-validation -p user.info
```
#### `crontab-io.txt`
Crontab configuration for the IO system with media-focused operations.
#### `crontab-racknerd.txt`
Crontab configuration for the RackNerd system with backup synchronization.
#### `crontab.txt.bak`
Backup of legacy crontab configuration for reference.
## Directory Structure
```
crontab-backups/
├── europa/ # Current system (example)
│ ├── current-crontab.backup
│ └── archive/
│ ├── europa-crontab-initial-20250526_101354.backup
│ └── europa-crontab-pre-install-20250526_100622.backup
├── io/ # Remote system backups
│ ├── current-crontab.backup
│ └── archive/
│ └── io-crontab-sample-20250526_101558.backup
└── racknerd/ # Another remote system
├── current-crontab.backup
└── archive/
└── racknerd-crontab-sample-20250526_101558.backup
```
## Enhanced Features
### Multi-System Support
- **Hostname-based organization**: Each system gets its own directory
- **Cross-system operations**: View, compare, and restore backups from any system
- **Centralized management**: Manage all systems from a single location
- **Legacy migration**: Automatic migration from old backup structures
### System Logging Integration
All crontab operations integrate with system logging using specific tags:
- `crontab-backup`: Crontab backup operations
- `plex-backup`: Plex database backup operations
- `backup-move`: Docker backup file transfers
- `plex-validation`: Backup integrity checks
- `backup-cleanup`: Cleanup operations
### Automated Operations
- **Daily automated backups** at midnight
- **Syntax validation** before installing changes
- **Automatic cleanup** of old backups (30-day retention)
- **Health monitoring** and status reporting
- **Error handling** with comprehensive logging
## Configuration
### Environment Variables
Key configuration parameters in the backup system:
```bash
# Retention settings
BACKUP_RETENTION_DAYS=30 # Keep backups for 30 days
AUTO_CLEANUP=true # Enable automatic cleanup
# Directory settings
BACKUP_ROOT="./crontab-backups"
LOG_DIR="./logs"
# System identification
HOSTNAME=$(hostname) # Automatic system identification
```
### File Naming Convention
Backup files follow the pattern: `{hostname}-crontab-{type}-{timestamp}.backup`
Examples:
- `europa-crontab-manual-20250526_101354.backup`
- `io-crontab-pre-upgrade-20250526_120000.backup`
- `racknerd-crontab-auto-20250526_000001.backup`
## Automation and Scheduling
### Automated Backup Setup
```bash
# Setup automated daily backups
./crontab-backup-system.sh setup-auto
```
This adds a daily backup entry to the system crontab:
```bash
0 0 * * * /path/to/crontab-backup-system.sh backup auto --auto-cleanup 2>&1 | logger -t crontab-backup -p user.info
```
### Cross-System Synchronization
For distributed environments, set up backup synchronization:
```bash
# Sync backups from remote systems
rsync -avz europa:/home/acedanger/shell/crontab-backups/europa/ ./crontab-backups/europa/
rsync -avz io:/home/acedanger/shell/crontab-backups/io/ ./crontab-backups/io/
```
## Monitoring and Health Checks
### System Status Monitoring
```bash
# View status for current system
./crontab-backup-system.sh status
# View status for specific system
./crontab-backup-system.sh status io
# View status for all systems
./crontab-backup-system.sh status all
```
### Log Monitoring
```bash
# View all backup-related logs
journalctl -t crontab-backup -t plex-backup -t backup-move -f
# View logs for specific operation
journalctl -t plex-backup --since "1 hour ago"
# Check for errors in the last 24 hours
journalctl --since '24 hours ago' --priority=err -t plex-backup
```
### Health Check Commands
```bash
# Check backup system health
./crontab-backup-system.sh status all
# Verify all scripts exist and are executable
./manage-enhanced-crontab.sh verify
# View recent backup activity
grep "SUCCESS" logs/crontab-management.log | tail -20
```
## Troubleshooting
### Common Issues
#### Permission Errors
Ensure proper file permissions on backup directories:
```bash
chmod 755 crontab-backups/
chmod 644 crontab-backups/*/*.backup
```
#### Missing Backups
Check automated backup cron entries:
```bash
sudo crontab -l | grep crontab-backup
```
#### Syntax Errors
Validate crontab syntax before applying:
```bash
./crontab-backup-system.sh validate new-crontab.txt
```
### Debug Mode
Enable verbose logging for troubleshooting:
```bash
./crontab-backup-system.sh status all 2>&1 | tee debug.log
```
### Recovery Procedures
#### Emergency Crontab Restoration
```bash
# List available backups
./crontab-backup-system.sh list
# Compare current with backup
./crontab-backup-system.sh compare current backup-file.backup
# Restore from specific backup
./crontab-backup-system.sh restore backup-file.backup
```
#### Disaster Recovery
```bash
# Restore from specific system backup during emergency
./crontab-backup-system.sh restore io-crontab-pre-incident-20250526_101354.backup
# Compare pre-incident and post-incident configurations
./crontab-backup-system.sh compare io-crontab-pre-incident-20250526_101354.backup current
```
## Advanced Use Cases
### Configuration Management
```bash
# Standardize crontab across multiple systems
./crontab-backup-system.sh compare europa-crontab-standard-20250526_101354.backup io-crontab-current-20250526_120000.backup
# Validate configurations before deployment
./crontab-backup-system.sh validate new-crontab-config.txt
```
### Compliance and Auditing
- **Change tracking**: Complete history of all crontab changes across systems
- **Audit trails**: System logs provide comprehensive audit information
- **Compliance reporting**: Generate reports showing backup frequency and success rates
### Cross-System Comparison
```bash
# Compare current crontab with backup from another system
./crontab-backup-system.sh compare current io-crontab-sample-20250526_101558.backup
# Compare two backups from different systems
./crontab-backup-system.sh compare europa-crontab-manual-20250526_101354.backup racknerd-crontab-sample-20250526_101558.backup
```
## Security Considerations
### File Permissions
- **Backup directories**: Restrict access to authorized users only
- **Log files**: Ensure proper log rotation and access controls
- **Remote access**: Use secure methods (SSH, rsync) for cross-system operations
### Backup Integrity
- **Validation**: Regular syntax validation of backup files
- **Checksums**: Consider adding checksum verification for critical backups
- **Retention**: Implement appropriate backup retention policies
### Access Control
- Scripts require appropriate sudo permissions for crontab operations
- Backup locations should have restricted access
- Log files contain operational data only (no sensitive information)
## Documentation
### Detailed Documentation
- **[enhanced-crontab-system.md](./enhanced-crontab-system.md)**: Comprehensive crontab system documentation
- **[multi-system-crontab-management.md](./multi-system-crontab-management.md)**: Multi-system management guide
- **[crontab-merging-issue-resolution.md](./crontab-merging-issue-resolution.md)**: Conflict resolution procedures
### Integration Notes
- All scripts follow repository coding standards
- Consistent logging and error handling
- Color-coded output for readability
- Comprehensive help systems
- System logging integration
## Best Practices
1. **Regular Testing**: Periodically test backup restoration procedures
2. **Documentation**: Keep records of system configurations and backup schedules
3. **Automation**: Use automated cleanup to prevent disk space issues
4. **Monitoring**: Implement comprehensive monitoring and alerting
5. **Security**: Regularly review and update access controls
## Migration Notes
When migrating from legacy crontab systems:
1. **Backup current configuration**: Create manual backup before changes
2. **Test new system**: Use validation and dry-run features
3. **Gradual migration**: Migrate one system at a time
4. **Monitor performance**: Check logs for any issues post-migration
The system automatically detects and migrates legacy backup structures while preserving all existing data.
---
*This crontab management system provides robust, scalable backup management for distributed environments while maintaining simplicity and reliability.*

732
crontab/crontab-backup-system.sh Executable file
View File

@@ -0,0 +1,732 @@
#!/bin/bash
# Crontab Backup and Recovery System
# This script provides comprehensive backup management for crontab entries
set -e
# Color codes for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
CYAN='\033[0;36m'
PURPLE='\033[0;35m'
NC='\033[0m' # No Color
SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
HOSTNAME=$(hostname)
BACKUP_ROOT="$SCRIPT_DIR/crontab-backups"
BACKUP_DIR="$BACKUP_ROOT/$HOSTNAME"
LOG_DIR="$SCRIPT_DIR/logs"
CURRENT_BACKUP="$BACKUP_DIR/current-crontab.backup"
ARCHIVE_DIR="$BACKUP_DIR/archive"
# Ensure directories exist
mkdir -p "$BACKUP_DIR" "$ARCHIVE_DIR" "$LOG_DIR"
log_message() {
local message="$1"
local log_file="$LOG_DIR/crontab-management.log"
echo -e "$(date '+%Y-%m-%d %H:%M:%S') $message"
echo "$(date '+%Y-%m-%d %H:%M:%S') $message" | sed 's/\x1b\[[0-9;]*m//g' >> "$log_file"
}
log_error() {
log_message "${RED}ERROR: $1${NC}"
}
log_success() {
log_message "${GREEN}SUCCESS: $1${NC}"
}
log_warning() {
log_message "${YELLOW}WARNING: $1${NC}"
}
log_info() {
log_message "${BLUE}INFO: $1${NC}"
}
create_timestamped_backup() {
local backup_type="${1:-manual}"
local timestamp=$(date +%Y%m%d_%H%M%S)
local backup_file="$ARCHIVE_DIR/${HOSTNAME}-crontab-${backup_type}-${timestamp}.backup"
log_info "Creating timestamped backup for $HOSTNAME: $backup_file"
if sudo crontab -l > "$backup_file" 2>/dev/null; then
log_success "Backup created: $backup_file"
# Also update the current backup
cp "$backup_file" "$CURRENT_BACKUP"
# Add metadata
echo "# Backup created: $(date)" >> "$backup_file"
echo "# Backup type: $backup_type" >> "$backup_file"
echo "# System: $HOSTNAME" >> "$backup_file"
echo "# User: root" >> "$backup_file"
echo "# Full system info: $(uname -a)" >> "$backup_file"
return 0
else
log_error "Failed to create backup or no crontab exists"
return 1
fi
}
list_backups() {
local target_hostname="${1:-$HOSTNAME}"
local target_dir="$BACKUP_ROOT/$target_hostname/archive"
log_info "Available crontab backups for $target_hostname:"
echo
if [ -d "$target_dir" ] && [ "$(ls -A "$target_dir" 2>/dev/null)" ]; then
printf "%-40s %-20s %-15s\n" "Filename" "Date Created" "Size"
printf "%-40s %-20s %-15s\n" "--------" "------------" "----"
for backup in "$target_dir"/*.backup; do
if [ -f "$backup" ]; then
local filename=$(basename "$backup")
local date_created=$(stat -c %y "$backup" | cut -d' ' -f1,2 | cut -d'.' -f1)
local size=$(stat -c %s "$backup")
printf "%-40s %-20s %-15s bytes\n" "$filename" "$date_created" "$size"
fi
done
else
log_warning "No backups found in $target_dir"
fi
echo
# Show all available systems if current system has no backups or if showing all
if [ "$target_hostname" = "$HOSTNAME" ] && [ ! -d "$target_dir" ]; then
log_info "Available systems with backups:"
for system_dir in "$BACKUP_ROOT"/*; do
if [ -d "$system_dir/archive" ] && [ "$(ls -A "$system_dir/archive" 2>/dev/null)" ]; then
local system_name=$(basename "$system_dir")
local backup_count=$(ls -1 "$system_dir/archive"/*.backup 2>/dev/null | wc -l)
echo " - $system_name ($backup_count backups)"
fi
done
echo
echo "Use: $0 list [hostname] to view backups for a specific system"
fi
}
restore_from_backup() {
local backup_file="$1"
local source_hostname=""
if [ -z "$backup_file" ]; then
log_error "No backup file specified"
list_backups
return 1
fi
# Handle different backup file formats and paths
if [[ "$backup_file" == *"/"* ]]; then
# Full or relative path provided
if [[ ! "$backup_file" = /* ]]; then
backup_file="$ARCHIVE_DIR/$backup_file"
fi
else
# Just filename provided - check current system first, then others
if [ -f "$ARCHIVE_DIR/$backup_file" ]; then
backup_file="$ARCHIVE_DIR/$backup_file"
else
# Search in other system directories
local found_file=""
for system_dir in "$BACKUP_ROOT"/*; do
if [ -f "$system_dir/archive/$backup_file" ]; then
found_file="$system_dir/archive/$backup_file"
source_hostname=$(basename "$system_dir")
break
fi
done
if [ -n "$found_file" ]; then
backup_file="$found_file"
log_warning "Backup file found in $source_hostname system directory"
fi
fi
fi
if [ ! -f "$backup_file" ]; then
log_error "Backup file not found: $backup_file"
echo
log_info "Available backups:"
list_backups
return 1
fi
# Extract source hostname from backup metadata if available
if [ -z "$source_hostname" ]; then
source_hostname=$(grep "^# System:" "$backup_file" 2>/dev/null | cut -d' ' -f3 || echo "unknown")
fi
log_info "Restoring crontab from: $backup_file"
if [ "$source_hostname" != "unknown" ] && [ "$source_hostname" != "$HOSTNAME" ]; then
log_warning "Restoring backup from different system: $source_hostname -> $HOSTNAME"
echo -n "Continue? [y/N]: "
read -r response
if [[ ! "$response" =~ ^[Yy]$ ]]; then
log_info "Restore cancelled"
return 1
fi
fi
# Create a safety backup before restoring
create_timestamped_backup "pre-restore"
# Remove metadata lines before restoring
grep -v "^# Backup" "$backup_file" > /tmp/crontab_restore.tmp
if sudo crontab /tmp/crontab_restore.tmp; then
log_success "Crontab restored successfully from $backup_file"
rm -f /tmp/crontab_restore.tmp
else
log_error "Failed to restore crontab"
rm -f /tmp/crontab_restore.tmp
return 1
fi
}
compare_crontabs() {
local file1="${1:-current}"
local file2="$2"
if [ "$file1" = "current" ]; then
sudo crontab -l > /tmp/current_crontab.tmp 2>/dev/null || echo "# No current crontab" > /tmp/current_crontab.tmp
file1="/tmp/current_crontab.tmp"
fi
if [ -z "$file2" ]; then
file2="$CURRENT_BACKUP"
fi
# Handle relative paths and cross-system backups
if [[ ! "$file2" = /* ]]; then
# Check current system first
if [ -f "$ARCHIVE_DIR/$file2" ]; then
file2="$ARCHIVE_DIR/$file2"
else
# Search in other system directories
local found_file=""
for system_dir in "$BACKUP_ROOT"/*; do
if [ -f "$system_dir/archive/$file2" ]; then
found_file="$system_dir/archive/$file2"
local source_hostname=$(basename "$system_dir")
log_info "Found backup in $source_hostname system directory"
break
fi
done
if [ -n "$found_file" ]; then
file2="$found_file"
else
file2="$ARCHIVE_DIR/$file2" # Default back to current system
fi
fi
fi
if [ ! -f "$file2" ]; then
log_error "Comparison file not found: $file2"
return 1
fi
log_info "Comparing current crontab ($HOSTNAME) with: $(basename "$file2")"
echo
# Clean comparison files (remove metadata)
grep -v "^# Backup" "$file1" > /tmp/clean_file1.tmp 2>/dev/null || touch /tmp/clean_file1.tmp
grep -v "^# Backup" "$file2" > /tmp/clean_file2.tmp 2>/dev/null || touch /tmp/clean_file2.tmp
if diff -u /tmp/clean_file1.tmp /tmp/clean_file2.tmp; then
log_success "Crontabs are identical"
else
log_warning "Crontabs differ (see above)"
fi
# Cleanup
rm -f /tmp/current_crontab.tmp /tmp/clean_file1.tmp /tmp/clean_file2.tmp
}
validate_crontab_syntax() {
local crontab_file="${1:-current}"
if [ "$crontab_file" = "current" ]; then
sudo crontab -l > /tmp/validate_crontab.tmp 2>/dev/null || echo "# No current crontab" > /tmp/validate_crontab.tmp
crontab_file="/tmp/validate_crontab.tmp"
fi
if [ ! -f "$crontab_file" ]; then
log_error "Crontab file not found: $crontab_file"
return 1
fi
log_info "Validating crontab syntax: $crontab_file"
local line_num=0
local errors=0
while IFS= read -r line; do
line_num=$((line_num + 1))
# Skip comments and empty lines
if [[ "$line" =~ ^[[:space:]]*# ]] || [[ "$line" =~ ^[[:space:]]*$ ]]; then
continue
fi
# Basic cron format validation
if ! [[ "$line" =~ ^[[:space:]]*([0-9*,-]+[[:space:]]+){4}[0-9*,-]+[[:space:]].+ ]]; then
log_error "Line $line_num: Invalid cron format: $line"
errors=$((errors + 1))
fi
# Check for common issues
if [[ "$line" =~ \$\? ]] && [[ ! "$line" =~ \{ ]]; then
log_warning "Line $line_num: \$? outside of command group may not work as expected"
fi
done < "$crontab_file"
if [ $errors -eq 0 ]; then
log_success "Crontab syntax validation passed"
else
log_error "Found $errors syntax errors"
fi
rm -f /tmp/validate_crontab.tmp
return $errors
}
cleanup_old_backups() {
local keep_days="${1:-30}"
local target_hostname="${2:-$HOSTNAME}"
local deleted_count=0
if [ "$target_hostname" = "all" ]; then
log_info "Cleaning up backups older than $keep_days days for all systems"
for system_dir in "$BACKUP_ROOT"/*; do
if [ -d "$system_dir/archive" ]; then
local system_name=$(basename "$system_dir")
log_info "Cleaning backups for $system_name"
while IFS= read -r -d '' backup; do
if [ -f "$backup" ]; then
rm "$backup"
((deleted_count++))
log_info "Deleted old backup: $(basename "$backup") from $system_name"
fi
done < <(find "$system_dir/archive" -name "*.backup" -mtime +$keep_days -print0 2>/dev/null)
fi
done
else
local target_dir="$BACKUP_ROOT/$target_hostname/archive"
log_info "Cleaning up backups older than $keep_days days for $target_hostname"
if [ -d "$target_dir" ]; then
while IFS= read -r -d '' backup; do
if [ -f "$backup" ]; then
rm "$backup"
((deleted_count++))
log_info "Deleted old backup: $(basename "$backup")"
fi
done < <(find "$target_dir" -name "*.backup" -mtime +$keep_days -print0 2>/dev/null)
else
log_warning "Backup directory not found: $target_dir"
fi
fi
if [ $deleted_count -eq 0 ]; then
log_info "No old backups found to clean up"
else
log_success "Cleaned up $deleted_count old backup(s)"
fi
}
setup_automated_backup() {
log_info "Setting up automated daily crontab backup"
local backup_script="$SCRIPT_DIR/crontab-backup-system.sh"
local backup_entry="0 0 * * * $backup_script backup auto --auto-cleanup 2>&1 | logger -t crontab-backup -p user.info"
# Check if backup entry already exists
if sudo crontab -l 2>/dev/null | grep -q "crontab-backup-system.sh"; then
log_warning "Automated backup entry already exists"
return 0
fi
# Add the backup entry to current crontab
(sudo crontab -l 2>/dev/null; echo "$backup_entry") | sudo crontab -
log_success "Automated daily backup configured for $HOSTNAME"
log_info "Backups will run daily at midnight and be logged to syslog"
}
migrate_legacy_backups() {
local legacy_dir="$SCRIPT_DIR/crontab-backups"
local legacy_archive="$legacy_dir/archive"
# Check if legacy structure exists (without hostname subdirectory)
if [ -d "$legacy_archive" ] && [ "$legacy_dir" != "$BACKUP_DIR" ]; then
log_info "Found legacy backup structure, migrating to hostname-based structure"
# Create new structure
mkdir -p "$BACKUP_DIR" "$ARCHIVE_DIR"
# Move backups and rename them to include hostname
local migrated_count=0
for backup in "$legacy_archive"/*.backup; do
if [ -f "$backup" ]; then
local filename=$(basename "$backup")
local new_filename="${HOSTNAME}-${filename}"
if cp "$backup" "$ARCHIVE_DIR/$new_filename"; then
log_success "Migrated: $filename -> $new_filename"
((migrated_count++))
else
log_error "Failed to migrate: $filename"
fi
fi
done
# Move current backup if it exists
if [ -f "$legacy_dir/current-crontab.backup" ]; then
cp "$legacy_dir/current-crontab.backup" "$CURRENT_BACKUP"
log_success "Migrated current backup"
fi
if [ $migrated_count -gt 0 ]; then
log_success "Migrated $migrated_count backup(s) to new structure"
echo
log_warning "Legacy backups remain in $legacy_archive"
log_info "You can safely remove the legacy directory after verifying the migration"
echo " rm -rf '$legacy_dir'"
fi
fi
}
list_all_systems() {
log_info "All systems with crontab backups:"
echo
if [ ! -d "$BACKUP_ROOT" ]; then
log_warning "No backup root directory found: $BACKUP_ROOT"
return 1
fi
printf "%-15s %-10s %-20s %-30s\n" "System" "Backups" "Latest Backup" "Status"
printf "%-15s %-10s %-20s %-30s\n" "------" "-------" "-------------" "------"
local found_systems=false
for system_dir in "$BACKUP_ROOT"/*; do
if [ -d "$system_dir" ]; then
local system_name=$(basename "$system_dir")
# Skip legacy archive directory - it's not a system
if [ "$system_name" = "archive" ]; then
continue
fi
found_systems=true
local backup_count=$(ls -1 "$system_dir/archive"/*.backup 2>/dev/null | wc -l || echo "0")
local latest_backup="None"
local status="Inactive"
if [ -f "$system_dir/current-crontab.backup" ]; then
latest_backup=$(stat -c %y "$system_dir/current-crontab.backup" | cut -d' ' -f1)
# Check if backup is recent (within 7 days)
local backup_age=$(( ($(date +%s) - $(stat -c %Y "$system_dir/current-crontab.backup")) / 86400 ))
if [ $backup_age -le 7 ]; then
status="Active"
elif [ $backup_age -le 30 ]; then
status="Recent"
else
status="Stale"
fi
fi
# Use printf with color formatting
if [ "$status" = "Active" ]; then
printf "%-15s %-10s %-20s ${GREEN}%-30s${NC}\n" "$system_name" "$backup_count" "$latest_backup" "$status"
elif [ "$status" = "Recent" ]; then
printf "%-15s %-10s %-20s ${YELLOW}%-30s${NC}\n" "$system_name" "$backup_count" "$latest_backup" "$status"
elif [ "$status" = "Stale" ]; then
printf "%-15s %-10s %-20s ${RED}%-30s${NC}\n" "$system_name" "$backup_count" "$latest_backup" "$status"
else
printf "%-15s %-10s %-20s %-30s\n" "$system_name" "$backup_count" "$latest_backup" "$status"
fi
fi
done
if [ "$found_systems" = false ]; then
log_warning "No systems found with backups"
fi
echo
}
show_status() {
local target_hostname="${1:-$HOSTNAME}"
if [ "$target_hostname" = "all" ]; then
log_info "Crontab Backup System Status - All Systems"
echo
for system_dir in "$BACKUP_ROOT"/*; do
if [ -d "$system_dir" ]; then
local system_name=$(basename "$system_dir")
# Skip legacy archive directory - it's not a system
if [ "$system_name" = "archive" ]; then
continue
fi
echo -e "${CYAN}=== $system_name ===${NC}"
local backup_count=$(ls -1 "$system_dir/archive"/*.backup 2>/dev/null | wc -l || echo "0")
echo " - Total backups: $backup_count"
echo " - Backup directory: $system_dir"
if [ -f "$system_dir/current-crontab.backup" ]; then
echo " - Latest backup: $(stat -c %y "$system_dir/current-crontab.backup" | cut -d'.' -f1)"
else
echo " - Latest backup: None"
fi
echo
fi
done
else
log_info "Crontab Backup System Status - $target_hostname"
echo
# Current crontab info
if [ "$target_hostname" = "$HOSTNAME" ]; then
local cron_count=$(sudo crontab -l 2>/dev/null | grep -c "^[^#]" || echo "0")
echo -e "${CYAN}Current Crontab ($HOSTNAME):${NC}"
echo " - Active entries: $cron_count"
echo " - Last modified: $(stat -c %y /var/spool/cron/crontabs/root 2>/dev/null | cut -d'.' -f1 || echo "Unknown")"
echo
fi
# Backup info for specified system
local target_dir="$BACKUP_ROOT/$target_hostname"
local backup_count=$(ls -1 "$target_dir/archive"/*.backup 2>/dev/null | wc -l || echo "0")
echo -e "${CYAN}Backups ($target_hostname):${NC}"
echo " - Total backups: $backup_count"
echo " - Backup directory: $target_dir"
if [ -f "$target_dir/current-crontab.backup" ]; then
echo " - Latest backup: $(stat -c %y "$target_dir/current-crontab.backup" | cut -d'.' -f1)"
else
echo " - Latest backup: None"
fi
echo
fi
# Log monitoring
echo -e "${CYAN}Log Monitoring:${NC}"
echo " - Management log: $LOG_DIR/crontab-management.log"
echo " - System logs: journalctl -t crontab-backup"
echo
}
show_usage() {
echo -e "${PURPLE}Crontab Backup and Recovery System (Multi-System)${NC}"
echo
echo "Usage: $0 [COMMAND] [OPTIONS]"
echo
echo "Commands:"
echo " backup [TYPE] Create a timestamped backup (default: manual)"
echo " list [HOSTNAME] List backups for hostname (default: current system)"
echo " list-systems Show all systems with backups"
echo " restore FILE Restore crontab from backup file"
echo " compare [FILE1] [FILE2] Compare current crontab with backup"
echo " validate [FILE] Validate crontab syntax"
echo " cleanup [DAYS] [HOSTNAME] Clean up old backups (default: 30 days, current system)"
echo " status [HOSTNAME|all] Show system status (default: current system)"
echo " setup-auto Setup automated daily backups"
echo " migrate Migrate legacy backups to hostname structure"
echo " import FILE SYSTEM [TYPE] Import backup from external source"
echo " create-test-systems Create test systems for demonstration"
echo " help Show this help message"
echo
echo "Multi-System Examples:"
echo " $0 backup pre-upgrade # Backup current system"
echo " $0 list io # List backups for 'io' system"
echo " $0 restore io-crontab-manual-20250526_120000.backup"
echo " $0 compare current europa-crontab-manual-20250526_120000.backup"
echo " $0 cleanup 7 all # Clean up all systems"
echo " $0 status all # Show status for all systems"
echo " $0 list-systems # Show all available systems"
echo " $0 import /path/to/crontab.backup io manual # Import backup for io system"
echo
echo "Current System: $HOSTNAME"
echo "Backup Root: $BACKUP_ROOT"
echo
}
create_test_systems() {
log_info "Creating test backup structure for io, europa, and racknerd systems"
# Create sample systems directories
local test_systems=("io" "racknerd")
for system in "${test_systems[@]}"; do
local system_dir="$BACKUP_ROOT/$system"
local system_archive="$system_dir/archive"
mkdir -p "$system_archive"
# Create a sample backup for each system
local timestamp=$(date +%Y%m%d_%H%M%S)
local sample_backup="$system_archive/${system}-crontab-sample-${timestamp}.backup"
# Create sample crontab content for each system
case "$system" in
"io")
cat > "$sample_backup" << EOF
# Sample crontab for io system
0 2 * * * /home/user/backup-docker.sh 2>&1 | logger -t docker-backup -p user.info
30 3 * * * /home/user/backup-media.sh 2>&1 | logger -t media-backup -p user.info
0 4 * * * /home/user/validate-backups.sh 2>&1 | logger -t backup-validation -p user.info
# Backup created: $(date)
# Backup type: sample
# System: $system
# User: root
# Full system info: Linux $system 5.15.0-generic #1 SMP x86_64 GNU/Linux
EOF
;;
"racknerd")
cat > "$sample_backup" << EOF
# Sample crontab for racknerd system
0 1 * * * /home/user/backup-plex.sh 2>&1 | logger -t plex-backup -p user.info
15 1 * * * /home/user/move-backups.sh 2>&1 | logger -t backup-move -p user.info
0 5 * * 0 /home/user/cleanup-old-backups.sh 2>&1 | logger -t backup-cleanup -p user.info
# Backup created: $(date)
# Backup type: sample
# System: $system
# User: root
# Full system info: Linux $system 5.4.0-generic #1 SMP x86_64 GNU/Linux
EOF
;;
esac
# Create current backup file
cp "$sample_backup" "$system_dir/current-crontab.backup"
log_success "Created test system: $system with sample backup"
done
log_success "Test systems created successfully"
}
import_backup() {
local source_file="$1"
local source_system="$2"
local backup_type="${3:-imported}"
if [ -z "$source_file" ] || [ -z "$source_system" ]; then
log_error "Usage: import_backup <source_file> <source_system> [backup_type]"
return 1
fi
if [ ! -f "$source_file" ]; then
log_error "Source file not found: $source_file"
return 1
fi
local target_dir="$BACKUP_ROOT/$source_system"
local target_archive="$target_dir/archive"
mkdir -p "$target_archive"
local timestamp=$(date +%Y%m%d_%H%M%S)
local target_file="$target_archive/${source_system}-crontab-${backup_type}-${timestamp}.backup"
# Copy and add metadata
cp "$source_file" "$target_file"
# Add metadata if not present
if ! grep -q "^# Backup created:" "$target_file"; then
cat >> "$target_file" << EOF
# Backup created: $(date)
# Backup type: $backup_type
# System: $source_system
# User: root
# Imported from: $source_file
EOF
fi
# Update current backup
cp "$target_file" "$target_dir/current-crontab.backup"
log_success "Imported backup for $source_system: $target_file"
}
# Main command handling
case "${1:-help}" in
backup)
# Check for auto-cleanup flag
if [[ "${@}" == *"--auto-cleanup"* ]]; then
create_timestamped_backup "${2:-auto}"
cleanup_old_backups 30
else
create_timestamped_backup "${2:-manual}"
fi
;;
list)
list_backups "$2"
;;
list-systems)
list_all_systems
;;
restore)
if [ -z "$2" ]; then
log_error "Please specify a backup file to restore"
list_backups
exit 1
fi
restore_from_backup "$2"
;;
compare)
compare_crontabs "$2" "$3"
;;
validate)
validate_crontab_syntax "$2"
;;
cleanup)
cleanup_old_backups "${2:-30}" "${3:-$HOSTNAME}"
;;
status)
show_status "${2:-$HOSTNAME}"
;;
setup-auto)
setup_automated_backup
;;
migrate)
migrate_legacy_backups
;;
import)
if [ -z "$2" ] || [ -z "$3" ]; then
log_error "Please specify source file and target system"
echo "Usage: $0 import <source_file> <target_system> [backup_type]"
exit 1
fi
import_backup "$2" "$3" "$4"
;;
create-test-systems)
create_test_systems
;;
help|*)
show_usage
;;
esac
# Auto-migrate legacy backups on first run
if [ ! -d "$BACKUP_DIR" ] && [ "$1" != "help" ] && [ "$1" != "migrate" ]; then
migrate_legacy_backups
fi

8
crontab/crontab-backups/europa/current-crontab.backup Normal file
View File

@@ -0,0 +1,8 @@
0 1 * * * /home/acedanger/shell/move-backups.sh 2>&1 | logger -t backup-move -p user.info
0 2 * * * { echo "Starting .env files backup"; /home/acedanger/shell/backup-env-files.sh; echo ".env backup completed with exit code: $?"; } 2>&1 | logger -t env-backup -p user.info
15 4 * * * { echo "Starting Plex backup"; /home/acedanger/shell/plex/backup-plex.sh --non-interactive --auto-repair; echo "Plex backup completed with exit code: $?"; } 2>&1 | logger -t plex-backup -p user.info
0 7 * * * { echo "Starting Plex backup validation"; /home/acedanger/shell/plex/validate-plex-backups.sh --fix; echo "Validation completed with exit code: $?"; } 2>&1 | logger -t plex-validation -p user.info
0 5 * * 1 { echo "Starting Immich database backup move"; if mv /mnt/share/media/immich/uploads/backups/immich-db-backup* /mnt/share/media/backups/immich 2>/dev/null; then echo "Immich backup move completed successfully"; else echo "No Immich backup files found or move failed"; fi; } 2>&1 | logger -t immich-backup -p user.info
0 4 * * * /home/acedanger/shell/crontab/crontab-backup-system.sh backup auto --auto-cleanup 2>&1 | logger -t crontab-backup -p user.info
0 8 * * 0 { echo "Starting weekly Plex backup report generation"; /home/acedanger/shell/plex/validate-plex-backups.sh --report; echo "Weekly report generation completed with exit code: $?"; } 2>&1 | logger -t plex-report -p user.info
30 8 * * 0 { echo "Starting .env backup validation"; /home/acedanger/shell/validate-env-backups.sh; echo ".env validation completed with exit code: $?"; } 2>&1 | logger -t env-validation -p user.info

4
crontab/crontab-backups/io/current-crontab.backup Normal file
View File

@@ -0,0 +1,4 @@
0 2 * * * { echo "Starting Docker backup"; /home/acedanger/shell/backup-docker.sh; echo "Docker backup completed with exit code: $?"; } 2>&1 | logger -t docker-backup -p user.info
0 4 * * * /home/acedanger/shell/crontab/crontab-backup-system.sh backup auto --auto-cleanup 2>&1 | logger -t crontab-backup -p user.info
0 3 * * * { echo "Starting .env files backup"; /home/acedanger/shell/backup-env-files.sh; echo ".env backup completed with exit code: $?"; } 2>&1 | logger -t env-backup -p user.info
30 8 * * 0 { echo "Starting .env backup validation"; /home/acedanger/shell/validate-env-backups.sh; echo ".env validation completed with exit code: $?"; } 2>&1 | logger -t env-validation -p user.info

2
crontab/crontab-backups/racknerd/current-crontab.backup Normal file
View File

@@ -0,0 +1,2 @@
0 0 * * * { echo "Starting Docker backup"; /home/acedanger/shell/backup-docker.sh; echo "Docker backup completed with exit code: $?"; } 2>&1 | logger -t docker-backup -p user.info
0 4 * * * /home/acedanger/shell/crontab/crontab-backup-system.sh backup auto --auto-cleanup 2>&1 | logger -t crontab-backup -p user.info

42
crontab/crontab-europa.txt Normal file
View File

@@ -0,0 +1,42 @@
# Enhanced Crontab Entries for Europa (Media Server)
#
# These entries include comprehensive logging to syslog with proper tags
# and error handling for better monitoring and troubleshooting
# Move the files previously backed up at 0100 (FROM racknerd TO europa)
# Logs both stdout and stderr with backup-move tag
0 1 * * * /home/acedanger/shell/move-backups.sh 2>&1 | logger -t backup-move -p user.info
# Daily .env files backup at 0200 with logging
# Backs up all Docker .env files to private Gitea repository
0 2 * * * { echo "Starting .env files backup"; /home/acedanger/shell/backup-env-files.sh; echo ".env backup completed with exit code: $?"; } 2>&1 | logger -t env-backup -p user.info
# Daily Plex database integrity check every 30 minutes
*/30 * * * * { echo "Check Plex database corruption"; /home/acedanger/shell/plex/backup-plex.sh --check-integrity --auto-repair; } 2>&1 | logger -t plex-database-integrity-check -p user.info
# Daily Plex backup at 0415 with enhanced logging
# Includes execution status and performance metrics
15 4 * * * { echo "Starting Plex backup"; /home/acedanger/shell/plex/backup-plex.sh --non-interactive --auto-repair; echo "Plex backup completed with exit code: $?"; } 2>&1 | logger -t plex-backup -p user.info
# Daily validation at 0700 with detailed logging
# Logs validation results and any auto-fixes performed
0 7 * * * { echo "Starting Plex backup validation"; /home/acedanger/shell/plex/validate-plex-backups.sh --fix; echo "Validation completed with exit code: $?"; } 2>&1 | logger -t plex-validation -p user.info
# Backup Immich database weekly (Mondays at 0500)
# Enhanced with proper logging and error handling
0 5 * * 1 { echo "Starting Immich database backup move"; if mv /mnt/share/media/immich/uploads/backups/immich-db-backup* /mnt/share/media/backups/immich 2>/dev/null; then echo "Immich backup move completed successfully"; else echo "No Immich backup files found or move failed"; fi; } 2>&1 | logger -t immich-backup -p user.info
# Daily system backup at 0400 with auto-cleanup
0 4 * * * /home/acedanger/shell/crontab/crontab-backup-system.sh backup auto --auto-cleanup 2>&1 | logger -t crontab-backup -p user.info
# Generate detailed weekly report (Sundays at 0800)
# Comprehensive reporting with system logging
0 8 * * 0 { echo "Starting weekly Plex backup report generation"; /home/acedanger/shell/plex/validate-plex-backups.sh --report; echo "Weekly report generation completed with exit code: $?"; } 2>&1 | logger -t plex-report -p user.info
# Weekly .env backup validation (Sundays at 0830)
# Validates integrity of .env backup repository
30 8 * * 0 { echo "Starting .env backup validation"; /home/acedanger/shell/validate-env-backups.sh; echo ".env validation completed with exit code: $?"; } 2>&1 | logger -t env-validation -p user.info
# Optional: Add a health check entry to monitor cron jobs (every 6 hours)
# This can help detect if any of the backup processes are failing
# 0 */6 * * * { echo "Cron health check - all backup jobs scheduled"; ps aux | grep -E "(backup-plex|validate-plex|move-backups)" | grep -v grep | wc -l; } 2>&1 | logger -t cron-health -p user.info

23
crontab/crontab-io.txt Normal file
View File

@@ -0,0 +1,23 @@
# Enhanced Crontab Entries for IO (Download/Acquisition Server)
#
# This system runs download management services (Radarr, Sonarr, SABnzbd, etc.)
# and should focus on Docker container management rather than media server tasks
# Daily Docker backup at 0200 with enhanced logging
# Backs up Docker container configurations and data
0 2 * * * { echo "Starting Docker backup"; /home/acedanger/shell/backup-docker.sh; echo "Docker backup completed with exit code: $?"; } 2>&1 | logger -t docker-backup -p user.info
# Daily system backup at 0400 with auto-cleanup
0 4 * * * /home/acedanger/shell/crontab/crontab-backup-system.sh backup auto --auto-cleanup 2>&1 | logger -t crontab-backup -p user.info
# Daily .env files backup at 0300 with enhanced logging
# Backs up Docker container .env files to Git repository
0 3 * * * { echo "Starting .env files backup"; /home/acedanger/shell/backup-env-files.sh; echo ".env backup completed with exit code: $?"; } 2>&1 | logger -t env-backup -p user.info
# Weekly .env backup validation at 0830 on Sundays
# Validates the integrity of .env backup repository
30 8 * * 0 { echo "Starting .env backup validation"; /home/acedanger/shell/validate-env-backups.sh; echo ".env validation completed with exit code: $?"; } 2>&1 | logger -t env-validation -p user.info
# Optional: Monitor Docker container health (every 6 hours)
# This can help detect if any download services are failing
# 0 */6 * * * { echo "Docker health check"; docker ps --format "table {{.Names}}\t{{.Status}}" | grep -v "Up"; } 2>&1 | logger -t docker-health -p user.info

22
crontab/crontab-racknerd.txt Normal file
View File

@@ -0,0 +1,22 @@
# Enhanced Crontab Entries for Racknerd (Backup Server)
#
# These entries include comprehensive logging to syslog with proper tags
# and error handling for better monitoring and troubleshooting
# Enhanced with proper logging and error handling
0 0 * * * { echo "Starting Docker backup"; /home/acedanger/shell/backup-docker.sh; echo "Docker backup completed with exit code: $?"; } 2>&1 | logger -t docker-backup -p user.info
# Daily system backup at 0400 with auto-cleanup
0 4 * * * /home/acedanger/shell/crontab/crontab-backup-system.sh backup auto --auto-cleanup 2>&1 | logger -t crontab-backup -p user.info
# Daily .env files backup at 0300 with enhanced logging
# Backs up Docker container .env files to Git repository
0 3 * * * { echo "Starting .env files backup"; /home/acedanger/shell/backup-env-files.sh; echo ".env backup completed with exit code: $?"; } 2>&1 | logger -t env-backup -p user.info
# Weekly .env backup validation at 0830 on Sundays
# Validates the integrity of .env backup repository
30 8 * * 0 { echo "Starting .env backup validation"; /home/acedanger/shell/validate-env-backups.sh; echo ".env validation completed with exit code: $?"; } 2>&1 | logger -t env-validation -p user.info
# Optional: Add a health check entry to monitor backup jobs (every 6 hours)
# This can help detect if the backup process is failing
# 0 */6 * * * { echo "Cron health check - Docker backup job scheduled"; ps aux | grep "backup-docker" | grep -v grep | wc -l; } 2>&1 | logger -t cron-health -p user.info

226
crontab/enhanced-crontab-system.md Normal file
View File

@@ -0,0 +1,226 @@
# Enhanced Crontab and Backup Monitoring System
## Overview
This enhanced system provides comprehensive crontab management with automatic backups, system logging, and advanced monitoring capabilities for your backup operations.
## Components
### 1. Crontab Entries (`crontab-*.txt`)
- **Move backups** (01:00): Transfers Docker backups with logging
- **Plex backup** (04:15): Daily Plex database backup with auto-repair
- **Plex validation** (07:00): Validates and fixes backup integrity
- **Immich backup** (05:00 Mon): Weekly Immich database backup move
- **Weekly report** (08:00 Sun): Comprehensive backup status report
### 2. Crontab Management (`manage-enhanced-crontab.sh`)
- Install enhanced crontab entries with validation
- Verify all backup scripts exist and are executable
- Setup automated backups and log rotation
- Integration with backup and monitoring systems
### 3. Crontab Backup System (`crontab-backup-system.sh`)
- Automated timestamped crontab backups
- Backup comparison and restoration
- Syntax validation
- Automated cleanup of old backups
- Daily automated backup scheduling
### 4. Backup Log Monitor (`backup-log-monitor.sh`)
- Real-time log monitoring with color coding
- Error analysis and reporting
- System health checks
- Comprehensive backup reports
- Service status overview
## Quick Start
### 1. Install the Enhanced System
```bash
# Make scripts executable
chmod +x manage-enhanced-crontab.sh crontab-backup-system.sh backup-log-monitor.sh
# Install enhanced crontab with all features
sudo ./manage-enhanced-crontab.sh install
```
### 2. Monitor Your Backups
```bash
# Real-time monitoring
./backup-log-monitor.sh monitor
# Check system health
./backup-log-monitor.sh health
# View recent activity
./backup-log-monitor.sh recent 24
# Generate weekly report
./backup-log-monitor.sh report 7
```
### 3. Manage Crontab Backups
```bash
# Create manual backup
./crontab-backup-system.sh backup manual
# List all backups
./crontab-backup-system.sh list
# Compare current with backup
./crontab-backup-system.sh compare current
# System status
./crontab-backup-system.sh status
```
## Features
### Enhanced Logging
All backup operations now log to syslog with specific tags:
- `plex-backup`: Plex database backup operations
- `backup-move`: Docker backup file transfers
- `plex-validation`: Backup integrity checks
- `immich-backup`: Immich database operations
- `plex-report`: Weekly reporting
- `crontab-backup`: Crontab backup operations
### Automatic Backups
- **Crontab backups**: Daily automated backups at midnight
- **Cleanup**: Automatic removal of backups older than 30 days
- **Validation**: Syntax checking before applying changes
- **Recovery**: Easy restoration from any backup point
### Health Monitoring
- Script existence and permissions
- Backup directory availability
- Recent activity tracking
- Error rate monitoring
- Overall system health scoring
### Error Handling
- Graceful failure handling in cron jobs
- Detailed error logging and reporting
- Exit code tracking for debugging
- Comprehensive error summaries
## Log Analysis
### View Real-time Logs
```bash
# All backup services
sudo journalctl -f -t plex-backup -t backup-move -t plex-validation -t immich-backup -t plex-report
# Specific service
sudo journalctl -f -t plex-backup
# With our monitor (recommended)
./backup-log-monitor.sh monitor
```
### Historical Analysis
```bash
# Last 24 hours
sudo journalctl --since '24 hours ago' -t plex-backup
# Last week with errors only
sudo journalctl --since '1 week ago' --priority=err -t plex-backup
# Using our tools
./backup-log-monitor.sh recent 24 plex-backup
./backup-log-monitor.sh errors 7
```
## Backup Recovery
### Restore Crontab from Backup
```bash
# List available backups
./crontab-backup-system.sh list
# Restore specific backup
./crontab-backup-system.sh restore crontab-manual-20250526_120000.backup
# Compare before restoring
./crontab-backup-system.sh compare current crontab-manual-20250526_120000.backup
```
### Emergency Recovery
If you need to quickly restore the original crontab:
```bash
# The system automatically creates pre-install backups
./crontab-backup-system.sh list | grep pre-install
./crontab-backup-system.sh restore [backup-filename]
```
## Maintenance
### Regular Tasks
```bash
# Weekly health check
./backup-log-monitor.sh health
# Monthly backup cleanup
./crontab-backup-system.sh cleanup 30
# Quarterly comprehensive report
./backup-log-monitor.sh report 90
```
### Troubleshooting
```bash
# Verify all components
./manage-enhanced-crontab.sh verify
# Check system status
./manage-enhanced-crontab.sh status
# View configuration
./manage-enhanced-crontab.sh show
# Monitor for issues
./backup-log-monitor.sh monitor
```
## Integration Notes
- All scripts follow the established shell repository coding standards
- Logging uses consistent tags and priorities
- Error handling preserves backup integrity
- Color-coded output for better readability
- Comprehensive documentation and help systems
## Security Considerations
- Scripts validate input and handle errors gracefully
- Backup files include metadata for tracking
- Permissions are properly managed
- Sensitive operations require sudo confirmation
- All operations are logged for audit trails
## Next Steps
1. **Install the system**: Run `sudo ./manage-enhanced-crontab.sh install`
2. **Test monitoring**: Use `./backup-log-monitor.sh monitor` during next backup
3. **Review reports**: Generate weekly reports to establish baseline
4. **Set up alerts**: Consider integrating with your notification system
5. **Document customizations**: Add any local modifications to this guide

141
crontab/fix-crontab-merging.sh Executable file
View File

@@ -0,0 +1,141 @@
#!/bin/bash
# Crontab Recovery Script
# This script fixes the crontab merging issue by restoring system-specific entries
set -e
# Color codes for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
CYAN='\033[0;36m'
NC='\033[0m' # No Color
SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
HOSTNAME=$(hostname)
log_message() {
echo -e "$(date '+%H:%M:%S') $1"
}
log_error() {
log_message "${RED}ERROR: $1${NC}"
}
log_success() {
log_message "${GREEN}SUCCESS: $1${NC}"
}
log_warning() {
log_message "${YELLOW}WARNING: $1${NC}"
}
log_info() {
log_message "${BLUE}INFO: $1${NC}"
}
show_current_problem() {
log_info "Current crontab merging issue analysis:"
echo
log_info "Current root crontab on $HOSTNAME:"
sudo crontab -l 2>/dev/null || log_warning "No crontab found"
echo
case "$HOSTNAME" in
"europa")
log_info "Europa should have:"
echo " - move-backups.sh (pulls files FROM racknerd)"
echo " - backup-plex.sh (backs up Plex)"
echo " - validate-plex-backups.sh (validates backups)"
echo " - Immich database backup move"
echo " - Weekly Plex reports"
;;
"racknerd")
log_info "Racknerd should have:"
echo " - backup-docker.sh (backs up Docker containers)"
echo " - NO move-backups.sh (that's Europa's job)"
echo " - NO Plex-related jobs (Plex runs on Europa)"
;;
*)
log_warning "Unknown hostname: $HOSTNAME"
;;
esac
echo
}
fix_crontab() {
local system_name="$1"
local crontab_file="$SCRIPT_DIR/crontab-${system_name}.txt"
if [ ! -f "$crontab_file" ]; then
log_error "System-specific crontab file not found: $crontab_file"
return 1
fi
log_info "Creating backup before fixing crontab"
if [ -f "$SCRIPT_DIR/crontab-backup-system.sh" ]; then
"$SCRIPT_DIR/crontab-backup-system.sh" backup "pre-fix"
fi
log_info "Installing correct crontab for $system_name"
# Extract just the cron entries (skip comments and empty lines)
grep -E '^[0-9]' "$crontab_file" > /tmp/cron_entries_fix.txt
if sudo crontab /tmp/cron_entries_fix.txt; then
log_success "Correct crontab installed for $system_name"
rm -f /tmp/cron_entries_fix.txt
# Create a post-fix backup
if [ -f "$SCRIPT_DIR/crontab-backup-system.sh" ]; then
"$SCRIPT_DIR/crontab-backup-system.sh" backup "post-fix"
fi
log_info "New crontab contents:"
sudo crontab -l
return 0
else
log_error "Failed to install correct crontab"
rm -f /tmp/cron_entries_fix.txt
return 1
fi
}
main() {
log_info "=== Crontab Recovery Script ==="
log_info "System: $HOSTNAME"
echo
show_current_problem
case "$HOSTNAME" in
"europa")
log_info "Fixing crontab for Europa (media server)"
fix_crontab "europa"
;;
"racknerd")
log_info "Fixing crontab for Racknerd (backup server)"
fix_crontab "racknerd"
;;
*)
log_error "Unknown system: $HOSTNAME"
log_info "This script supports: europa, racknerd"
exit 1
;;
esac
echo
log_success "Crontab recovery completed for $HOSTNAME"
log_info "The management script now uses system-specific files:"
log_info " - crontab-europa.txt"
log_info " - crontab-racknerd.txt"
echo
log_info "To manage crontabs going forward, use:"
log_info " ./manage-enhanced-crontab.sh install"
}
main "$@"

283
crontab/manage-enhanced-crontab.sh Executable file
View File

@@ -0,0 +1,283 @@
#!/bin/bash
# Enhanced Crontab Management Script
# This script helps install and manage the enhanced crontab entries with system logging
set -e
# Color codes for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
CYAN='\033[0;36m'
NC='\033[0m' # No Color
SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
HOSTNAME=$(hostname)
ENHANCED_CRONTAB_FILE="$SCRIPT_DIR/crontab-${HOSTNAME}.txt"
BACKUP_CRONTAB_FILE="/tmp/crontab-backup-$(date +%Y%m%d_%H%M%S)"
log_message() {
echo -e "$(date '+%H:%M:%S') $1"
}
log_error() {
log_message "${RED}ERROR: $1${NC}"
}
log_success() {
log_message "${GREEN}SUCCESS: $1${NC}"
}
log_warning() {
log_message "${YELLOW}WARNING: $1${NC}"
}
log_info() {
log_message "${BLUE}INFO: $1${NC}"
}
backup_current_crontab() {
log_info "Creating structured backup using crontab-backup-system for host: $HOSTNAME"
# Use the comprehensive backup system
if [ -f "$SCRIPT_DIR/crontab-backup-system.sh" ]; then
if "$SCRIPT_DIR/crontab-backup-system.sh" backup manual; then
log_success "Structured crontab backup created successfully"
else
log_warning "Structured backup failed, falling back to temporary backup"
# Fallback to simple backup
log_info "Backing up current root crontab to $BACKUP_CRONTAB_FILE"
if sudo crontab -l > "$BACKUP_CRONTAB_FILE" 2>/dev/null; then
log_success "Temporary backup created successfully"
else
log_warning "No existing crontab found or backup failed"
fi
fi
else
log_warning "crontab-backup-system.sh not found, using temporary backup"
# Fallback to simple backup
log_info "Backing up current root crontab to $BACKUP_CRONTAB_FILE"
if sudo crontab -l > "$BACKUP_CRONTAB_FILE" 2>/dev/null; then
log_success "Temporary backup created successfully"
else
log_warning "No existing crontab found or backup failed"
fi
fi
}
install_enhanced_crontab() {
log_info "Installing enhanced crontab entries for system: $HOSTNAME"
# Check for system-specific crontab file first
if [ ! -f "$ENHANCED_CRONTAB_FILE" ]; then
log_warning "System-specific crontab file not found: $ENHANCED_CRONTAB_FILE"
log_info "Available crontab files:"
ls -la "$SCRIPT_DIR"/crontab-*.txt 2>/dev/null || log_warning "No system-specific crontab files found"
log_error "No suitable crontab file found. Please create $ENHANCED_CRONTAB_FILE"
return 1
fi
# Create a backup before making changes
if [ -f "$SCRIPT_DIR/crontab-backup-system.sh" ]; then
log_info "Creating pre-install backup"
if ! "$SCRIPT_DIR/crontab-backup-system.sh" backup pre-install; then
log_warning "Pre-install backup failed (normal for systems with no existing crontab)"
fi
fi
# Extract just the cron entries (skip comments and empty lines)
grep -E '^[0-9]' "$ENHANCED_CRONTAB_FILE" > /tmp/cron_entries_only.txt
# Validate the crontab syntax before installing
log_info "Validating crontab syntax"
if [ -f "$SCRIPT_DIR/crontab-backup-system.sh" ]; then
if ! "$SCRIPT_DIR/crontab-backup-system.sh" validate /tmp/cron_entries_only.txt; then
log_error "Crontab syntax validation failed"
rm -f /tmp/cron_entries_only.txt
return 1
fi
else
log_warning "Backup script not found, skipping validation"
fi
if sudo crontab /tmp/cron_entries_only.txt; then
log_success "Enhanced crontab entries installed successfully"
rm -f /tmp/cron_entries_only.txt
# Create a post-install backup
if [ -f "$SCRIPT_DIR/crontab-backup-system.sh" ]; then
if ! "$SCRIPT_DIR/crontab-backup-system.sh" backup post-install; then
log_warning "Post-install backup failed, but crontab installation was successful"
fi
fi
else
log_error "Failed to install enhanced crontab entries"
rm -f /tmp/cron_entries_only.txt
return 1
fi
}
show_current_crontab() {
log_info "Current root crontab entries:"
echo
sudo crontab -l 2>/dev/null || log_warning "No crontab entries found"
echo
}
show_log_monitoring_commands() {
log_info "Commands to monitor backup logs:"
echo
echo -e "${CYAN}# Use the enhanced backup log monitor:${NC}"
echo "./backup-log-monitor.sh monitor # Real-time monitoring"
echo "./backup-log-monitor.sh recent 24 # Last 24 hours"
echo "./backup-log-monitor.sh health # System health check"
echo "./backup-log-monitor.sh report 7 # Weekly report"
echo
echo -e "${CYAN}# Direct journalctl commands:${NC}"
echo "sudo journalctl -f -t plex-backup -t backup-move -t plex-validation -t immich-backup -t plex-report"
echo
echo -e "${CYAN}# View logs from the last 24 hours:${NC}"
echo "sudo journalctl --since '24 hours ago' -t plex-backup -t backup-move -t plex-validation -t immich-backup -t plex-report"
echo
echo -e "${CYAN}# View only error logs:${NC}"
echo "sudo journalctl --priority=err -t plex-backup -t backup-move -t plex-validation -t immich-backup -t plex-report"
echo
echo -e "${CYAN}# View logs for a specific backup type (e.g., plex-backup):${NC}"
echo "sudo journalctl -t plex-backup --since '1 week ago'"
echo
}
setup_logrotate() {
log_info "Setting up logrotate for backup logs"
cat > /tmp/backup-logs-logrotate << 'EOF'
# Logrotate configuration for backup logs
# This ensures syslog doesn't grow too large with backup logs
/var/log/syslog {
daily
missingok
rotate 7
compress
delaycompress
notifempty
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}
EOF
if sudo cp /tmp/backup-logs-logrotate /etc/logrotate.d/backup-logs; then
log_success "Logrotate configuration installed"
else
log_warning "Failed to install logrotate configuration"
fi
rm -f /tmp/backup-logs-logrotate
}
verify_scripts_exist() {
log_info "Verifying all backup scripts exist and are executable"
local scripts=(
"/home/acedanger/shell/move-backups.sh"
"/home/acedanger/shell/plex/backup-plex.sh"
"/home/acedanger/shell/plex/validate-plex-backups.sh"
)
local all_good=true
for script in "${scripts[@]}"; do
if [ -f "$script" ]; then
if [ -x "$script" ]; then
log_success "✓ $script exists and is executable"
else
log_warning "! $script exists but is not executable"
sudo chmod +x "$script"
log_success "✓ Made $script executable"
fi
else
log_error "✗ $script not found"
all_good=false
fi
done
if $all_good; then
log_success "All backup scripts are ready"
else
log_error "Some backup scripts are missing"
return 1
fi
}
show_usage() {
echo "Enhanced Crontab Management Script"
echo
echo "Usage: $0 [OPTION]"
echo
echo "Options:"
echo " install Install the enhanced crontab entries with backup system"
echo " show Show current crontab entries"
echo " backup Backup current crontab only"
echo " verify Verify all scripts exist and are executable"
echo " monitor Show log monitoring commands"
echo " logrotate Setup logrotate for backup logs"
echo " status Show backup system health status"
echo " help Show this help message"
echo
echo "Additional Tools:"
echo " ./crontab-backup-system.sh Comprehensive crontab backup management"
echo " ./backup-log-monitor.sh Advanced backup log monitoring"
echo
}
case "${1:-help}" in
install)
# Check if --help flag is present
if [[ "$*" == *"--help"* ]]; then
show_usage
exit 0
fi
verify_scripts_exist
backup_current_crontab
install_enhanced_crontab
show_current_crontab
setup_logrotate
show_log_monitoring_commands
# Setup automated backup system
if [ -f "$SCRIPT_DIR/crontab-backup-system.sh" ]; then
log_info "Setting up automated crontab backup system"
"$SCRIPT_DIR/crontab-backup-system.sh" setup-auto
fi
;;
show)
show_current_crontab
;;
backup)
backup_current_crontab
;;
verify)
verify_scripts_exist
;;
monitor)
show_log_monitoring_commands
;;
logrotate)
setup_logrotate
;;
status)
if [ -f "$SCRIPT_DIR/backup-log-monitor.sh" ]; then
"$SCRIPT_DIR/backup-log-monitor.sh" health
else
log_warning "Backup log monitor not found, showing basic status"
show_current_crontab
fi
;;
help|*)
show_usage
;;
esac

280
crontab/multi-system-crontab-management.md Normal file
View File

@@ -0,0 +1,280 @@
# Multi-System Crontab Management Guide
## Overview
The enhanced crontab backup system now supports managing crontab backups across multiple systems using a hostname-based directory structure. This enables centralized backup management for distributed environments.
## System Architecture
### Directory Structure
```
crontab-backups/
├── europa/ # Current system (example)
│ ├── current-crontab.backup
│ └── archive/
│ ├── europa-crontab-initial-20250526_101354.backup
│ └── europa-crontab-pre-install-20250526_100622.backup
├── io/ # Remote system backups
│ ├── current-crontab.backup
│ └── archive/
│ └── io-crontab-sample-20250526_101558.backup
└── racknerd/ # Another remote system
├── current-crontab.backup
└── archive/
└── racknerd-crontab-sample-20250526_101558.backup
```
### File Naming Convention
- Format: `{hostname}-crontab-{type}-{timestamp}.backup`
- Examples:
- `europa-crontab-manual-20250526_101354.backup`
- `io-crontab-pre-upgrade-20250526_120000.backup`
- `racknerd-crontab-auto-20250526_000001.backup`
## Features
### 🔄 Multi-System Support
- **Hostname-based organization**: Each system gets its own directory
- **Cross-system operations**: View, compare, and restore backups from any system
- **Centralized management**: Manage all systems from a single location
### 📊 System Status Monitoring
```bash
# View status for current system
./crontab-backup-system.sh status
# View status for specific system
./crontab-backup-system.sh status io
# View status for all systems
./crontab-backup-system.sh status all
```
### 🗂️ Backup Operations
```bash
# Create backup on current system
./crontab-backup-system.sh backup pre-upgrade
# List backups for specific system
./crontab-backup-system.sh list io
# List all systems with backups
./crontab-backup-system.sh list-systems
# Import backup from external source
./crontab-backup-system.sh import /path/to/backup.txt io manual
```
### 🔍 Cross-System Comparison
```bash
# Compare current crontab with backup from another system
./crontab-backup-system.sh compare current io-crontab-sample-20250526_101558.backup
# Compare two backups from different systems
./crontab-backup-system.sh compare europa-crontab-manual-20250526_101354.backup racknerd-crontab-sample-20250526_101558.backup
```
### 🧹 Cleanup Management
```bash
# Clean up backups older than 30 days for current system
./crontab-backup-system.sh cleanup 30
# Clean up backups for specific system
./crontab-backup-system.sh cleanup 7 io
# Clean up backups for all systems
./crontab-backup-system.sh cleanup 30 all
```
## Enhanced Logging Integration
All backup operations now integrate with system logging:
### Syslog Integration
- **Tag-based logging**: Each operation uses specific syslog tags
- **Priority levels**: Different priorities for info, warnings, and errors
- **Performance monitoring**: Execution time tracking for all operations
### Example Enhanced Crontab Entries
```bash
# Plex backup with comprehensive logging
15 4 * * * /home/acedanger/shell/plex/backup-plex.sh 2>&1 | logger -t plex-backup -p user.info
# Backup move operation with error handling
0 1 * * * /home/acedanger/shell/move-backups.sh 2>&1 | logger -t backup-move -p user.info
# Validation with performance tracking
0 7 * * * /home/acedanger/shell/validate-plex-backups.sh --fix 2>&1 | logger -t plex-validation -p user.info
```
### Log Monitoring
```bash
# View all backup-related logs
journalctl -t plex-backup -t backup-move -t plex-validation -f
# View logs for specific operation
journalctl -t plex-backup --since "1 hour ago"
# Monitor backup performance
./backup-log-monitor.sh --real-time
```
## Migration from Legacy Structure
The system automatically detects and migrates legacy backup structures:
### Automatic Migration
- **Legacy detection**: Automatically detects old `crontab-backups/archive/` structure
- **Hostname prefix**: Adds hostname prefix to existing backup files
- **Backward compatibility**: Preserves all existing backup data
- **Safe migration**: Original files remain untouched until manual cleanup
### Manual Migration
```bash
# Force migration of legacy backups
./crontab-backup-system.sh migrate
```
## Production Deployment
### System Setup
1. **Deploy script**: Copy `crontab-backup-system.sh` to each system
2. **Configure permissions**: Ensure proper read/write access to backup directories
3. **Setup automation**: Configure automated daily backups
### Automated Backup Setup
```bash
# Setup automated daily backups on each system
./crontab-backup-system.sh setup-auto
```
This adds the following entry to the system crontab:
```bash
0 0 * * * /path/to/crontab-backup-system.sh backup auto --auto-cleanup 2>&1 | logger -t crontab-backup -p user.info
```
### Cross-System Synchronization
For distributed environments, consider setting up backup synchronization:
```bash
# Example rsync command to sync backups from remote systems
rsync -avz europa:/home/acedanger/shell/crontab-backups/europa/ /home/acedanger/shell/crontab-backups/europa/
rsync -avz io:/home/acedanger/shell/crontab-backups/io/ /home/acedanger/shell/crontab-backups/io/
```
## Security Considerations
### File Permissions
- **Backup directories**: Restrict access to authorized users only
- **Log files**: Ensure proper log rotation and access controls
- **Remote access**: Use secure methods (SSH, rsync) for cross-system operations
### Backup Integrity
- **Validation**: Regular syntax validation of backup files
- **Checksums**: Consider adding checksum verification for critical backups
- **Retention**: Implement appropriate backup retention policies
## Advanced Use Cases
### Disaster Recovery
```bash
# Restore from specific system backup during emergency
./crontab-backup-system.sh restore io-crontab-pre-incident-20250526_101354.backup
# Compare pre-incident and post-incident configurations
./crontab-backup-system.sh compare io-crontab-pre-incident-20250526_101354.backup current
```
### Configuration Management
```bash
# Standardize crontab across multiple systems
./crontab-backup-system.sh compare europa-crontab-standard-20250526_101354.backup io-crontab-current-20250526_120000.backup
# Validate configurations before deployment
./crontab-backup-system.sh validate new-crontab-config.txt
```
### Compliance and Auditing
- **Change tracking**: Complete history of all crontab changes across systems
- **Audit trails**: System logs provide comprehensive audit information
- **Compliance reporting**: Generate reports showing backup frequency and success rates
## Monitoring and Alerting
### Health Checks
```bash
# Check backup system health
./crontab-backup-system.sh status all
# Monitor recent backup activity
./backup-log-monitor.sh --health-check
```
### Alert Integration
Consider integrating with monitoring systems:
- **Backup failures**: Alert when backups fail or are missing
- **Old backups**: Alert when systems haven't been backed up recently
- **Disk space**: Monitor backup directory disk usage
## Best Practices
1. **Regular Testing**: Periodically test backup restoration procedures
2. **Documentation**: Keep records of system configurations and backup schedules
3. **Automation**: Use automated cleanup to prevent disk space issues
4. **Monitoring**: Implement comprehensive monitoring and alerting
5. **Security**: Regularly review and update access controls
## Support and Troubleshooting
### Common Issues
- **Permission errors**: Ensure proper file permissions on backup directories
- **Missing backups**: Check automated backup cron entries
- **Syntax errors**: Use validation feature before deploying new crontabs
### Debug Mode
Enable verbose logging for troubleshooting:
```bash
# Add debug logging to any command
./crontab-backup-system.sh status all 2>&1 | tee debug.log
```
### Log Analysis
```bash
# Analyze backup patterns
grep "SUCCESS" logs/crontab-management.log | tail -20
# Check for errors
grep "ERROR" logs/crontab-management.log | tail -10
```
---
*This multi-system crontab management solution provides robust, scalable backup management for distributed environments while maintaining simplicity and reliability.*