shell/docs/enhanced-crontab-system.md

Enhanced Crontab and Backup Monitoring System

Overview

This enhanced system provides comprehensive crontab management with automatic backups, system logging, and advanced monitoring capabilities for your backup operations.

Components

1. Enhanced Crontab Entries (enhanced-crontab.txt)

• Move backups (01:00): Transfers Docker backups with logging
• Plex backup (04:15): Daily Plex database backup with auto-repair
• Plex validation (07:00): Validates and fixes backup integrity
• Immich backup (05:00 Mon): Weekly Immich database backup move
• Weekly report (08:00 Sun): Comprehensive backup status report

2. Crontab Management (manage-enhanced-crontab.sh)

• Install enhanced crontab entries with validation
• Verify all backup scripts exist and are executable
• Setup automated backups and log rotation
• Integration with backup and monitoring systems

3. Crontab Backup System (crontab-backup-system.sh)

• Automated timestamped crontab backups
• Backup comparison and restoration
• Syntax validation
• Automated cleanup of old backups
• Daily automated backup scheduling

4. Backup Log Monitor (backup-log-monitor.sh)

• Real-time log monitoring with color coding
• Error analysis and reporting
• System health checks
• Comprehensive backup reports
• Service status overview

Quick Start

1. Install the Enhanced System

# Make scripts executable
chmod +x manage-enhanced-crontab.sh crontab-backup-system.sh backup-log-monitor.sh

# Install enhanced crontab with all features
sudo ./manage-enhanced-crontab.sh install

2. Monitor Your Backups

# Real-time monitoring
./backup-log-monitor.sh monitor

# Check system health
./backup-log-monitor.sh health

# View recent activity
./backup-log-monitor.sh recent 24

# Generate weekly report
./backup-log-monitor.sh report 7

3. Manage Crontab Backups

# Create manual backup
./crontab-backup-system.sh backup manual

# List all backups
./crontab-backup-system.sh list

# Compare current with backup
./crontab-backup-system.sh compare current

# System status
./crontab-backup-system.sh status

Features

Enhanced Logging

All backup operations now log to syslog with specific tags:

• plex-backup: Plex database backup operations
• backup-move: Docker backup file transfers
• plex-validation: Backup integrity checks
• immich-backup: Immich database operations
• plex-report: Weekly reporting
• crontab-backup: Crontab backup operations

Automatic Backups

• Crontab backups: Daily automated backups at midnight
• Cleanup: Automatic removal of backups older than 30 days
• Validation: Syntax checking before applying changes
• Recovery: Easy restoration from any backup point

Health Monitoring

• Script existence and permissions
• Backup directory availability
• Recent activity tracking
• Error rate monitoring
• Overall system health scoring

Error Handling

• Graceful failure handling in cron jobs
• Detailed error logging and reporting
• Exit code tracking for debugging
• Comprehensive error summaries

Log Analysis

View Real-time Logs

# All backup services
sudo journalctl -f -t plex-backup -t backup-move -t plex-validation -t immich-backup -t plex-report

# Specific service
sudo journalctl -f -t plex-backup

# With our monitor (recommended)
./backup-log-monitor.sh monitor

Historical Analysis

# Last 24 hours
sudo journalctl --since '24 hours ago' -t plex-backup

# Last week with errors only
sudo journalctl --since '1 week ago' --priority=err -t plex-backup

# Using our tools
./backup-log-monitor.sh recent 24 plex-backup
./backup-log-monitor.sh errors 7

Backup Recovery

Restore Crontab from Backup

# List available backups
./crontab-backup-system.sh list

# Restore specific backup
./crontab-backup-system.sh restore crontab-manual-20250526_120000.backup

# Compare before restoring
./crontab-backup-system.sh compare current crontab-manual-20250526_120000.backup

Emergency Recovery

If you need to quickly restore the original crontab:

# The system automatically creates pre-install backups
./crontab-backup-system.sh list | grep pre-install
./crontab-backup-system.sh restore [backup-filename]

Maintenance

Regular Tasks

# Weekly health check
./backup-log-monitor.sh health

# Monthly backup cleanup
./crontab-backup-system.sh cleanup 30

# Quarterly comprehensive report
./backup-log-monitor.sh report 90

Troubleshooting

# Verify all components
./manage-enhanced-crontab.sh verify

# Check system status
./manage-enhanced-crontab.sh status

# View configuration
./manage-enhanced-crontab.sh show

# Monitor for issues
./backup-log-monitor.sh monitor

Integration Notes

• All scripts follow the established shell repository coding standards
• Logging uses consistent tags and priorities
• Error handling preserves backup integrity
• Color-coded output for better readability
• Comprehensive documentation and help systems

Security Considerations

• Scripts validate input and handle errors gracefully
• Backup files include metadata for tracking
• Permissions are properly managed
• Sensitive operations require sudo confirmation
• All operations are logged for audit trails

Next Steps

1. Install the system: Run sudo ./manage-enhanced-crontab.sh install
2. Test monitoring: Use ./backup-log-monitor.sh monitor during next backup
3. Review reports: Generate weekly reports to establish baseline
4. Set up alerts: Consider integrating with your notification system
5. Document customizations: Add any local modifications to this guide