# Enhanced Crontab and Backup Monitoring System

## Overview

This enhanced system provides comprehensive crontab management with automatic backups, system logging, and advanced monitoring capabilities for your backup operations.

## Components

### 1. Crontab Entries (`crontab-*.txt`)

- **Move backups** (01:00): Transfers Docker backups with logging
- **Plex backup** (04:15): Daily Plex database backup with auto-repair
- **Plex validation** (07:00): Validates and fixes backup integrity
- **Immich backup** (05:00 Mon): Weekly Immich database backup move
- **Weekly report** (08:00 Sun): Comprehensive backup status report

### 2. Crontab Management (`manage-enhanced-crontab.sh`)

- Install enhanced crontab entries with validation
- Verify all backup scripts exist and are executable
- Setup automated backups and log rotation
- Integration with backup and monitoring systems

### 3. Crontab Backup System (`crontab-backup-system.sh`)

- Automated timestamped crontab backups
- Backup comparison and restoration
- Syntax validation
- Automated cleanup of old backups
- Daily automated backup scheduling

### 4. Backup Log Monitor (`backup-log-monitor.sh`)

- Real-time log monitoring with color coding
- Error analysis and reporting
- System health checks
- Comprehensive backup reports
- Service status overview

## Quick Start

### 1. Install the Enhanced System

```bash
# Make scripts executable
chmod +x manage-enhanced-crontab.sh crontab-backup-system.sh backup-log-monitor.sh

# Install enhanced crontab with all features
sudo ./manage-enhanced-crontab.sh install
```

### 2. Monitor Your Backups

```bash
# Real-time monitoring
./backup-log-monitor.sh monitor

# Check system health
./backup-log-monitor.sh health

# View recent activity
./backup-log-monitor.sh recent 24

# Generate weekly report
./backup-log-monitor.sh report 7
```

### 3. Manage Crontab Backups

```bash
# Create manual backup
./crontab-backup-system.sh backup manual

# List all backups
./crontab-backup-system.sh list

# Compare current with backup
./crontab-backup-system.sh compare current

# System status
./crontab-backup-system.sh status
```

## Features

### Enhanced Logging

All backup operations now log to syslog with specific tags:

- `plex-backup`: Plex database backup operations
- `backup-move`: Docker backup file transfers
- `plex-validation`: Backup integrity checks
- `immich-backup`: Immich database operations
- `plex-report`: Weekly reporting
- `crontab-backup`: Crontab backup operations

### Automatic Backups

- **Crontab backups**: Daily automated backups at midnight
- **Cleanup**: Automatic removal of backups older than 30 days
- **Validation**: Syntax checking before applying changes
- **Recovery**: Easy restoration from any backup point

### Health Monitoring

- Script existence and permissions
- Backup directory availability
- Recent activity tracking
- Error rate monitoring
- Overall system health scoring

### Error Handling

- Graceful failure handling in cron jobs
- Detailed error logging and reporting
- Exit code tracking for debugging
- Comprehensive error summaries

## Log Analysis

### View Real-time Logs

```bash
# All backup services
sudo journalctl -f -t plex-backup -t backup-move -t plex-validation -t immich-backup -t plex-report

# Specific service
sudo journalctl -f -t plex-backup

# With our monitor (recommended)
./backup-log-monitor.sh monitor
```

### Historical Analysis

```bash
# Last 24 hours
sudo journalctl --since '24 hours ago' -t plex-backup

# Last week with errors only
sudo journalctl --since '1 week ago' --priority=err -t plex-backup

# Using our tools
./backup-log-monitor.sh recent 24 plex-backup
./backup-log-monitor.sh errors 7
```

## Backup Recovery

### Restore Crontab from Backup

```bash
# List available backups
./crontab-backup-system.sh list

# Restore specific backup
./crontab-backup-system.sh restore crontab-manual-20250526_120000.backup

# Compare before restoring
./crontab-backup-system.sh compare current crontab-manual-20250526_120000.backup
```

### Emergency Recovery

If you need to quickly restore the original crontab:

```bash
# The system automatically creates pre-install backups
./crontab-backup-system.sh list | grep pre-install
./crontab-backup-system.sh restore [backup-filename]
```

## Maintenance

### Regular Tasks

```bash
# Weekly health check
./backup-log-monitor.sh health

# Monthly backup cleanup
./crontab-backup-system.sh cleanup 30

# Quarterly comprehensive report
./backup-log-monitor.sh report 90
```

### Troubleshooting

```bash
# Verify all components
./manage-enhanced-crontab.sh verify

# Check system status
./manage-enhanced-crontab.sh status

# View configuration
./manage-enhanced-crontab.sh show

# Monitor for issues
./backup-log-monitor.sh monitor
```

## Integration Notes

- All scripts follow the established shell repository coding standards
- Logging uses consistent tags and priorities
- Error handling preserves backup integrity
- Color-coded output for better readability
- Comprehensive documentation and help systems

## Security Considerations

- Scripts validate input and handle errors gracefully
- Backup files include metadata for tracking
- Permissions are properly managed
- Sensitive operations require sudo confirmation
- All operations are logged for audit trails

## Next Steps

1. **Install the system**: Run `sudo ./manage-enhanced-crontab.sh install`
2. **Test monitoring**: Use `./backup-log-monitor.sh monitor` during next backup
3. **Review reports**: Generate weekly reports to establish baseline
4. **Set up alerts**: Consider integrating with your notification system
5. **Document customizations**: Add any local modifications to this guide