shell/docs/enhanced-plex-backup.md

# Enhanced Plex Backup Script Documentation

This document provides comprehensive documentation for the enhanced `backup-plex.sh` script. This advanced backup solution includes performance monitoring, parallel processing, intelligent notifications, WAL file handling, and automated testing capabilities.

## Script Overview

The enhanced script performs the following advanced tasks:

1. **Performance Monitoring**: Tracks backup operations with JSON-based performance logging
2. **Intelligent Backup Detection**: Only backs up files that have changed since last backup
3. **WAL File Handling**: Properly handles SQLite Write-Ahead Logging files
4. **Database Integrity Verification**: Comprehensive integrity checks with automated repair options
5. **Parallel Processing**: Concurrent verification for improved performance
6. **Multi-Channel Notifications**: Console, webhook, and email notification support
7. **Checksum Caching**: Intelligent caching to avoid recalculating unchanged file checksums
8. **Enhanced Service Management**: Safe Plex service management with progress indicators
9. **Comprehensive Logging**: Detailed logs with color-coded output and timestamps
10. **Automated Cleanup**: Configurable retention policies for old backups

## Enhanced Features

### Performance Tracking

- **JSON Performance Logs**: All operations are timed and logged to `logs/plex-backup-performance.json`
- **Performance Reports**: Automatic generation of average performance metrics
- **Operation Monitoring**: Tracks backup, verification, service management, and overall script execution times

### Notification System

The script supports multiple notification channels:

#### Console Notifications

- Color-coded status messages (Success: Green, Error: Red, Warning: Yellow, Info: Blue)
- Timestamped log entries with clear formatting

#### Webhook Notifications

```bash
./backup-plex.sh --webhook=https://your-webhook-url.com/endpoint
```

Sends JSON payloads with backup status, hostname, and timestamps.

#### Email Notifications

```bash
./backup-plex.sh --email=admin@example.com
```

Requires `sendmail` to be configured on the system.

### WAL File Management

The script now properly handles SQLite Write-Ahead Logging files:

- **Automatic Detection**: Identifies and backs up `.db-wal` and `.db-shm` files when present
- **WAL Checkpointing**: Performs `PRAGMA wal_checkpoint(FULL)` before integrity checks
- **Safe Backup**: Ensures WAL files are properly backed up alongside main database files

### Database Integrity & Repair

Enhanced database management features:

- **Pre-backup Integrity Checks**: Verifies database health before backup operations
- **Automated Repair**: Optional automatic repair of corrupted databases using advanced techniques
- **Interactive Repair Mode**: Prompts for repair decisions when issues are detected
- **Post-repair Verification**: Re-checks integrity after repair operations

### Parallel Processing

- **Concurrent Verification**: Parallel backup verification for improved performance
- **Fallback Safety**: Automatically falls back to sequential processing if parallel mode fails
- **Configurable**: Can be disabled with `--no-parallel` for maximum safety

## Command Line Options

```bash
Usage: ./backup-plex.sh [OPTIONS]

Options:
  --auto-repair        Automatically attempt to repair corrupted databases
  --check-integrity    Only check database integrity, don't backup
  --non-interactive    Run in non-interactive mode (for automation)
  --no-parallel        Disable parallel verification (slower but safer)
  --no-performance     Disable performance monitoring
  --webhook=URL        Send notifications to webhook URL
  --email=ADDRESS      Send notifications to email address
  -h, --help          Show help message
```

## Configuration Files

### Performance Log Format

The performance log (`logs/plex-backup-performance.json`) contains entries like:

```json
[
  {
    "operation": "backup",
    "duration_seconds": 45.3,
    "timestamp": "2025-05-25T19:45:23-05:00"
  },
  {
    "operation": "verification",
    "duration_seconds": 12.8,
    "timestamp": "2025-05-25T19:46:08-05:00"
  }
]
```

### Backup Tracking Log

The backup tracking log (`logs/plex-backup.json`) tracks last backup times:

```json
{
  "/var/lib/plexmediaserver/.../com.plexapp.plugins.library.db": 1732567523,
  "/var/lib/plexmediaserver/.../Preferences.xml": 1732567523
}
```

## Usage Examples

### Basic Backup

```bash
./backup-plex.sh
```

Performs a standard backup with all enhanced features enabled.

### Integrity Check Only

```bash
./backup-plex.sh --check-integrity
```

Only checks database integrity without performing backup.

### Automated Backup with Notifications

```bash
./backup-plex.sh --non-interactive --auto-repair --webhook=https://notify.example.com/backup
```

Runs in automated mode with auto-repair and webhook notifications.

### Performance-Optimized Backup

```bash
./backup-plex.sh --no-parallel --no-performance
```

Runs with parallel processing and performance monitoring disabled for maximum compatibility.

## Automation and Scheduling

### Cron Job Setup

For daily automated backups at 2 AM:

```bash
# Edit crontab
crontab -e

# Add this line for daily backup
0 2 * * * /home/acedanger/shell/backup-plex.sh --non-interactive --auto-repair --email=admin@example.com 2>&1 | logger -t plex-backup
```

### Systemd Service

Create a systemd service for more control:

```ini
[Unit]
Description=Plex Backup Service
After=network.target

[Service]
Type=oneshot
User=root
ExecStart=/home/acedanger/shell/backup-plex.sh --non-interactive --auto-repair
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
```

### Systemd Timer

Create a timer for regular execution:

```ini
[Unit]
Description=Daily Plex Backup
Requires=plex-backup.service

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
```

## Monitoring and Alerts

### Performance Monitoring

The script automatically tracks:

- Backup operation duration
- Verification times
- Service start/stop times
- Overall script execution time

### Health Checks

Regular health monitoring can be implemented by checking:

```bash
# Check last backup success
jq -r '.[-1] | select(.operation == "total_script") | .timestamp' logs/plex-backup-performance.json

# Check average backup performance
jq '[.[] | select(.operation == "backup") | .duration_seconds] | add/length' logs/plex-backup-performance.json
```

## Troubleshooting

### Common Issues

1. **Permission Denied Errors**
   - Ensure script runs with appropriate sudo permissions
   - Check Plex file ownership and permissions

2. **WAL File Warnings**
   - Now handled automatically by the enhanced script
   - WAL checkpointing ensures data consistency

3. **Performance Issues**
   - Use `--no-parallel` if concurrent operations cause problems
   - Monitor performance logs for bottlenecks

4. **Notification Failures**
   - Verify webhook URLs are accessible
   - Check sendmail configuration for email notifications

### Debug Mode

Enable verbose logging by modifying the script or using:

```bash
bash -x ./backup-plex.sh --check-integrity
```

## Testing Framework

The script includes a comprehensive testing framework (`test-plex-backup.sh`):

### Running Tests

```bash
# Run all tests
./test-plex-backup.sh all

# Run only unit tests
./test-plex-backup.sh unit

# Run performance benchmarks
./test-plex-backup.sh performance
```

### Test Categories

- **Unit Tests**: Core functionality verification
- **Integration Tests**: Full system testing (requires Plex installation)
- **Performance Tests**: Benchmarking and performance validation

## Security Considerations

### File Permissions

- Backup files are created with appropriate permissions
- Sensitive files maintain original ownership and permissions
- Temporary files are properly cleaned up

### Network Security

- Webhook notifications use HTTPS when possible
- Email notifications respect system sendmail configuration
- No sensitive data is included in notifications

### Access Control

- Script requires appropriate sudo permissions
- Backup locations should have restricted access
- Log files contain operational data, not sensitive information

## Backup Strategy

The enhanced script implements a robust backup strategy:

### 3-2-1 Backup Rule

1. **3 Copies**: Original data + local backup + compressed archive
2. **2 Different Media**: Local disk + network storage capability
3. **1 Offsite**: Ready for remote synchronization

### Retention Policy

- Configurable maximum backup age (default: 30 days)
- Configurable maximum backup count (default: 10 backups)
- Automatic cleanup of old backups

### Verification Strategy

- Checksum verification for all backed up files
- Database integrity checks before and after operations
- Optional parallel verification for improved performance

## Migration from Legacy Script

To migrate from the original backup script:

1. **Backup Current Configuration**: Save any custom modifications
2. **Test New Script**: Run with `--check-integrity` first
3. **Update Automation**: Modify cron jobs to use new options
4. **Monitor Performance**: Check performance logs for optimization opportunities

The enhanced script maintains backward compatibility while adding significant new capabilities.