shell/plex/plex-backup.md

# Enhanced Plex Backup Script Documentation

**Author:** Peter Wood <peter@peterwood.dev>

This document provides comprehensive documentation for the enhanced `backup-plex.sh` script. This advanced backup solution includes database integrity checking with automatic repair, performance monitoring, parallel processing, intelligent notifications, and WAL file handling.

## Enhanced Service Management Integration

The backup script integrates seamlessly with the enhanced `plex.sh` service management script:

- **Porcelain Mode**: Uses `--porcelain` for clean, automation-friendly output
- **Color Detection**: Automatically adapts to terminal capabilities when run interactively
- **Error Handling**: Proper integration with the enhanced error handling in `plex.sh`
- **Non-Interactive Operation**: Designed for automated backup scenarios
- **Service Presence Detection**: Gracefully handles systems without Plex installed

Example service management calls:

```bash
# Stop Plex with porcelain output for automation
/home/acedanger/shell/plex/plex.sh --porcelain stop

# Start Plex after backup completion  
/home/acedanger/shell/plex/plex.sh --porcelain start

# View logs for troubleshooting backup issues
/home/acedanger/shell/plex/plex.sh logs --lines 50
```

## Script Overview

**Author:** Peter Wood <peter@peterwood.dev>

The enhanced script performs the following advanced tasks:

1. **Database Integrity Checking**: Automatic detection and repair of database corruption
2. **Performance Monitoring**: Tracks backup operations with JSON-based performance logging
3. **Full Backup Operations**: Performs complete backups of all Plex files every time
4. **WAL File Handling**: Properly handles SQLite Write-Ahead Logging files
5. **Database Recovery**: Multiple repair strategies from gentle to aggressive
6. **Parallel Processing**: Concurrent verification for improved performance
7. **Multi-Channel Notifications**: Console, webhook, and email notification support
8. **Enhanced Service Management**: Safe Plex service management with progress indicators
9. **Comprehensive Logging**: Detailed logs with color-coded output and timestamps
10. **Safe Automated Cleanup**: Retention policies based on age and backup count

## Enhanced Features

### Database Integrity & Auto-Repair (NEW)

The script now includes comprehensive database integrity checking and automatic repair:

- **What it does**: Checks database integrity before backup and automatically repairs corruption
- **Benefits**:
  - Prevents backing up corrupted databases
  - Automatically fixes common database issues
  - Provides multiple repair strategies
  - Comprehensive logging of all repair attempts
- **Usage**:
  - `./backup-plex.sh` (auto-repair enabled by default)
  - `./backup-plex.sh --disable-auto-repair` (skip auto-repair)
  - `./backup-plex.sh --check-integrity` (integrity check only)

### Full Backup Operation

The script performs complete backups every time it runs:

- **What it does**: Backs up all Plex files regardless of modification status
- **Benefits**:
  - Guarantees every backup is a complete restoration point
  - Eliminates risk of file loss from incomplete backup coverage
  - Simplifies backup management and restoration
- **Usage**: `./backup-plex.sh` (no options needed)

### Performance Tracking

- **JSON Performance Logs**: All operations are timed and logged to `logs/plex-backup-performance.json`
- **Performance Reports**: Automatic generation of average performance metrics
- **Operation Monitoring**: Tracks backup, verification, service management, and overall script execution times

### Notification System

The script supports multiple notification channels:

#### Console Notifications

- Color-coded status messages (Success: Green, Error: Red, Warning: Yellow, Info: Blue)
- Timestamped log entries with clear formatting

#### Webhook Notifications

```bash
./backup-plex.sh --webhook=https://your-webhook-url.com/endpoint
```

**Default Webhook**: The script includes a default webhook URL (`https://notify.peterwood.rocks/lab`) that will be used if no custom webhook is specified. To use a different webhook, specify it with the `--webhook` option.

Sends JSON payloads with backup status, hostname, and timestamps. Notifications include tags for filtering (backup, plex, hostname, and status-specific tags like "errors" or "warnings").

#### Email Notifications

```bash
./backup-plex.sh --email=admin@example.com
```

Requires `sendmail` to be configured on the system.

### WAL File Management

The script now properly handles SQLite Write-Ahead Logging files:

- **Automatic Detection**: Identifies and backs up `.db-wal` and `.db-shm` files when present
- **WAL Checkpointing**: Performs `PRAGMA wal_checkpoint(FULL)` before integrity checks
- **Safe Backup**: Ensures WAL files are properly backed up alongside main database files

### Database Integrity & Repair

Enhanced database management features:

- **Pre-backup Integrity Checks**: Verifies database health before backup operations
- **Automated Repair**: Optional automatic repair of corrupted databases using advanced techniques
- **Interactive Repair Mode**: Prompts for repair decisions when issues are detected
- **Post-repair Verification**: Re-checks integrity after repair operations

### Parallel Processing

- **Concurrent Verification**: Parallel backup verification for improved performance
- **Fallback Safety**: Automatically falls back to sequential processing if parallel mode fails
- **Configurable**: Can be disabled with `--no-parallel` for maximum safety

## Command Line Options

```bash
Usage: ./backup-plex.sh [OPTIONS]

Options:
  --auto-repair        Automatically attempt to repair corrupted databases
  --check-integrity    Only check database integrity, don't backup
  --non-interactive    Run in non-interactive mode (for automation)
  --no-parallel        Disable parallel verification (slower but safer)
  --no-performance     Disable performance monitoring
  --webhook=URL        Send notifications to webhook URL
  --email=ADDRESS      Send notifications to email address
  -h, --help          Show help message
```

## Detailed Backup Process Steps

The backup script follows these detailed steps to ensure data integrity and reliability:

### 1. Create Log Directory

```bash
mkdir -p /mnt/share/media/backups/logs || { echo "Failed to create log directory"; exit 1; }
```

This command ensures that the log directory exists. If it doesn't, it creates the directory. If the directory creation fails, the script exits with an error message.

### 2. Define Log File

```bash
LOG_FILE="/mnt/share/media/backups/logs/backup_log_$(date +%Y%m%d_%H%M%S).md"
```

This line defines the log file path, including the current date and time in the filename to ensure uniqueness.

### 3. Stop Plex Media Server Service

```bash
if systemctl is-active --quiet plexmediaserver.service; then
  /home/acedanger/shell/plex/plex.sh --porcelain stop || { echo "Failed to stop plexmediaserver.service"; exit 1; }
fi
```

This block checks if the Plex Media Server service is running. If it is, the script stops the service using the custom `plex.sh` script with porcelain mode enabled for automation. The `--porcelain` option ensures clean, non-interactive output suitable for backup scripts.

### 4. Backup Plex Database Files and Preferences

The enhanced backup system creates compressed archives directly, eliminating intermediate directories:

```bash
# Files are copied to temporary staging area for verification
cp "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db" "$BACKUP_PATH/"
cp "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.blobs.db" "$BACKUP_PATH/"
cp "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Preferences.xml" "$BACKUP_PATH/"
```

These commands copy the Plex database files and preferences directly to the backup root directory. Each file copy operation includes integrity verification and checksum validation.

### 5. Create Compressed Archive

```bash
# Create archive directly with timestamp naming convention
final_archive="${BACKUP_ROOT}/plex-backup-$(date '+%Y%m%d_%H%M%S').tar.gz"
tar -czf "$final_archive" -C "$temp_staging_dir" .
```

The system creates compressed archives directly using a timestamp-based naming convention (`plex-backup-YYYYMMDD_HHMMSS.tar.gz`), eliminating the need for intermediate dated directories.

### 6. Archive Validation and Cleanup

```bash
# Validate archive integrity
if tar -tzf "$final_archive" >/dev/null 2>&1; then
    log_success "Archive created and validated: $(basename "$final_archive")"
    rm -rf "$temp_staging_dir"
else
    log_error "Archive validation failed"
    rm -f "$final_archive"
fi
```

The system validates the created archive and removes temporary staging files, ensuring only valid compressed backups are retained in the backup root directory.

### 7. Send Notification

```bash
curl \
    -H tags:popcorn,backup,plex,${HOSTNAME} \
    -d "The Plex databases have been saved to the /media/backups/plex folder as plex-backup-YYYYMMDD_HHMMSS.tar.gz" \
    https://notify.peterwood.rocks/lab || { echo "Failed to send notification"; exit 1; }
```

This command sends a notification upon completion of the backup process, indicating the compressed archive has been created.

### 8. Restart Plex Media Server Service

```bash
if systemctl is-enabled --quiet plexmediaserver.service; then
  /home/acedanger/shell/plex/plex.sh --porcelain start || { echo "Failed to start plexmediaserver.service"; exit 1; }
fi
```

This block checks if the Plex Media Server service is enabled. If it is, the script restarts the service using the custom `plex.sh` script with porcelain mode for automation. The `--porcelain` option ensures clean output without ANSI escape sequences when called from the backup script.

### 9. Legacy Cleanup

```bash
# Clean up any remaining dated directories from old backup structure
find "${BACKUP_ROOT}" -maxdepth 1 -type d -name "????????" -exec rm -rf {} \; 2>/dev/null || true
```

The enhanced system includes cleanup of legacy dated directories from previous backup structure versions, ensuring a clean tar.gz-only backup directory.

## Configuration Files

### Performance Log Format

The performance log (`logs/plex-backup-performance.json`) contains entries like:

```json
[
  {
    "operation": "backup",
    "duration_seconds": 45.3,
    "timestamp": "2025-05-25T19:45:23-05:00"
  },
  {
    "operation": "verification",
    "duration_seconds": 12.8,
    "timestamp": "2025-05-25T19:46:08-05:00"
  }
]
```

## Usage Examples

### Basic Backup

```bash
./backup-plex.sh
```

Performs a standard backup with all enhanced features enabled.

### Integrity Check Only

```bash
./backup-plex.sh --check-integrity
```

Only checks database integrity without performing backup.

### Automated Backup with Notifications

```bash
./backup-plex.sh --non-interactive --auto-repair --webhook=https://notify.example.com/backup
```

Runs in automated mode with auto-repair and custom webhook notifications.

**Note**: If no `--webhook` option is specified, the script will use the default webhook URL (`https://notify.peterwood.rocks/lab`) for notifications.

### Performance-Optimized Backup

```bash
./backup-plex.sh --no-parallel --no-performance
```

Runs with parallel processing and performance monitoring disabled for maximum compatibility.

## Automation and Scheduling

### Cron Job Setup

For daily automated backups at 2 AM:

```bash
# Edit crontab
crontab -e

# Add this line for daily backup with email notifications
0 2 * * * /home/acedanger/shell/backup-plex.sh --non-interactive --auto-repair --email=admin@example.com 2>&1 | logger -t plex-backup

# Or for daily backup with default webhook notifications (https://notify.peterwood.rocks/lab)
0 2 * * * /home/acedanger/shell/backup-plex.sh --non-interactive --auto-repair 2>&1 | logger -t plex-backup
```

**Note**: The script will automatically use the default webhook URL for notifications unless a custom webhook is specified with `--webhook=URL`.

### Systemd Service

Create a systemd service for more control:

```ini
[Unit]
Description=Plex Backup Service
After=network.target

[Service]
Type=oneshot
User=root
ExecStart=/home/acedanger/shell/backup-plex.sh --non-interactive --auto-repair
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
```

### Systemd Timer

Create a timer for regular execution:

```ini
[Unit]
Description=Daily Plex Backup
Requires=plex-backup.service

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
```

## Monitoring and Alerts

### Performance Monitoring

The script automatically tracks:

- Backup operation duration
- Verification times
- Service start/stop times
- Overall script execution time

### Health Checks

Regular health monitoring can be implemented by checking:

```bash
# Check last backup success
jq -r '.[-1] | select(.operation == "total_script") | .timestamp' logs/plex-backup-performance.json

# Check average backup performance
jq '[.[] | select(.operation == "backup") | .duration_seconds] | add/length' logs/plex-backup-performance.json
```

## Troubleshooting

### Common Issues

1. **Permission Denied Errors**
   - Ensure script runs with appropriate sudo permissions
   - Check Plex file ownership and permissions

2. **WAL File Warnings**
   - Now handled automatically by the enhanced script
   - WAL checkpointing ensures data consistency

3. **Performance Issues**
   - Use `--no-parallel` if concurrent operations cause problems
   - Monitor performance logs for bottlenecks

4. **Notification Failures**
   - Verify webhook URLs are accessible
   - Check sendmail configuration for email notifications

### Debug Mode

Enable verbose logging by modifying the script or using:

```bash
bash -x ./backup-plex.sh --check-integrity
```

## Testing Framework

The script includes a comprehensive testing framework (`test-plex-backup.sh`):

### Running Tests

```bash
# Run all tests
./test-plex-backup.sh all

# Run only unit tests
./test-plex-backup.sh unit

# Run performance benchmarks
./test-plex-backup.sh performance
```

### Test Categories

- **Unit Tests**: Core functionality verification
- **Integration Tests**: Full system testing (requires Plex installation)
- **Performance Tests**: Benchmarking and performance validation

## Security Considerations

### File Permissions

- Backup files are created with appropriate permissions
- Sensitive files maintain original ownership and permissions
- Temporary files are properly cleaned up

### Network Security

- Webhook notifications use HTTPS when possible
- Email notifications respect system sendmail configuration
- No sensitive data is included in notifications

### Access Control

- Script requires appropriate sudo permissions
- Backup locations should have restricted access
- Log files contain operational data, not sensitive information

## Backup Strategy

The enhanced script implements a robust backup strategy with a streamlined tar.gz-only structure:

### Archive-Only Directory Structure

The new backup system eliminates intermediate dated directories and stores only compressed archives:

```text
/mnt/share/media/backups/plex/
├── plex-backup-20250125_143022.tar.gz    # Latest backup
├── plex-backup-20250124_143011.tar.gz    # Previous backup
├── plex-backup-20250123_143008.tar.gz    # Older backup
└── logs/
    ├── backup_log_20250125_143022.md
    └── plex-backup-performance.json
```

### Archive Naming Convention

Backup files follow the naming convention `plex-backup-YYYYMMDD_HHMMSS.tar.gz` for easy identification and sorting.

## Important Information

- Ensure that the [`plex.sh`](https://github.com/acedanger/shell/blob/main/plex.sh) script is available and executable. This script is used to stop and start the Plex Media Server service.
- The script uses `systemctl` to manage the Plex Media Server service. Ensure that `systemctl` is available on your system.
- **New Directory Structure**: The enhanced backup system stores only compressed `.tar.gz` files directly in the backup root directory, eliminating intermediate dated directories.
- **Archive Naming**: Backup files follow the naming convention `plex-backup-YYYYMMDD_HHMMSS.tar.gz` for easy identification and sorting.
- **Legacy Compatibility**: The system automatically cleans up old dated directories from previous backup versions during operation.
- The backup directory path is configurable through the `BACKUP_ROOT` variable. Modify this path as needed to fit your environment.
- The script logs important actions and errors to timestamped log files. Check the log files for details if any issues arise.
- **Backup Validation**: All archives undergo integrity checking to ensure backup reliability.

## Final Directory Structure

```text
/mnt/share/media/backups/plex/
├── plex-backup-20250125_143022.tar.gz    # Latest backup
├── plex-backup-20250124_143011.tar.gz    # Previous backup
├── plex-backup-20250123_143008.tar.gz    # Older backup
└── logs/
    ├── backup_log_20250125_143022.md
    └── plex-backup-performance.json
```

Backup files follow the pattern: `plex-backup-YYYYMMDD_HHMMSS.tar.gz`

- **YYYYMMDD**: Date of backup (e.g., 20250125)
- **HHMMSS**: Time of backup (e.g., 143022)
- **tar.gz**: Compressed archive format

### Key Improvements

1. **Direct Archive Creation**: No intermediate directories required
2. **Efficient Storage**: Only compressed files stored permanently
3. **Easy Identification**: Timestamp-based naming for sorting
4. **Legacy Cleanup**: Automatic removal of old dated directories
5. **Archive Validation**: Integrity checking of compressed files

### 3-2-1 Backup Rule

1. **3 Copies**: Original data + local backup + compressed archive
2. **2 Different Media**: Local disk + network storage capability
3. **1 Offsite**: Ready for remote synchronization

### Retention Policy

- Configurable maximum backup age (default: 30 days)
- Configurable maximum backup count (default: 10 backups)
- Automatic cleanup of old backups

### Verification Strategy

- Checksum verification for all backed up files
- Database integrity checks before and after operations
- Optional parallel verification for improved performance

## Migration from Legacy Script

To migrate from the original backup script:

1. **Backup Current Configuration**: Save any custom modifications
2. **Test New Script**: Run with `--check-integrity` first
3. **Update Automation**: Modify cron jobs to use new options
4. **Monitor Performance**: Check performance logs for optimization opportunities

The enhanced script maintains backward compatibility while adding significant new capabilities.