Add comprehensive Plex backup management scripts

- Introduced `restore-plex.sh` for restoring Plex backups with logging and validation. - Created `test-plex-backup.sh` for automated testing of backup functionalities. - Developed `validate-plex-backups.sh` for validating backup integrity and monitoring. - Updated `update.sh` to reference the correct path for Plex service management.
2025-12-06 05:40:11 -08:00 · 2025-05-26 13:20:12 -04:00
parent a91fc79449
commit c9b69ea789
30 changed files with 702 additions and 25 deletions
--- a/plex/plex-backup.md
+++ b/plex/plex-backup.md
@@ -0,0 +1,495 @@
+# Enhanced Plex Backup Script Documentation
+
+This document provides comprehensive documentation for the enhanced `backup-plex.sh` script. This advanced backup solution includes performance monitoring, parallel processing, intelligent notifications, and WAL file handling.
+
+## Script Overview
+
+The enhanced script performs the following advanced tasks:
+
+1. **Performance Monitoring**: Tracks backup operations with JSON-based performance logging
+2. **Full Backup Operations**: Performs complete backups of all Plex files every time
+3. **WAL File Handling**: Properly handles SQLite Write-Ahead Logging files
+4. **Database Integrity Verification**: Comprehensive integrity checks with automated repair options
+5. **Parallel Processing**: Concurrent verification for improved performance
+6. **Multi-Channel Notifications**: Console, webhook, and email notification support
+7. **Enhanced Service Management**: Safe Plex service management with progress indicators
+8. **Comprehensive Logging**: Detailed logs with color-coded output and timestamps
+9. **Safe Automated Cleanup**: Retention policies based on age and backup count
+
+## Enhanced Features
+
+### Full Backup Operation
+
+The script performs complete backups every time it runs:
+
+- **What it does**: Backs up all Plex files regardless of modification status
+- **Benefits**:
+  - Guarantees every backup is a complete restoration point
+  - Eliminates risk of file loss from incomplete backup coverage
+  - Simplifies backup management and restoration
+- **Usage**: `./backup-plex.sh` (no options needed)
+
+### Performance Tracking
+
+- **JSON Performance Logs**: All operations are timed and logged to `logs/plex-backup-performance.json`
+- **Performance Reports**: Automatic generation of average performance metrics
+- **Operation Monitoring**: Tracks backup, verification, service management, and overall script execution times
+
+### Notification System
+
+The script supports multiple notification channels:
+
+#### Console Notifications
+
+- Color-coded status messages (Success: Green, Error: Red, Warning: Yellow, Info: Blue)
+- Timestamped log entries with clear formatting
+
+#### Webhook Notifications
+
+```bash
+./backup-plex.sh --webhook=https://your-webhook-url.com/endpoint
+```
+
+**Default Webhook**: The script includes a default webhook URL (`https://notify.peterwood.rocks/lab`) that will be used if no custom webhook is specified. To use a different webhook, specify it with the `--webhook` option.
+
+Sends JSON payloads with backup status, hostname, and timestamps. Notifications include tags for filtering (backup, plex, hostname, and status-specific tags like "errors" or "warnings").
+
+#### Email Notifications
+
+```bash
+./backup-plex.sh --email=admin@example.com
+```
+
+Requires `sendmail` to be configured on the system.
+
+### WAL File Management
+
+The script now properly handles SQLite Write-Ahead Logging files:
+
+- **Automatic Detection**: Identifies and backs up `.db-wal` and `.db-shm` files when present
+- **WAL Checkpointing**: Performs `PRAGMA wal_checkpoint(FULL)` before integrity checks
+- **Safe Backup**: Ensures WAL files are properly backed up alongside main database files
+
+### Database Integrity & Repair
+
+Enhanced database management features:
+
+- **Pre-backup Integrity Checks**: Verifies database health before backup operations
+- **Automated Repair**: Optional automatic repair of corrupted databases using advanced techniques
+- **Interactive Repair Mode**: Prompts for repair decisions when issues are detected
+- **Post-repair Verification**: Re-checks integrity after repair operations
+
+### Parallel Processing
+
+- **Concurrent Verification**: Parallel backup verification for improved performance
+- **Fallback Safety**: Automatically falls back to sequential processing if parallel mode fails
+- **Configurable**: Can be disabled with `--no-parallel` for maximum safety
+
+## Command Line Options
+
+```bash
+Usage: ./backup-plex.sh [OPTIONS]
+
+Options:
+  --auto-repair        Automatically attempt to repair corrupted databases
+  --check-integrity    Only check database integrity, don't backup
+  --non-interactive    Run in non-interactive mode (for automation)
+  --no-parallel        Disable parallel verification (slower but safer)
+  --no-performance     Disable performance monitoring
+  --webhook=URL        Send notifications to webhook URL
+  --email=ADDRESS      Send notifications to email address
+  -h, --help          Show help message
+```
+
+## Detailed Backup Process Steps
+
+The backup script follows these detailed steps to ensure data integrity and reliability:
+
+### 1. Create Log Directory
+
+```bash
+mkdir -p /mnt/share/media/backups/logs || { echo "Failed to create log directory"; exit 1; }
+```
+
+This command ensures that the log directory exists. If it doesn't, it creates the directory. If the directory creation fails, the script exits with an error message.
+
+### 2. Define Log File
+
+```bash
+LOG_FILE="/mnt/share/media/backups/logs/backup_log_$(date +%Y%m%d_%H%M%S).md"
+```
+
+This line defines the log file path, including the current date and time in the filename to ensure uniqueness.
+
+### 3. Stop Plex Media Server Service
+
+```bash
+if systemctl is-active --quiet plexmediaserver.service; then
+  /home/acedanger/shell/plex/plex.sh stop || { echo "Failed to stop plexmediaserver.service"; exit 1; }
+fi
+```
+
+This block checks if the Plex Media Server service is running. If it is, the script stops the service using a custom script (`plex.sh`).
+
+### 4. Backup Plex Database Files and Preferences
+
+The enhanced backup system creates compressed archives directly, eliminating intermediate directories:
+
+```bash
+# Files are copied to temporary staging area for verification
+cp "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db" "$BACKUP_PATH/"
+cp "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.blobs.db" "$BACKUP_PATH/"
+cp "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Preferences.xml" "$BACKUP_PATH/"
+```
+
+These commands copy the Plex database files and preferences directly to the backup root directory. Each file copy operation includes integrity verification and checksum validation.
+
+### 5. Create Compressed Archive
+
+```bash
+# Create archive directly with timestamp naming convention
+final_archive="${BACKUP_ROOT}/plex-backup-$(date '+%Y%m%d_%H%M%S').tar.gz"
+tar -czf "$final_archive" -C "$temp_staging_dir" .
+```
+
+The system creates compressed archives directly using a timestamp-based naming convention (`plex-backup-YYYYMMDD_HHMMSS.tar.gz`), eliminating the need for intermediate dated directories.
+
+### 6. Archive Validation and Cleanup
+
+```bash
+# Validate archive integrity
+if tar -tzf "$final_archive" >/dev/null 2>&1; then
+    log_success "Archive created and validated: $(basename "$final_archive")"
+    rm -rf "$temp_staging_dir"
+else
+    log_error "Archive validation failed"
+    rm -f "$final_archive"
+fi
+```
+
+The system validates the created archive and removes temporary staging files, ensuring only valid compressed backups are retained in the backup root directory.
+
+### 7. Send Notification
+
+```bash
+curl \
+    -H tags:popcorn,backup,plex,${HOSTNAME} \
+    -d "The Plex databases have been saved to the /media/backups/plex folder as plex-backup-YYYYMMDD_HHMMSS.tar.gz" \
+    https://notify.peterwood.rocks/lab || { echo "Failed to send notification"; exit 1; }
+```
+
+This command sends a notification upon completion of the backup process, indicating the compressed archive has been created.
+
+### 8. Restart Plex Media Server Service
+
+```bash
+if systemctl is-enabled --quiet plexmediaserver.service; then
+  /home/acedanger/shell/plex/plex.sh start || { echo "Failed to start plexmediaserver.service"; exit 1; }
+fi
+```
+
+This block checks if the Plex Media Server service is enabled. If it is, the script restarts the service using a custom script (`plex.sh`).
+
+### 9. Legacy Cleanup
+
+```bash
+# Clean up any remaining dated directories from old backup structure
+find "${BACKUP_ROOT}" -maxdepth 1 -type d -name "????????" -exec rm -rf {} \; 2>/dev/null || true
+```
+
+The enhanced system includes cleanup of legacy dated directories from previous backup structure versions, ensuring a clean tar.gz-only backup directory.
+
+## Configuration Files
+
+### Performance Log Format
+
+The performance log (`logs/plex-backup-performance.json`) contains entries like:
+
+```json
+[
+  {
+    "operation": "backup",
+    "duration_seconds": 45.3,
+    "timestamp": "2025-05-25T19:45:23-05:00"
+  },
+  {
+    "operation": "verification",
+    "duration_seconds": 12.8,
+    "timestamp": "2025-05-25T19:46:08-05:00"
+  }
+]
+```
+
+## Usage Examples
+
+### Basic Backup
+
+```bash
+./backup-plex.sh
+```
+
+Performs a standard backup with all enhanced features enabled.
+
+### Integrity Check Only
+
+```bash
+./backup-plex.sh --check-integrity
+```
+
+Only checks database integrity without performing backup.
+
+### Automated Backup with Notifications
+
+```bash
+./backup-plex.sh --non-interactive --auto-repair --webhook=https://notify.example.com/backup
+```
+
+Runs in automated mode with auto-repair and custom webhook notifications.
+
+**Note**: If no `--webhook` option is specified, the script will use the default webhook URL (`https://notify.peterwood.rocks/lab`) for notifications.
+
+### Performance-Optimized Backup
+
+```bash
+./backup-plex.sh --no-parallel --no-performance
+```
+
+Runs with parallel processing and performance monitoring disabled for maximum compatibility.
+
+## Automation and Scheduling
+
+### Cron Job Setup
+
+For daily automated backups at 2 AM:
+
+```bash
+# Edit crontab
+crontab -e
+
+# Add this line for daily backup with email notifications
+0 2 * * * /home/acedanger/shell/backup-plex.sh --non-interactive --auto-repair --email=admin@example.com 2>&1 | logger -t plex-backup
+
+# Or for daily backup with default webhook notifications (https://notify.peterwood.rocks/lab)
+0 2 * * * /home/acedanger/shell/backup-plex.sh --non-interactive --auto-repair 2>&1 | logger -t plex-backup
+```
+
+**Note**: The script will automatically use the default webhook URL for notifications unless a custom webhook is specified with `--webhook=URL`.
+
+### Systemd Service
+
+Create a systemd service for more control:
+
+```ini
+[Unit]
+Description=Plex Backup Service
+After=network.target
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/home/acedanger/shell/backup-plex.sh --non-interactive --auto-repair
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target
+```
+
+### Systemd Timer
+
+Create a timer for regular execution:
+
+```ini
+[Unit]
+Description=Daily Plex Backup
+Requires=plex-backup.service
+
+[Timer]
+OnCalendar=daily
+Persistent=true
+
+[Install]
+WantedBy=timers.target
+```
+
+## Monitoring and Alerts
+
+### Performance Monitoring
+
+The script automatically tracks:
+
+- Backup operation duration
+- Verification times
+- Service start/stop times
+- Overall script execution time
+
+### Health Checks
+
+Regular health monitoring can be implemented by checking:
+
+```bash
+# Check last backup success
+jq -r '.[-1] | select(.operation == "total_script") | .timestamp' logs/plex-backup-performance.json
+
+# Check average backup performance
+jq '[.[] | select(.operation == "backup") | .duration_seconds] | add/length' logs/plex-backup-performance.json
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Permission Denied Errors**
+   - Ensure script runs with appropriate sudo permissions
+   - Check Plex file ownership and permissions
+
+2. **WAL File Warnings**
+   - Now handled automatically by the enhanced script
+   - WAL checkpointing ensures data consistency
+
+3. **Performance Issues**
+   - Use `--no-parallel` if concurrent operations cause problems
+   - Monitor performance logs for bottlenecks
+
+4. **Notification Failures**
+   - Verify webhook URLs are accessible
+   - Check sendmail configuration for email notifications
+
+### Debug Mode
+
+Enable verbose logging by modifying the script or using:
+
+```bash
+bash -x ./backup-plex.sh --check-integrity
+```
+
+## Testing Framework
+
+The script includes a comprehensive testing framework (`test-plex-backup.sh`):
+
+### Running Tests
+
+```bash
+# Run all tests
+./test-plex-backup.sh all
+
+# Run only unit tests
+./test-plex-backup.sh unit
+
+# Run performance benchmarks
+./test-plex-backup.sh performance
+```
+
+### Test Categories
+
+- **Unit Tests**: Core functionality verification
+- **Integration Tests**: Full system testing (requires Plex installation)
+- **Performance Tests**: Benchmarking and performance validation
+
+## Security Considerations
+
+### File Permissions
+
+- Backup files are created with appropriate permissions
+- Sensitive files maintain original ownership and permissions
+- Temporary files are properly cleaned up
+
+### Network Security
+
+- Webhook notifications use HTTPS when possible
+- Email notifications respect system sendmail configuration
+- No sensitive data is included in notifications
+
+### Access Control
+
+- Script requires appropriate sudo permissions
+- Backup locations should have restricted access
+- Log files contain operational data, not sensitive information
+
+## Backup Strategy
+
+The enhanced script implements a robust backup strategy with a streamlined tar.gz-only structure:
+
+### Archive-Only Directory Structure
+
+The new backup system eliminates intermediate dated directories and stores only compressed archives:
+
+```text
+/mnt/share/media/backups/plex/
+├── plex-backup-20250125_143022.tar.gz    # Latest backup
+├── plex-backup-20250124_143011.tar.gz    # Previous backup
+├── plex-backup-20250123_143008.tar.gz    # Older backup
+└── logs/
+    ├── backup_log_20250125_143022.md
+    └── plex-backup-performance.json
+```
+
+### Archive Naming Convention
+
+Backup files follow the naming convention `plex-backup-YYYYMMDD_HHMMSS.tar.gz` for easy identification and sorting.
+
+## Important Information
+
+- Ensure that the [`plex.sh`](https://github.com/acedanger/shell/blob/main/plex.sh) script is available and executable. This script is used to stop and start the Plex Media Server service.
+- The script uses `systemctl` to manage the Plex Media Server service. Ensure that `systemctl` is available on your system.
+- **New Directory Structure**: The enhanced backup system stores only compressed `.tar.gz` files directly in the backup root directory, eliminating intermediate dated directories.
+- **Archive Naming**: Backup files follow the naming convention `plex-backup-YYYYMMDD_HHMMSS.tar.gz` for easy identification and sorting.
+- **Legacy Compatibility**: The system automatically cleans up old dated directories from previous backup versions during operation.
+- The backup directory path is configurable through the `BACKUP_ROOT` variable. Modify this path as needed to fit your environment.
+- The script logs important actions and errors to timestamped log files. Check the log files for details if any issues arise.
+- **Backup Validation**: All archives undergo integrity checking to ensure backup reliability.
+
+## Final Directory Structure
+
+```text
+/mnt/share/media/backups/plex/
+├── plex-backup-20250125_143022.tar.gz    # Latest backup
+├── plex-backup-20250124_143011.tar.gz    # Previous backup  
+├── plex-backup-20250123_143008.tar.gz    # Older backup
+└── logs/
+    ├── backup_log_20250125_143022.md
+    └── plex-backup-performance.json
+```
+
+Backup files follow the pattern: `plex-backup-YYYYMMDD_HHMMSS.tar.gz`
+
+- **YYYYMMDD**: Date of backup (e.g., 20250125)
+- **HHMMSS**: Time of backup (e.g., 143022)
+- **tar.gz**: Compressed archive format
+
+### Key Improvements
+
+1. **Direct Archive Creation**: No intermediate directories required
+2. **Efficient Storage**: Only compressed files stored permanently
+3. **Easy Identification**: Timestamp-based naming for sorting
+4. **Legacy Cleanup**: Automatic removal of old dated directories
+5. **Archive Validation**: Integrity checking of compressed files
+
+### 3-2-1 Backup Rule
+
+1. **3 Copies**: Original data + local backup + compressed archive
+2. **2 Different Media**: Local disk + network storage capability
+3. **1 Offsite**: Ready for remote synchronization
+
+### Retention Policy
+
+- Configurable maximum backup age (default: 30 days)
+- Configurable maximum backup count (default: 10 backups)
+- Automatic cleanup of old backups
+
+### Verification Strategy
+
+- Checksum verification for all backed up files
+- Database integrity checks before and after operations
+- Optional parallel verification for improved performance
+
+## Migration from Legacy Script
+
+To migrate from the original backup script:
+
+1. **Backup Current Configuration**: Save any custom modifications
+2. **Test New Script**: Run with `--check-integrity` first
+3. **Update Automation**: Modify cron jobs to use new options
+4. **Monitor Performance**: Check performance logs for optimization opportunities
+
+The enhanced script maintains backward compatibility while adding significant new capabilities.