shell/docs/production-deployment-guide.md

Plex Backup System - Production Deployment Guide

This guide helps you deploy the enhanced Plex backup system safely in a production environment.

Pre-Deployment Checklist

1. System Requirements Verification

• Operating System: Linux (tested on Ubuntu/Debian)
• Shell: Bash 4.0+ available
• Dependencies Installed:
  • jq - JSON processing (required for performance logging)
  • sqlite3 - Database tools (for fallback integrity checks)
  • curl - HTTP client (for webhook notifications)
  • sendmail - Email delivery (if using email notifications)
  • tar and gzip - Archive tools
  • sudo access to Plex files and service management

2. Environment Setup

• Backup Directory: Ensure /mnt/share/media/backups/plex exists and has sufficient space
• Log Directory: Ensure /mnt/share/media/backups/logs exists and is writable
• Script Directory: Place scripts in /home/acedanger/shell or update paths accordingly
• Permissions: Verify script user can read Plex files and control Plex service

3. Configuration Verification

• Plex Service Management: Test systemctl stop plexmediaserver and systemctl start plexmediaserver
• File Paths: Verify Plex database locations in script match your installation
• Plex SQLite Binary: Confirm /usr/lib/plexmediaserver/Plex SQLite exists
• Disk Space: Ensure backup location has 2x current Plex database size available

Testing Phase

1. Run Unit Tests

cd /home/acedanger/shell
./test-plex-backup.sh unit

Expected Result: All 9 tests should pass (100% success rate)

2. Run Integration Tests

cd /home/acedanger/shell
./integration-test-plex.sh

Expected Result: All integration tests should pass

3. Test Dry Run

# Test integrity check only (non-destructive)
sudo ./backup-plex.sh --check-integrity --non-interactive

Expected Result: Should complete without stopping Plex or creating backups

4. Test Notification Systems

Webhook Testing

# Replace with your actual webhook URL
sudo ./backup-plex.sh --check-integrity --webhook=https://your-webhook-endpoint.com/test

Email Testing

# Replace with your email address
sudo ./backup-plex.sh --check-integrity --email=admin@yourdomain.com

Production Deployment Steps

1. Initial Backup Test

# Create a manual backup during maintenance window
sudo ./backup-plex.sh --non-interactive

Verify:

• Plex service stopped and restarted properly
• Backup files created in /mnt/share/media/backups/plex/YYYYMMDD/
• Log files updated in /mnt/share/media/backups/logs/
• Performance log created if enabled
• Notifications sent if configured

2. Validate Backup Integrity

# Run validation on the created backup
./validate-plex-backups.sh --report

3. Test Restore Process (Optional)

# In a test environment, verify restore functionality
./restore-plex.sh --list
./restore-plex.sh --validate YYYYMMDD

Automated Scheduling

1. Cron Configuration

Create a cron job for automated backups:

# Edit crontab for root user
sudo crontab -e

# Add entry for daily backup at 2 AM
0 2 * * * /home/acedanger/shell/backup-plex.sh --non-interactive --webhook=https://your-webhook.com/plex-backup 2>&1 | logger -t plex-backup

2. Systemd Timer (Alternative)

Create systemd service and timer files:

# Create service file
sudo tee /etc/systemd/system/plex-backup.service > /dev/null << 'EOF'
[Unit]
Description=Plex Media Server Backup
After=network.target

[Service]
Type=oneshot
User=root
ExecStart=/home/acedanger/shell/backup-plex.sh --non-interactive
StandardOutput=journal
StandardError=journal
EOF

# Create timer file
sudo tee /etc/systemd/system/plex-backup.timer > /dev/null << 'EOF'
[Unit]
Description=Run Plex backup daily
Requires=plex-backup.service

[Timer]
OnCalendar=daily
Persistent=true
RandomizedDelaySec=30m

[Install]
WantedBy=timers.target
EOF

# Enable and start timer
sudo systemctl daemon-reload
sudo systemctl enable plex-backup.timer
sudo systemctl start plex-backup.timer

Monitoring and Maintenance

1. Log Monitoring

Monitor backup logs for issues:

# Check recent backup logs
tail -f /mnt/share/media/backups/logs/plex-backup-$(date +%Y-%m-%d).log

# Check system logs for backup service
sudo journalctl -u plex-backup.service -f

2. Performance Log Analysis

# View performance trends
jq '.[] | select(.operation == "full_backup") | {timestamp, duration_seconds}' \
  /home/acedanger/shell/logs/plex-backup-performance.json | tail -10

3. Regular Validation

Schedule weekly backup validation:

# Add to crontab
0 3 * * 0 /home/acedanger/shell/validate-plex-backups.sh --report --fix 2>&1 | logger -t plex-backup-validation

Troubleshooting Guide

Common Issues

1. Permission Denied Errors

# Fix script permissions
chmod +x /home/acedanger/shell/*.sh

# Fix backup directory permissions
sudo chown -R $(whoami):$(whoami) /mnt/share/media/backups/

2. Plex Service Issues

# Check Plex service status
sudo systemctl status plexmediaserver

# Manually restart if needed
sudo systemctl restart plexmediaserver

3. Insufficient Disk Space

# Check available space
df -h /mnt/share/media/backups/

# Clean old backups manually if needed
./backup-plex.sh # Script will auto-cleanup based on retention policy

4. Database Integrity Issues

# Run integrity check only
sudo ./backup-plex.sh --check-integrity --auto-repair

# Manual database repair if needed
sudo ./backup-plex.sh --auto-repair

Performance Optimization

1. Parallel Processing

• Enable parallel verification for faster backups (default: enabled)
• Disable with --no-parallel if experiencing issues

2. Performance Monitoring

• Disable with --no-performance if not needed
• Monitor trends to optimize backup timing

3. Notification Optimization

• Use webhooks instead of email for faster notifications
• Configure webhook endpoints with proper error handling

Security Considerations

1. File Permissions

# Secure backup files
chmod 600 /home/acedanger/shell/logs/plex-backup*.json
chmod 700 /mnt/share/media/backups/plex/

2. Webhook Security

• Use HTTPS endpoints for webhooks
• Implement webhook signature verification if possible
• Avoid including sensitive data in webhook payloads

3. Access Control

• Limit script execution to authorized users
• Consider using dedicated backup user account
• Regularly audit file access permissions

Backup Retention Strategy

The script automatically manages backup retention:

• Default: Keep 10 most recent backups
• Age-based: Remove backups older than 30 days
• Configurable: Modify MAX_BACKUPS_TO_KEEP and MAX_BACKUP_AGE_DAYS in script

Recovery Planning

1. Backup Restoration

# List available backups
./restore-plex.sh --list

# Restore specific backup
sudo ./restore-plex.sh --restore YYYYMMDD

2. Emergency Procedures

1. Stop Plex service: sudo systemctl stop plexmediaserver
2. Backup current data: ./restore-plex.sh --backup-current
3. Restore from backup: sudo ./restore-plex.sh --restore YYYYMMDD
4. Start Plex service: sudo systemctl start plexmediaserver

Success Metrics

Monitor these metrics to ensure backup system health:

• Backup Success Rate: >99% successful backups
• Backup Duration: Consistent timing (tracked in performance logs)
• Storage Usage: Within acceptable limits
• Service Downtime: Minimal Plex service interruption
• Notification Delivery: Reliable alert delivery
• Validation Results: Regular successful backup validation

Support and Updates

Getting Help

1. Check logs for error messages
2. Run validation tools for diagnosis
3. Review troubleshooting guide
4. Test with --check-integrity for safe debugging

Script Updates

• Keep scripts updated with latest features
• Test updates in non-production environment first
• Backup current scripts before updating
• Review changelog for breaking changes

---

Note: This deployment guide assumes a typical Plex Media Server installation. Adjust paths and configurations based on your specific environment.