shell/docs/production-deployment-guide.md

# Plex Backup System - Production Deployment Guide

This guide helps you deploy the enhanced Plex backup system safely in a production environment.

## Pre-Deployment Checklist

### 1. System Requirements Verification

- [ ] **Operating System**: Linux (tested on Ubuntu/Debian)
- [ ] **Shell**: Bash 4.0+ available
- [ ] **Dependencies Installed**:
  - [ ] `jq` - JSON processing (required for performance logging)
  - [ ] `sqlite3` - Database tools (for fallback integrity checks)
  - [ ] `curl` - HTTP client (for webhook notifications)
  - [ ] `sendmail` - Email delivery (if using email notifications)
  - [ ] `tar` and `gzip` - Archive tools
  - [ ] `sudo` access to Plex files and service management

### 2. Environment Setup

- [ ] **Backup Directory**: Ensure `/mnt/share/media/backups/plex` exists and has sufficient space
- [ ] **Log Directory**: Ensure `/mnt/share/media/backups/logs` exists and is writable
- [ ] **Script Directory**: Place scripts in `/home/acedanger/shell` or update paths accordingly
- [ ] **Permissions**: Verify script user can read Plex files and control Plex service

### 3. Configuration Verification

- [ ] **Plex Service Management**: Test `systemctl stop plexmediaserver` and `systemctl start plexmediaserver`
- [ ] **File Paths**: Verify Plex database locations in script match your installation
- [ ] **Plex SQLite Binary**: Confirm `/usr/lib/plexmediaserver/Plex SQLite` exists
- [ ] **Disk Space**: Ensure backup location has 2x current Plex database size available

## Testing Phase

### 1. Run Unit Tests

```bash
cd /home/acedanger/shell
./test-plex-backup.sh unit
```

**Expected Result**: All 9 tests should pass (100% success rate)

### 2. Run Integration Tests

```bash
cd /home/acedanger/shell
./integration-test-plex.sh
```

**Expected Result**: All integration tests should pass

### 3. Test Dry Run

```bash
# Test integrity check only (non-destructive)
sudo ./backup-plex.sh --check-integrity --non-interactive
```

**Expected Result**: Should complete without stopping Plex or creating backups

### 4. Test Notification Systems

#### Webhook Testing

```bash
# Replace with your actual webhook URL
sudo ./backup-plex.sh --check-integrity --webhook=https://your-webhook-endpoint.com/test
```

#### Email Testing

```bash
# Replace with your email address
sudo ./backup-plex.sh --check-integrity --email=admin@yourdomain.com
```

## Production Deployment Steps

### 1. Initial Backup Test

```bash
# Create a manual backup during maintenance window
sudo ./backup-plex.sh --non-interactive
```

**Verify**:

- [ ] Plex service stopped and restarted properly
- [ ] Backup files created in `/mnt/share/media/backups/plex/YYYYMMDD/`
- [ ] Log files updated in `/mnt/share/media/backups/logs/`
- [ ] Performance log created if enabled
- [ ] Notifications sent if configured

### 2. Validate Backup Integrity

```bash
# Run validation on the created backup
./validate-plex-backups.sh --report
```

### 3. Test Restore Process (Optional)

```bash
# In a test environment, verify restore functionality
./restore-plex.sh --list
./restore-plex.sh --validate YYYYMMDD
```

## Automated Scheduling

### 1. Cron Configuration

Create a cron job for automated backups:

```bash
# Edit crontab for root user
sudo crontab -e

# Add entry for daily backup at 2 AM
0 2 * * * /home/acedanger/shell/backup-plex.sh --non-interactive --webhook=https://your-webhook.com/plex-backup 2>&1 | logger -t plex-backup
```

### 2. Systemd Timer (Alternative)

Create systemd service and timer files:

```bash
# Create service file
sudo tee /etc/systemd/system/plex-backup.service > /dev/null << 'EOF'
[Unit]
Description=Plex Media Server Backup
After=network.target

[Service]
Type=oneshot
User=root
ExecStart=/home/acedanger/shell/backup-plex.sh --non-interactive
StandardOutput=journal
StandardError=journal
EOF

# Create timer file
sudo tee /etc/systemd/system/plex-backup.timer > /dev/null << 'EOF'
[Unit]
Description=Run Plex backup daily
Requires=plex-backup.service

[Timer]
OnCalendar=daily
Persistent=true
RandomizedDelaySec=30m

[Install]
WantedBy=timers.target
EOF

# Enable and start timer
sudo systemctl daemon-reload
sudo systemctl enable plex-backup.timer
sudo systemctl start plex-backup.timer
```

## Monitoring and Maintenance

### 1. Log Monitoring

Monitor backup logs for issues:

```bash
# Check recent backup logs
tail -f /mnt/share/media/backups/logs/plex-backup-$(date +%Y-%m-%d).log

# Check system logs for backup service
sudo journalctl -u plex-backup.service -f
```

### 2. Performance Log Analysis

```bash
# View performance trends
jq '.[] | select(.operation == "full_backup") | {timestamp, duration_seconds}' \
  /home/acedanger/shell/logs/plex-backup-performance.json | tail -10
```

### 3. Regular Validation

Schedule weekly backup validation:

```bash
# Add to crontab
0 3 * * 0 /home/acedanger/shell/validate-plex-backups.sh --report --fix 2>&1 | logger -t plex-backup-validation
```

## Troubleshooting Guide

### Common Issues

#### 1. Permission Denied Errors

```bash
# Fix script permissions
chmod +x /home/acedanger/shell/*.sh

# Fix backup directory permissions
sudo chown -R $(whoami):$(whoami) /mnt/share/media/backups/
```

#### 2. Plex Service Issues

```bash
# Check Plex service status
sudo systemctl status plexmediaserver

# Manually restart if needed
sudo systemctl restart plexmediaserver
```

#### 3. Insufficient Disk Space

```bash
# Check available space
df -h /mnt/share/media/backups/

# Clean old backups manually if needed
./backup-plex.sh # Script will auto-cleanup based on retention policy
```

#### 4. Database Integrity Issues

```bash
# Run integrity check only
sudo ./backup-plex.sh --check-integrity --auto-repair

# Manual database repair if needed
sudo ./backup-plex.sh --auto-repair
```

### Performance Optimization

#### 1. Parallel Processing

- Enable parallel verification for faster backups (default: enabled)
- Disable with `--no-parallel` if experiencing issues

#### 2. Performance Monitoring

- Disable with `--no-performance` if not needed
- Monitor trends to optimize backup timing

#### 3. Notification Optimization

- Use webhooks instead of email for faster notifications
- Configure webhook endpoints with proper error handling

## Security Considerations

### 1. File Permissions

```bash
# Secure backup files
chmod 600 /home/acedanger/shell/logs/plex-backup*.json
chmod 700 /mnt/share/media/backups/plex/
```

### 2. Webhook Security

- Use HTTPS endpoints for webhooks
- Implement webhook signature verification if possible
- Avoid including sensitive data in webhook payloads

### 3. Access Control

- Limit script execution to authorized users
- Consider using dedicated backup user account
- Regularly audit file access permissions

## Backup Retention Strategy

The script automatically manages backup retention:

- **Default**: Keep 10 most recent backups
- **Age-based**: Remove backups older than 30 days
- **Configurable**: Modify `MAX_BACKUPS_TO_KEEP` and `MAX_BACKUP_AGE_DAYS` in script

## Recovery Planning

### 1. Backup Restoration

```bash
# List available backups
./restore-plex.sh --list

# Restore specific backup
sudo ./restore-plex.sh --restore YYYYMMDD
```

### 2. Emergency Procedures

1. Stop Plex service: `sudo systemctl stop plexmediaserver`
2. Backup current data: `./restore-plex.sh --backup-current`
3. Restore from backup: `sudo ./restore-plex.sh --restore YYYYMMDD`
4. Start Plex service: `sudo systemctl start plexmediaserver`

## Success Metrics

Monitor these metrics to ensure backup system health:

- [ ] **Backup Success Rate**: >99% successful backups
- [ ] **Backup Duration**: Consistent timing (tracked in performance logs)
- [ ] **Storage Usage**: Within acceptable limits
- [ ] **Service Downtime**: Minimal Plex service interruption
- [ ] **Notification Delivery**: Reliable alert delivery
- [ ] **Validation Results**: Regular successful backup validation

## Support and Updates

### Getting Help

1. Check logs for error messages
2. Run validation tools for diagnosis
3. Review troubleshooting guide
4. Test with `--check-integrity` for safe debugging

### Script Updates

- Keep scripts updated with latest features
- Test updates in non-production environment first
- Backup current scripts before updating
- Review changelog for breaking changes

---

**Note**: This deployment guide assumes a typical Plex Media Server installation. Adjust paths and configurations based on your specific environment.