shell/plex/README.md

# Plex Backup and Management Scripts

**Author:** Peter Wood <peter@peterwood.dev>

This directory contains a streamlined suite of scripts for Plex Media Server backup, monitoring, and management operations. After identifying and resolving the root cause of database corruption issues (aggressive auto-repair schedules), the toolkit has been simplified to focus on reliable, safe operations.

## 🚨 Important Notes

- **Database corruption issues resolved** (June 21, 2025)
- **Custom backup system DISABLED** (June 21, 2025) - Replaced by Plex built-in backups
- **Plex built-in backups active** - Automatic database backups every 3 days
- **Manual intervention required** for database repairs
- **Redundant scripts moved** to `deprecated/` folder

## ⚙️ Backup System Status

**Current State:** Using **Plex Built-in Scheduled Backups**
- ✅ **Plex Native Backups:** Active (every 3 days)
- ✅ **Backup Location:** `/mnt/share/media/backups/plex` (NAS)
- ❌ **Custom backup scripts:** DISABLED (June 21, 2025)
- ✅ **Health monitoring tools:** Available for manual use

**Why the change?**
Plex's built-in scheduled backup system is more reliable and integrated with the application lifecycle. The custom backup system has been disabled to avoid conflicts and redundancy.

## 🎯 Quick Start

**With Plex Built-in Backups (Current Setup):**

```bash
# Check database health (safe, read-only)
./plex-db-manager.sh check

# Basic Plex service management
./plex.sh status

# Emergency nuclear recovery (LAST RESORT ONLY)
sudo ./nuclear-plex-recovery.sh --dry-run    # Check what would happen
sudo ./nuclear-plex-recovery.sh --auto       # Actual recovery

# Check built-in Plex backups
./check-plex-builtin-backups.sh
```

**Recommended Operations:**
```bash
# Weekly health check (manual)
./plex-db-manager.sh check && echo "✅ Databases healthy"

# Check Plex built-in backup status
./check-plex-builtin-backups.sh

# Monthly validation (if needed)
./validate-plex-backups.sh --latest
```

**🚫 Disabled Operations:**
```bash
# These are NO LONGER NEEDED (Plex handles backups automatically)
# ./backup-plex.sh --non-interactive --disable-auto-repair
# ./validate-plex-backups.sh --fix
```

## 📁 Active Scripts

### 🔧 Core Management

#### `plex-db-manager.sh` ⭐ **NEW**
**Consolidated database management tool - SAFE operations only**

**Features:**
- Read-only database integrity checking (currently active)
- Proper service management with synchronization
- Manual repair intervention (currently disabled for safety)
- Comprehensive logging and error handling

**Current Status:**
- ✅ Integrity checking: ACTIVE and safe
- ⚠️ Repair functionality: DISABLED (safety measure)
- ✅ Service management: ACTIVE

**Usage:**
```bash
./plex-db-manager.sh check              # Safe integrity check
./plex-db-manager.sh repair             # Shows disabled message
./plex-db-manager.sh help               # Show all options
./plex-db-manager.sh                    # Show help (no args)
```

**Example Output:**
```bash
$ ./plex-db-manager.sh check
+================================================================+
|              PLEX DATABASE MANAGEMENT TOOL                    |
+================================================================+

[2025-06-21 14:30:15] INFO: Checking database integrity for all databases
[2025-06-21 14:30:16] SUCCESS: Database integrity check passed for main database
[2025-06-21 14:30:17] SUCCESS: Database integrity check passed for blobs database
[2025-06-21 14:30:18] SUCCESS: All database integrity checks completed successfully
```

### 🔄 Backup & Recovery

#### `backup-plex.sh` ⚠️ **DISABLED**
**Enhanced Plex backup script - NOW DISABLED**

**Status:** ❌ **DISABLED** (June 21, 2025)
**Reason:** Replaced by Plex's built-in scheduled backup system

**Previous Features:**
- Database integrity checking (auto-repair **disabled** by default)
- WAL file handling for SQLite databases
- Performance monitoring with JSON-based logging
- Multi-channel notifications (console, webhook, email)
- Safe service management with extended timeouts
- Automated cleanup of old backups

**Current Recommendation:**
Use Plex's built-in scheduled backups instead. Available for emergency manual use only.

**Manual Usage (Emergency Only):**
```bash
./backup-plex.sh --help                     # Show available options
./backup-plex.sh --check-integrity          # Database health check only
./backup-plex.sh --non-interactive --disable-auto-repair  # Manual backup
```

**Plex Built-in Backup Status:**
```bash
# Check current built-in backups
ls -la "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/"*.backup.*

# Recent built-in backup
ls -lath /var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases/*.backup.* | head -1
```

#### `nuclear-plex-recovery.sh`
**Last-resort complete database replacement**

⚠️ **WARNING:** Completely replaces databases with backup copies. All data since backup will be lost.

**Features:**
- Complete database replacement from backups
- Automatic service management
- Pre-recovery validation and safety checks
- Rollback capability if recovery fails

**Usage:**
```bash
sudo ./nuclear-plex-recovery.sh         # Interactive recovery
sudo ./nuclear-plex-recovery.sh --auto  # Automated recovery
sudo ./nuclear-plex-recovery.sh --dry-run  # Show what would be done
sudo ./nuclear-plex-recovery.sh --help  # Show all options
```

**Example Workflow:**
```bash
# First, check what would be restored
$ sudo ./nuclear-plex-recovery.sh --dry-run
Nuclear Recovery Plan:
- Latest backup: plex-backup-20250621.tar.gz
- Databases to replace: 2 files
- Service downtime: ~5 minutes

# If satisfied, proceed with recovery
$ sudo ./nuclear-plex-recovery.sh --auto
[WARNING] This will DESTROY current databases
[SUCCESS] Recovery completed successfully
```
### 🔍 Validation & Monitoring

#### `validate-plex-backups.sh` ⚠️ **LIMITED USE**
**Backup validation - Now limited to legacy backup validation**

**Status:** ⚠️ **LIMITED USE** (June 21, 2025)
**Reason:** Mainly applies to old custom backup files, not Plex built-in backups

**Current Functionality:**
- Can validate legacy custom backup archives
- Database integrity checking within old backups
- Historical backup analysis for troubleshooting

**Limited Usage:**
```bash
./validate-plex-backups.sh --latest      # Validate latest custom backup (if any)
./validate-plex-backups.sh --help        # Show available options
```

**For Plex Built-in Backups:**
```bash
# Plex built-in backups don't need external validation
# They're automatically managed and verified by Plex
ls -la "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/"*.backup.*
```

#### `monitor-plex-backup.sh` ⚠️ **LIMITED USE**
**System monitoring - Limited functionality with built-in backups**

**Status:** ⚠️ **LIMITED USE** (June 21, 2025)
**Reason:** Custom backup monitoring less relevant with Plex built-in system

**Available Functionality:**
- General system health monitoring
- Service status verification
- Disk space monitoring

**Usage:**
```bash
./monitor-plex-backup.sh --summary   # Basic system summary
./monitor-plex-backup.sh --help      # Show available options
```

**Alternative: Check Plex Built-in Backup Status**
```bash
# Check when last backup was created
stat "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/"*.backup.* 2>/dev/null | grep Modify | tail -1

# Check backup file sizes
ls -lah "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/"*.backup.* 2>/dev/null
```
### 🎛️ Service Management

#### `plex.sh`
**Basic Plex service management and system operations**

**Features:**
- Service start/stop/restart/status operations
- Interactive menu system for easy management
- Process monitoring and health checks
- Integration with other backup/repair tools

**Usage:**
```bash
./plex.sh start                     # Start Plex service
./plex.sh stop                      # Stop Plex service
./plex.sh restart                   # Restart Plex service
./plex.sh status                    # Show service status
./plex.sh repair                    # Launch repair menu (calls other tools)
./plex.sh nuclear                   # Nuclear recovery option
./plex.sh                           # Interactive menu
```

**Example Usage:**
```bash
# Check current service status
$ ./plex.sh status
Plex Media Server Status: RUNNING
Uptime: 2 days, 14 hours, 23 minutes

# Interactive menu
$ ./plex.sh
=== Plex Management Menu ===
1) Start Service
2) Stop Service  
3) Restart Service
4) Service Status
5) Database Check
6) Exit
Enter your choice [1-6]:
```

### 📊 Utility Scripts

#### `check-plex-builtin-backups.sh` ⭐ **NEW**
**Monitor Plex's built-in backup system status**

**Features:**
- Check status of Plex's automatic backups
- Display latest backup information and frequency
- Validate backup location accessibility
- Show legacy custom backup file counts

**Usage:**
```bash
./check-plex-builtin-backups.sh          # Basic backup status
./check-plex-builtin-backups.sh --detailed # Detailed file information
./check-plex-builtin-backups.sh --help   # Show help
```

**Example Output:**
```bash
$ ./check-plex-builtin-backups.sh
✅ Found 1 built-in backup file(s)

📅 Latest backup:
   File: com.plexapp.plugins.library.db.backup.20250621_063833
   Date: 2025-06-21 06:38:56
   Size: 149M

📊 Summary:
✅ Custom backup system: DISABLED (June 21, 2025)
✅ Plex built-in backups: ACTIVE
ℹ️  Backup frequency: Every 3 days (automatic)
```

#### `plex-recent-additions.sh`
**Recent media additions reporting**

**Features:**
- Recent additions reporting
- Media library analysis
- Usage statistics

**Usage:**
```bash
./plex-recent-additions.sh          # Generate recent additions report
```

#### `cleanup-plex-databases.sh` ⭐ **NEW**
**Safe cleanup of temporary and corrupted database files**

**Features:**
- Removes temporary files, recovery files, and corrupted directories
- Preserves active databases and legitimate backups
- Dry-run mode for safe preview
- Comprehensive safety checks and validation

**Usage:**
```bash
./cleanup-plex-databases.sh --dry-run   # Preview what will be removed
sudo ./cleanup-plex-databases.sh        # Perform actual cleanup
./cleanup-plex-databases.sh --help      # Show all options
```

**What it removes:**
- Temporary files (`*-tmp`, `*-2025-*-tmp`)
- MD5 checksum files (`*.md5`)  
- Recovery files (`*recovery*`)
- Repaired files (`*repaired*`)
- Empty backup files (`*empty-backup*`)
- Repair log files (`DBRepair.log`)
- Corrupted database directories (`corrupted-*`)

**What it preserves:**
- Active database files (`*.db`, `*.db-shm`, `*.db-wal`)
- Plex built-in backups (`*.backup.*`)
- Historical dated backups (`*.db.YYYYMMDD`)
- EPG databases (`tv.plex.providers.epg.cloud-*.db`)
### 🧪 Testing & Integration

#### `test-plex-backup.sh`
**Comprehensive testing suite**

**Features:**
- Unit testing for individual backup components
- Integration testing for full backup workflows
- Database integrity test scenarios
- Performance benchmarking
- Error condition simulation and recovery testing

**Usage:**
```bash
./test-plex-backup.sh              # Run full test suite
./test-plex-backup.sh --unit       # Unit tests only
./test-plex-backup.sh --integration # Integration tests only
./test-plex-backup.sh --quick      # Quick smoke tests
./test-plex-backup.sh --help       # Show all options
```

**Example Run:**
```bash
$ ./test-plex-backup.sh --quick
[TEST] Quick smoke test suite
[PASS] Database access test
[PASS] Service management test  
[PASS] Backup directory access test
[PASS] All 3 tests passed in 15 seconds
```

#### `integration-test-plex.sh`
**End-to-end integration testing**

**Features:**
- Complete backup/restore workflow testing
- Multi-system compatibility testing
- Performance regression testing
- Stress testing and load simulation

**Usage:**
```bash
./integration-test-plex.sh         # Full integration test suite
./integration-test-plex.sh --stress # Stress testing
```

#### `validate-plex-recovery.sh`
**Recovery validation and verification**

**Features:**
- Database integrity verification
- Service functionality testing
- Library accessibility checks
- Performance validation

**Usage:**
```bash
./validate-plex-recovery.sh        # Full validation suite
./validate-plex-recovery.sh --quick # Quick validation checks
```
## 📁 Deprecated Scripts

Scripts moved to `deprecated/` folder (June 21, 2025):

- **`plex-database-repair.sh`** → Use `plex-db-manager.sh`
- **`recover-plex-database.sh`** → Use `nuclear-plex-recovery.sh` 
- **`restore-plex.sh`** → Use `nuclear-plex-recovery.sh`
- **`icu-aware-recovery.sh`** → Functionality built into other scripts

See `deprecated/README.md` for details and emergency access.

## 🏗️ Current System Architecture

### Active Script Relationships

```mermaid
graph TD
    A[plex-db-manager.sh] --> B[Database Health Checks]
    C[backup-plex.sh] --> D[validate-plex-backups.sh]
    C --> E[monitor-plex-backup.sh]
    F[nuclear-plex-recovery.sh] --> G[validate-plex-recovery.sh]
    H[plex.sh] --> A
    H --> C
    H --> F
    I[test-plex-backup.sh] --> C
    I --> A
    J[integration-test-plex.sh] --> C
    K[plex-recent-additions.sh]
    
    style A fill:#e1f5fe
    style C fill:#f3e5f5
    style F fill:#ffebee
    style H fill:#e8f5e8
```

**Legend:**
- 🔵 **Blue** (plex-db-manager.sh): Safe database operations
- 🟣 **Purple** (backup-plex.sh): Backup operations
- 🔴 **Red** (nuclear-plex-recovery.sh): Emergency recovery
- 🟢 **Green** (plex.sh): Service management hub

### Current Data Flow

**With Plex Built-in Backups (Current System):**

1. **Automated Operations:**
   - **Plex Built-in Backups:** Every 3 days (automatic, no user intervention)
   - **Service Monitoring:** `plex.sh status` for operational checks (manual)

2. **Manual Health Monitoring (as needed):**
   - **Database Health:** `plex-db-manager.sh check` for manual assessment
   - **Service Status:** `plex.sh status` for service verification

3. **Emergency Recovery Workflow (unchanged):**
   - **Issue Detection:** Service failures or manual observation
   - **Initial Assessment:** `plex-db-manager.sh check` to diagnose problems
   - **Service Recovery:** `plex.sh restart` for service-level issues
   - **Nuclear Recovery:** `nuclear-plex-recovery.sh --auto` for database corruption
   - **Post-Recovery Verification:** `validate-plex-recovery.sh` confirms recovery

**Built-in Backup Management:**
```bash
# Check latest Plex built-in backup
ls -lath "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/"*.backup.* | head -1

# Monitor backup frequency (should be every 3 days)
ls -la "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/"*.backup.*
```

**🚫 Disabled Workflows:**
- ~~Daily custom backup creation~~ → Replaced by Plex built-in
- ~~Backup validation and monitoring~~ → Not needed for built-in backups
- ~~Performance metrics collection~~ → Plex handles internally

## 🔧 Configuration & Setup

### Cron Schedule (Current)

**Plex Built-in Backup System Active (June 21, 2025):**

```bash
# Plex automatically creates backups every 3 days
# Location: /var/lib/plexmediaserver/.../Databases/*.backup.*
# No manual cron jobs needed for backups

# Optional: Weekly database health check (currently disabled)
# 0 6 * * 0 /home/acedanger/shell/plex/plex-db-manager.sh check

# DISABLED - Custom backup system (replaced by Plex built-in)
# 15 4 * * * /home/acedanger/shell/plex/backup-plex.sh --non-interactive --disable-auto-repair
# 0 7 * * * /home/acedanger/shell/plex/validate-plex-backups.sh --fix
```

**✅ Current Active Schedule:**
- **Plex Built-in Backups:** Every 3 days (automatic)
- **Custom Scripts:** Disabled (manual use only)

**⚠️ Previous schedule (DISABLED):**
- ~~Daily integrity check~~ → No longer needed
- ~~Daily custom backup~~ → Replaced by Plex built-in
- ~~Daily backup validation~~ → Not applicable to built-in backups
- ~~Every 30 minutes auto-repair~~ → **Fixed: Caused database corruption**

### Environment Configuration

- **Backup Location:** `/mnt/share/media/backups/plex`
- **Log Location:** `./logs/` (local) and `/mnt/share/media/backups/logs` (shared)
- **Plex Database Path:** `/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/`
- **Service Name:** `plexmediaserver`

### Notification Configuration

Scripts support multiple notification channels:
- **Webhook notifications:** Custom webhook URL support
- **Email notifications:** Via sendmail (if configured)
- **Console output:** Color-coded status messages

## 📊 Monitoring & Health Checks

### Current Health Status

✅ **Database Corruption Issues Resolved** (June 21, 2025)
- Root cause: 30-minute auto-repair cron job
- Solution: Disabled aggressive auto-repair schedule + switched to Plex built-in backups
- Current status: All databases healthy, Plex built-in backups active

✅ **Backup System Status** (June 21, 2025)
- **Plex Built-in Backups:** ✅ Active (every 3 days)
- **Custom Backup Scripts:** ❌ Disabled (available for manual use)
- **Backup Location:** `/mnt/share/media/backups/plex` (NAS mounted)

**Latest Built-in Backup Check:**
```bash
# Check when Plex last created a backup
ls -lath "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/"*.backup.* | head -1
```

### Key Metrics Tracked

- Backup success/failure rates
- Database integrity status
- Service uptime and performance  
- Disk space utilization
- Recovery operation success

## 🚨 Emergency Procedures

### Database Issues

**Step-by-step troubleshooting:**

1. **Quick Health Check** (safe)
   ```bash
   ./plex-db-manager.sh check
   ```

2. **Review Service Status**  
   ```bash
   ./plex.sh status
   systemctl status plexmediaserver
   ```

3. **Check Recent Logs**
   ```bash
   tail -f logs/plex-backup-$(date +%Y-%m-%d).log
   journalctl -u plexmediaserver -f
   ```

4. **Service Recovery** (if needed)
   ```bash
   ./plex.sh restart
   ./plex-db-manager.sh check  # Verify after restart
   ```

5. **Nuclear Recovery** (LAST RESORT)
   ```bash
   # Always dry-run first
   sudo ./nuclear-plex-recovery.sh --dry-run
   
   # If satisfied with plan:
   sudo ./nuclear-plex-recovery.sh --auto
   
   # Validate recovery
   ./validate-plex-recovery.sh
   ```

### Service Issues

**Diagnostic steps:**

1. **Basic Status Check**
   ```bash
   ./plex.sh status
   ps aux | grep -i plex
   ```

2. **Port and Network Check**
   ```bash
   netstat -tlnp | grep :32400
   curl -I http://localhost:32400/web/
   ```

3. **Safe Service Restart**
   ```bash
   ./plex.sh stop    # Graceful shutdown
   sleep 5           # Allow complete shutdown
   ./plex.sh start   # Clean start
   ```

4. **Database Verification** (after service issues)
   ```bash
   ./plex-db-manager.sh check
   ```

## 📚 Documentation Files

### Available Guides

- **[BACKUP-SYSTEM-TRANSITION.md](BACKUP-SYSTEM-TRANSITION.md)** - Transition from custom to Plex built-in backups
- **[CORRUPTION-RESOLUTION-SUMMARY.md](CORRUPTION-RESOLUTION-SUMMARY.md)** - Root cause analysis and resolution
- **[plex-backup.md](plex-backup.md)** - Detailed backup system documentation (legacy)
- **[plex-management.md](plex-management.md)** - Service management procedures
- **[deprecated/README.md](deprecated/README.md)** - Information about retired scripts

### Key Documentation

- **Primary Tool:** `plex-db-manager.sh` - New consolidated database management
- **Backup Strategy:** Conservative approach with manual repair intervention
- **Recovery Method:** Nuclear recovery for emergency situations
- **Monitoring:** Comprehensive logging and health tracking

## 🏷️ Version Information

- **Script Suite Version:** 3.0 (Streamlined)
- **Major Update:** June 21, 2025 - Corruption issues resolved
- **Author:** Peter Wood <peter@peterwood.dev>
- **Compatibility:** Ubuntu 20.04+, Debian 11+
- **Plex Version:** Compatible with Plex Media Server 1.25+

## 📞 Support & Changes

**Recent Major Changes (June 21, 2025):**
- ✅ Fixed database corruption root cause (30-minute auto-repair loop)
- ✅ **DISABLED custom backup system** - Replaced with Plex built-in backups  
- ✅ Retired redundant repair scripts to `deprecated/` folder
- ✅ Implemented Plex native scheduled backups (every 3 days)
- ✅ Created consolidated database manager (`plex-db-manager.sh`)
- ✅ Enhanced service management with safer shutdown procedures

## 💡 Best Practices

### Current Recommended Workflow (Updated)

1. **Let Plex Handle Backups:**
   - ✅ Plex built-in backups run automatically every 3 days
   - ✅ No manual intervention needed for routine backups
   - ✅ Backups are stored in NAS location: `/mnt/share/media/backups/plex`

2. **Manual Health Monitoring (as needed):**
   - Use `plex-db-manager.sh check` for occasional health checks
   - Use `plex.sh status` for service status verification
   - Monitor system logs if issues are suspected

3. **When Issues Occur:**
   - **Always start with read-only checks** (`plex-db-manager.sh check`)
   - **Service issues first:** `plex.sh restart` before database interventions
   - **Nuclear recovery as last resort** with proper validation

4. **Backup Verification:**
   ```bash
   # Check Plex built-in backup status
   ls -lath "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/"*.backup.*
   
   # Verify backup location on NAS
   ls -lah /mnt/share/media/backups/plex/
   ```

### What NOT to Do

❌ **Don't re-enable daily custom backup scripts** (conflicts with Plex built-in)
❌ **Don't enable 30-minute auto-repair schedules** (causes corruption)
❌ **Don't use force-kill on Plex processes** (corrupts databases)  
❌ **Don't bypass safety checks** in scripts
❌ **Don't run multiple repair operations simultaneously**
❌ **Don't ignore failed service shutdowns** (wait for graceful exit)

For issues or questions:
- **Author:** Peter Wood <peter@peterwood.dev>
- **Repository:** Part of comprehensive shell script collection