peterwood/shell
1
0
Fork 0
mirror of https://github.com/acedanger/shell.git synced 2025-12-05 22:50:18 -08:00
Code Issues 19 Packages Projects Releases Wiki Activity

Add advanced Plex database recovery and restoration scripts

Browse Source 
- Introduced `recover-plex-database.sh` for comprehensive database recovery with multiple strategies, logging, and rollback capabilities.
- Added `restore-plex.sh` for safe restoration of Plex backups, including validation and dry-run options.
- Created `plex-db-manager.sh` to consolidate database management functionalities, including integrity checks and service management.
- Enhanced logging and error handling across all scripts for better user feedback and troubleshooting.
- Implemented safety measures to prevent running scripts as root and ensure proper service management during operations.
This commit is contained in:
Peter Wood
2025-06-21 07:23:33 -04:00
parent 30a252a500
commit 9b83924597
13 changed files with 1410 additions and 300 deletions
Download Patch File Download Diff File Expand all files Collapse all files

804
plex/README.md
View File

@@ -2,232 +2,317 @@
**Author:** Peter Wood <peter@peterwood.dev>
This directory contains a comprehensive suite of scripts for Plex Media Server backup, restoration, validation, recovery, and management operations. The system provides enterprise-grade backup capabilities with automated integrity checking, multiple recovery strategies, and extensive monitoring.
This directory contains a streamlined suite of scripts for Plex Media Server backup, monitoring, and management operations. After identifying and resolving the root cause of database corruption issues (aggressive auto-repair schedules), the toolkit has been simplified to focus on reliable, safe operations.
## 🚨 Important Notes
- **Database corruption issues resolved** (June 21, 2025)
- **Custom backup system DISABLED** (June 21, 2025) - Replaced by Plex built-in backups
- **Plex built-in backups active** - Automatic database backups every 3 days
- **Manual intervention required** for database repairs
- **Redundant scripts moved** to `deprecated/` folder
## ⚙️ Backup System Status
**Current State:** Using **Plex Built-in Scheduled Backups**
- ✅ **Plex Native Backups:** Active (every 3 days)
- ✅ **Backup Location:** `/mnt/share/media/backups/plex` (NAS)
- ❌ **Custom backup scripts:** DISABLED (June 21, 2025)
- ✅ **Health monitoring tools:** Available for manual use
**Why the change?**
Plex's built-in scheduled backup system is more reliable and integrated with the application lifecycle. The custom backup system has been disabled to avoid conflicts and redundancy.
## 🎯 Quick Start
**With Plex Built-in Backups (Current Setup):**
```bash
# Create a backup with automatic integrity checking and repair
./backup-plex.sh
# Monitor backup system health
./monitor-plex-backup.sh --watch
# Validate all existing backups
./validate-plex-backups.sh
# Restore from a specific backup
./restore-plex.sh plex-backup-20250605_143022.tar.gz
# Check database health (safe, read-only)
./plex-db-manager.sh check
# Basic Plex service management
./plex.sh status
# Emergency nuclear recovery (LAST RESORT ONLY)
sudo ./nuclear-plex-recovery.sh --dry-run # Check what would happen
sudo ./nuclear-plex-recovery.sh --auto # Actual recovery
# Check built-in Plex backups
./check-plex-builtin-backups.sh
```
## 📁 Scripts Overview
**Recommended Operations:**
```bash
# Weekly health check (manual)
./plex-db-manager.sh check && echo "✅ Databases healthy"
### 🔄 Core Backup & Restoration
# Check Plex built-in backup status
./check-plex-builtin-backups.sh
#### `backup-plex.sh`
# Monthly validation (if needed)
./validate-plex-backups.sh --latest
```
### Enhanced Plex backup script with advanced features
**🚫 Disabled Operations:**
```bash
# These are NO LONGER NEEDED (Plex handles backups automatically)
# ./backup-plex.sh --non-interactive --disable-auto-repair
# ./validate-plex-backups.sh --fix
```
**Author:** Peter Wood <peter@peterwood.dev>
## 📁 Active Scripts
### 🔧 Core Management
#### `plex-db-manager.sh` ⭐ **NEW**
**Consolidated database management tool - SAFE operations only**
**Features:**
- Read-only database integrity checking (currently active)
- Proper service management with synchronization
- Manual repair intervention (currently disabled for safety)
- Comprehensive logging and error handling
- Database integrity checking with automatic repair capabilities
**Current Status:**
- ✅ Integrity checking: ACTIVE and safe
- ⚠️ Repair functionality: DISABLED (safety measure)
- ✅ Service management: ACTIVE
**Usage:**
```bash
./plex-db-manager.sh check # Safe integrity check
./plex-db-manager.sh repair # Shows disabled message
./plex-db-manager.sh help # Show all options
./plex-db-manager.sh # Show help (no args)
```
**Example Output:**
```bash
$ ./plex-db-manager.sh check
+================================================================+
| PLEX DATABASE MANAGEMENT TOOL |
+================================================================+
[2025-06-21 14:30:15] INFO: Checking database integrity for all databases
[2025-06-21 14:30:16] SUCCESS: Database integrity check passed for main database
[2025-06-21 14:30:17] SUCCESS: Database integrity check passed for blobs database
[2025-06-21 14:30:18] SUCCESS: All database integrity checks completed successfully
```
### 🔄 Backup & Recovery
#### `backup-plex.sh` ⚠️ **DISABLED**
**Enhanced Plex backup script - NOW DISABLED**
**Status:** ❌ **DISABLED** (June 21, 2025)
**Reason:** Replaced by Plex's built-in scheduled backup system
**Previous Features:**
- Database integrity checking (auto-repair **disabled** by default)
- WAL file handling for SQLite databases
- Performance monitoring with JSON-based logging
- Parallel verification for improved performance
- Multi-channel notifications (console, webhook, email)
- Comprehensive error handling and recovery
- Safe service management with extended timeouts
- Automated cleanup of old backups
**Usage:**
**Current Recommendation:**
Use Plex's built-in scheduled backups instead. Available for emergency manual use only.
**Manual Usage (Emergency Only):**
```bash
./backup-plex.sh # Standard backup with auto-repair
./backup-plex.sh --disable-auto-repair # Backup without auto-repair
./backup-plex.sh --check-integrity # Integrity check only
./backup-plex.sh --non-interactive # Automated mode for cron jobs
./backup-plex.sh --webhook=URL # Custom webhook notifications
./backup-plex.sh --help # Show available options
./backup-plex.sh --check-integrity # Database health check only
./backup-plex.sh --non-interactive --disable-auto-repair # Manual backup
```
#### `restore-plex.sh`
### Safe restoration script with validation
**Author:** Peter Wood <peter@peterwood.dev>
**Features:**
- Interactive backup selection from available archives
- Backup validation before restoration
- Dry-run mode for testing restoration process
- Automatic backup of current data before restoration
- Service management (stop/start Plex during restoration)
- File ownership and permission restoration
**Usage:**
**Plex Built-in Backup Status:**
```bash
./restore-plex.sh # List available backups
./restore-plex.sh plex-backup-20250605_143022.tar.gz # Restore specific backup
./restore-plex.sh --dry-run backup-file.tar.gz # Test restoration process
./restore-plex.sh --list # List all available backups
```
# Check current built-in backups
ls -la "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/"*.backup.*
### 🔍 Validation & Monitoring
#### `validate-plex-backups.sh`
### Backup validation and health monitoring
**Author:** Peter Wood <peter@peterwood.dev>
**Features:**
- Archive integrity verification (checksum validation)
- Database integrity checking within backups
- Backup completeness validation
- Automated repair suggestions and fixes
- Historical backup analysis
- Performance metrics and reporting
**Usage:**
```bash
./validate-plex-backups.sh # Validate all backups
./validate-plex-backups.sh --fix # Validate and fix issues
./validate-plex-backups.sh --report # Generate detailed report
./validate-plex-backups.sh --latest # Validate only latest backup
```
#### `monitor-plex-backup.sh`
### Real-time backup system monitoring dashboard
**Author:** Peter Wood <peter@peterwood.dev>
**Features:**
- Real-time backup system health monitoring
- Performance metrics and trending
- Backup schedule and execution tracking
- Disk space monitoring and alerts
- Service status verification
- Watch mode with auto-refresh
**Usage:**
```bash
./monitor-plex-backup.sh # Single status check
./monitor-plex-backup.sh --watch # Continuous monitoring
./monitor-plex-backup.sh --help # Show help information
```
### 🛠️ Database Recovery Scripts
#### `recover-plex-database.sh`
### Advanced database recovery with multiple strategies
**Author:** Peter Wood <peter@peterwood.dev>
**Features:**
- Progressive recovery strategy (gentle to aggressive)
- Multiple repair techniques (VACUUM, dump/restore, rebuild)
- Automatic backup before recovery attempts
- Database integrity verification at each step
- Rollback capability if recovery fails
- Comprehensive logging and reporting
**Usage:**
```bash
./recover-plex-database.sh # Interactive recovery
./recover-plex-database.sh --auto # Automated recovery
./recover-plex-database.sh --dry-run # Show recovery plan
./recover-plex-database.sh --gentle # Gentle repair only
```
#### `icu-aware-recovery.sh`
### ICU-aware database recovery for Unicode issues
**Author:** Peter Wood <peter@peterwood.dev>
**Features:**
- ICU collation sequence detection and repair
- Unicode-aware database reconstruction
- Advanced SQLite recovery techniques
- Plex service management during recovery
**Usage:**
```bash
./icu-aware-recovery.sh # Interactive recovery
./icu-aware-recovery.sh --auto # Automated recovery
./icu-aware-recovery.sh --check-only # Check ICU status only
# Recent built-in backup
ls -lath /var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases/*.backup.* | head -1
```
#### `nuclear-plex-recovery.sh`
**Last-resort complete database replacement**
### Last-resort complete database replacement
**Author:** Peter Wood <peter@peterwood.dev>
**⚠️ WARNING:** This script completely replaces existing databases!
⚠️ **WARNING:** Completely replaces databases with backup copies. All data since backup will be lost.
**Features:**
- Complete database replacement from backups
- Automatic backup of current (corrupted) databases
- Rollback capability if replacement fails
- Verification of restored database integrity
- Automatic service management
- Pre-recovery validation and safety checks
- Rollback capability if recovery fails
**Usage:**
```bash
./nuclear-plex-recovery.sh # Interactive recovery
./nuclear-plex-recovery.sh --auto # Automated recovery
./nuclear-plex-recovery.sh --dry-run # Show what would be done
sudo ./nuclear-plex-recovery.sh # Interactive recovery
sudo ./nuclear-plex-recovery.sh --auto # Automated recovery
sudo ./nuclear-plex-recovery.sh --dry-run # Show what would be done
sudo ./nuclear-plex-recovery.sh --help # Show all options
```
#### `validate-plex-recovery.sh`
**Example Workflow:**
```bash
# First, check what would be restored
$ sudo ./nuclear-plex-recovery.sh --dry-run
Nuclear Recovery Plan:
- Latest backup: plex-backup-20250621.tar.gz
- Databases to replace: 2 files
- Service downtime: ~5 minutes
### Recovery validation and verification
# If satisfied, proceed with recovery
$ sudo ./nuclear-plex-recovery.sh --auto
[WARNING] This will DESTROY current databases
[SUCCESS] Recovery completed successfully
```
### 🔍 Validation & Monitoring
**Author:** Peter Wood <peter@peterwood.dev>
#### `validate-plex-backups.sh` ⚠️ **LIMITED USE**
**Backup validation - Now limited to legacy backup validation**
**Status:** ⚠️ **LIMITED USE** (June 21, 2025)
**Reason:** Mainly applies to old custom backup files, not Plex built-in backups
**Current Functionality:**
- Can validate legacy custom backup archives
- Database integrity checking within old backups
- Historical backup analysis for troubleshooting
**Limited Usage:**
```bash
./validate-plex-backups.sh --latest # Validate latest custom backup (if any)
./validate-plex-backups.sh --help # Show available options
```
**For Plex Built-in Backups:**
```bash
# Plex built-in backups don't need external validation
# They're automatically managed and verified by Plex
ls -la "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/"*.backup.*
```
#### `monitor-plex-backup.sh` ⚠️ **LIMITED USE**
**System monitoring - Limited functionality with built-in backups**
**Status:** ⚠️ **LIMITED USE** (June 21, 2025)
**Reason:** Custom backup monitoring less relevant with Plex built-in system
**Available Functionality:**
- General system health monitoring
- Service status verification
- Disk space monitoring
**Usage:**
```bash
./monitor-plex-backup.sh --summary # Basic system summary
./monitor-plex-backup.sh --help # Show available options
```
**Alternative: Check Plex Built-in Backup Status**
```bash
# Check when last backup was created
stat "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/"*.backup.* 2>/dev/null | grep Modify | tail -1
# Check backup file sizes
ls -lah "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/"*.backup.* 2>/dev/null
```
### 🎛️ Service Management
#### `plex.sh`
**Basic Plex service management and system operations**
**Features:**
- Database integrity verification
- Service functionality testing
- Library accessibility checks
- Performance validation
- Web interface connectivity testing
- Service start/stop/restart/status operations
- Interactive menu system for easy management
- Process monitoring and health checks
- Integration with other backup/repair tools
**Usage:**
```bash
./validate-plex-recovery.sh # Full validation suite
./validate-plex-recovery.sh --quick # Quick validation checks
./validate-plex-recovery.sh --detailed # Detailed analysis and reporting
./plex.sh start # Start Plex service
./plex.sh stop # Stop Plex service
./plex.sh restart # Restart Plex service
./plex.sh status # Show service status
./plex.sh repair # Launch repair menu (calls other tools)
./plex.sh nuclear # Nuclear recovery option
./plex.sh # Interactive menu
```
### 🧪 Testing Framework
**Example Usage:**
```bash
# Check current service status
$ ./plex.sh status
Plex Media Server Status: RUNNING
Uptime: 2 days, 14 hours, 23 minutes
# Interactive menu
$ ./plex.sh
=== Plex Management Menu ===
1) Start Service
2) Stop Service
3) Restart Service
4) Service Status
5) Database Check
6) Exit
Enter your choice [1-6]:
```
### 📊 Utility Scripts
#### `check-plex-builtin-backups.sh` ⭐ **NEW**
**Monitor Plex's built-in backup system status**
**Features:**
- Check status of Plex's automatic backups
- Display latest backup information and frequency
- Validate backup location accessibility
- Show legacy custom backup file counts
**Usage:**
```bash
./check-plex-builtin-backups.sh # Basic backup status
./check-plex-builtin-backups.sh --detailed # Detailed file information
./check-plex-builtin-backups.sh --help # Show help
```
**Example Output:**
```bash
$ ./check-plex-builtin-backups.sh
✅ Found 1 built-in backup file(s)
📅 Latest backup:
File: com.plexapp.plugins.library.db.backup.20250621_063833
Date: 2025-06-21 06:38:56
Size: 149M
📊 Summary:
✅ Custom backup system: DISABLED (June 21, 2025)
✅ Plex built-in backups: ACTIVE
ℹ️ Backup frequency: Every 3 days (automatic)
```
#### `plex-recent-additions.sh`
**Recent media additions reporting**
**Features:**
- Recent additions reporting
- Media library analysis
- Usage statistics
**Usage:**
```bash
./plex-recent-additions.sh # Generate recent additions report
```
### 🧪 Testing & Integration
#### `test-plex-backup.sh`
### Comprehensive testing suite
**Author:** Peter Wood <peter@peterwood.dev>
**Comprehensive testing suite**
**Features:**
- Unit testing for individual backup components
- Integration testing for full backup workflows
- Database integrity test scenarios
@@ -235,116 +320,157 @@ This directory contains a comprehensive suite of scripts for Plex Media Server b
- Error condition simulation and recovery testing
**Usage:**
```bash
./test-plex-backup.sh # Run full test suite
./test-plex-backup.sh --unit # Unit tests only
./test-plex-backup.sh --integration # Integration tests only
./test-plex-backup.sh --quick # Quick smoke tests
./test-plex-backup.sh --help # Show all options
```
**Example Run:**
```bash
$ ./test-plex-backup.sh --quick
[TEST] Quick smoke test suite
[PASS] Database access test
[PASS] Service management test
[PASS] Backup directory access test
[PASS] All 3 tests passed in 15 seconds
```
#### `integration-test-plex.sh`
### End-to-end integration testing
**Author:** Peter Wood <peter@peterwood.dev>
**End-to-end integration testing**
**Features:**
- Full workflow integration testing
- Isolated test environment creation
- Production-safe testing procedures
- Multi-scenario testing (normal, error, edge cases)
- Cross-script compatibility testing
- Complete backup/restore workflow testing
- Multi-system compatibility testing
- Performance regression testing
- Stress testing and load simulation
**Usage:**
```bash
./integration-test-plex.sh # Full integration test suite
./integration-test-plex.sh --quick # Quick smoke tests
./integration-test-plex.sh --performance # Performance benchmarks
./integration-test-plex.sh # Full integration test suite
./integration-test-plex.sh --stress # Stress testing
```
### 🎮 Management & Utilities
#### `plex.sh`
### Modern Plex service management
**Author:** Peter Wood <peter@peterwood.dev>
#### `validate-plex-recovery.sh`
**Recovery validation and verification**
**Features:**
- Service start/stop/restart/status operations
- Web interface launcher
- Styled console output with Unicode symbols
- Service health monitoring
- Interactive menu system
- Database integrity verification
- Service functionality testing
- Library accessibility checks
- Performance validation
**Usage:**
```bash
./plex.sh start # Start Plex service
./plex.sh stop # Stop Plex service
./plex.sh restart # Restart Plex service
./plex.sh status # Show service status
./plex.sh web # Open web interface
./plex.sh # Interactive menu
./validate-plex-recovery.sh # Full validation suite
./validate-plex-recovery.sh --quick # Quick validation checks
```
## 📁 Deprecated Scripts
#### `plex-recent-additions.sh`
Scripts moved to `deprecated/` folder (June 21, 2025):
### Recent media additions reporting
- **`plex-database-repair.sh`** → Use `plex-db-manager.sh`
- **`recover-plex-database.sh`** → Use `nuclear-plex-recovery.sh`
- **`restore-plex.sh`** → Use `nuclear-plex-recovery.sh`
- **`icu-aware-recovery.sh`** → Functionality built into other scripts
**Author:** Peter Wood <peter@peterwood.dev>
See `deprecated/README.md` for details and emergency access.
**Features:**
## 🏗️ Current System Architecture
- Recent additions reporting (configurable time range)
- Library section filtering
- Formatted output with headers and columns
- Direct SQLite database querying
**Usage:**
```bash
./plex-recent-additions.sh # Show additions from last 7 days
./plex-recent-additions.sh 30 # Show additions from last 30 days
```
## 🏗️ System Architecture
### Script Relationships
### Active Script Relationships
```mermaid
graph TD
A[backup-plex.sh] --> B[validate-plex-backups.sh]
A --> C[monitor-plex-backup.sh]
B --> D[restore-plex.sh]
D --> E[validate-plex-recovery.sh]
F[recover-plex-database.sh] --> E
G[icu-aware-recovery.sh] --> E
H[nuclear-plex-recovery.sh] --> E
I[test-plex-backup.sh] --> A
J[integration-test-plex.sh] --> A
K[plex.sh] --> A
L[plex-recent-additions.sh] --> A
A[plex-db-manager.sh] --> B[Database Health Checks]
C[backup-plex.sh] --> D[validate-plex-backups.sh]
C --> E[monitor-plex-backup.sh]
F[nuclear-plex-recovery.sh] --> G[validate-plex-recovery.sh]
H[plex.sh] --> A
H --> C
H --> F
I[test-plex-backup.sh] --> C
I --> A
J[integration-test-plex.sh] --> C
K[plex-recent-additions.sh]
style A fill:#e1f5fe
style C fill:#f3e5f5
style F fill:#ffebee
style H fill:#e8f5e8
```
### Data Flow
**Legend:**
- 🔵 **Blue** (plex-db-manager.sh): Safe database operations
- 🟣 **Purple** (backup-plex.sh): Backup operations
- 🔴 **Red** (nuclear-plex-recovery.sh): Emergency recovery
- 🟢 **Green** (plex.sh): Service management hub
1. **Backup Creation:** `backup-plex.sh` creates validated backups
2. **Monitoring:** `monitor-plex-backup.sh` tracks system health
3. **Validation:** `validate-plex-backups.sh` ensures backup integrity
4. **Recovery:** Multiple recovery scripts handle different failure scenarios
5. **Restoration:** `restore-plex.sh` safely restores from backups
6. **Verification:** `validate-plex-recovery.sh` confirms successful recovery
### Current Data Flow
## 🔧 Configuration
**With Plex Built-in Backups (Current System):**
### Environment Setup
1. **Automated Operations:**
- **Plex Built-in Backups:** Every 3 days (automatic, no user intervention)
- **Service Monitoring:** `plex.sh status` for operational checks (manual)
All scripts share common configuration patterns:
2. **Manual Health Monitoring (as needed):**
- **Database Health:** `plex-db-manager.sh check` for manual assessment
- **Service Status:** `plex.sh status` for service verification
3. **Emergency Recovery Workflow (unchanged):**
- **Issue Detection:** Service failures or manual observation
- **Initial Assessment:** `plex-db-manager.sh check` to diagnose problems
- **Service Recovery:** `plex.sh restart` for service-level issues
- **Nuclear Recovery:** `nuclear-plex-recovery.sh --auto` for database corruption
- **Post-Recovery Verification:** `validate-plex-recovery.sh` confirms recovery
**Built-in Backup Management:**
```bash
# Check latest Plex built-in backup
ls -lath "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/"*.backup.* | head -1
# Monitor backup frequency (should be every 3 days)
ls -la "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/"*.backup.*
```
**🚫 Disabled Workflows:**
- ~~Daily custom backup creation~~ → Replaced by Plex built-in
- ~~Backup validation and monitoring~~ → Not needed for built-in backups
- ~~Performance metrics collection~~ → Plex handles internally
## 🔧 Configuration & Setup
### Cron Schedule (Current)
**Plex Built-in Backup System Active (June 21, 2025):**
```bash
# Plex automatically creates backups every 3 days
# Location: /var/lib/plexmediaserver/.../Databases/*.backup.*
# No manual cron jobs needed for backups
# Optional: Weekly database health check (currently disabled)
# 0 6 * * 0 /home/acedanger/shell/plex/plex-db-manager.sh check
# DISABLED - Custom backup system (replaced by Plex built-in)
# 15 4 * * * /home/acedanger/shell/plex/backup-plex.sh --non-interactive --disable-auto-repair
# 0 7 * * * /home/acedanger/shell/plex/validate-plex-backups.sh --fix
```
**✅ Current Active Schedule:**
- **Plex Built-in Backups:** Every 3 days (automatic)
- **Custom Scripts:** Disabled (manual use only)
**⚠️ Previous schedule (DISABLED):**
- ~~Daily integrity check~~ → No longer needed
- ~~Daily custom backup~~ → Replaced by Plex built-in
- ~~Daily backup validation~~ → Not applicable to built-in backups
- ~~Every 30 minutes auto-repair~~ → **Fixed: Caused database corruption**
### Environment Configuration
- **Backup Location:** `/mnt/share/media/backups/plex`
- **Log Location:** `./logs/` (local) and `/mnt/share/media/backups/logs` (shared)
@@ -354,71 +480,179 @@ All scripts share common configuration patterns:
### Notification Configuration
Scripts support multiple notification channels:
- **Webhook notifications:** Custom webhook URL support
- **Email notifications:** Via sendmail (if configured)
- **Console output:** Color-coded status messages
### Performance Tuning
## 📊 Monitoring & Health Checks
- **Parallel verification:** Enabled by default for faster operations
- **Performance monitoring:** JSON-based metrics collection
- **Automatic cleanup:** Configurable retention policies
### Current Health Status
## 📊 Monitoring & Alerting
✅ **Database Corruption Issues Resolved** (June 21, 2025)
- Root cause: 30-minute auto-repair cron job
- Solution: Disabled aggressive auto-repair schedule + switched to Plex built-in backups
- Current status: All databases healthy, Plex built-in backups active
### Health Checks
✅ **Backup System Status** (June 21, 2025)
- **Plex Built-in Backups:** ✅ Active (every 3 days)
- **Custom Backup Scripts:** ❌ Disabled (available for manual use)
- **Backup Location:** `/mnt/share/media/backups/plex` (NAS mounted)
The monitoring system tracks:
**Latest Built-in Backup Check:**
```bash
# Check when Plex last created a backup
ls -lath "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/"*.backup.* | head -1
```
### Key Metrics Tracked
- Backup success/failure rates
- Database integrity status
- Service uptime and performance
- Service uptime and performance
- Disk space utilization
- Recovery operation success
### Performance Metrics
- Backup duration and size trends
- Database operation performance
- Service start/stop times
- Recovery operation benchmarks
## 🚨 Emergency Procedures
### Database Corruption
### Database Issues
1. **First Response:** Run `backup-plex.sh --check-integrity`
2. **Gentle Recovery:** Try `recover-plex-database.sh --gentle`
3. **Advanced Recovery:** Use `icu-aware-recovery.sh` for Unicode issues
4. **Last Resort:** Execute `nuclear-plex-recovery.sh` with known good backup
5. **Validation:** Always run `validate-plex-recovery.sh` after recovery
**Step-by-step troubleshooting:**
1. **Quick Health Check** (safe)
```bash
./plex-db-manager.sh check
```
2. **Review Service Status**
```bash
./plex.sh status
systemctl status plexmediaserver
```
3. **Check Recent Logs**
```bash
tail -f logs/plex-backup-$(date +%Y-%m-%d).log
journalctl -u plexmediaserver -f
```
4. **Service Recovery** (if needed)
```bash
./plex.sh restart
./plex-db-manager.sh check # Verify after restart
```
5. **Nuclear Recovery** (LAST RESORT)
```bash
# Always dry-run first
sudo ./nuclear-plex-recovery.sh --dry-run
# If satisfied with plan:
sudo ./nuclear-plex-recovery.sh --auto
# Validate recovery
./validate-plex-recovery.sh
```
### Service Issues
1. **Check Status:** `./plex.sh status`
2. **Restart Service:** `./plex.sh restart`
3. **Monitor Logs:** Check system logs and script logs
4. **Validate Database:** Run integrity checks if service fails to start
**Diagnostic steps:**
## 📚 Additional Documentation
1. **Basic Status Check**
```bash
./plex.sh status
ps aux | grep -i plex
```
- **[Plex Backup System Guide](plex-backup.md)** - Detailed backup system documentation
- **[Plex Management Guide](plex-management.md)** - Service management procedures
- **[Troubleshooting Guide](troubleshooting.md)** - Common issues and solutions
2. **Port and Network Check**
```bash
netstat -tlnp | grep :32400
curl -I http://localhost:32400/web/
```
3. **Safe Service Restart**
```bash
./plex.sh stop # Graceful shutdown
sleep 5 # Allow complete shutdown
./plex.sh start # Clean start
```
4. **Database Verification** (after service issues)
```bash
./plex-db-manager.sh check
```
## 📚 Documentation Files
### Available Guides
- **[BACKUP-SYSTEM-TRANSITION.md](BACKUP-SYSTEM-TRANSITION.md)** - Transition from custom to Plex built-in backups
- **[CORRUPTION-RESOLUTION-SUMMARY.md](CORRUPTION-RESOLUTION-SUMMARY.md)** - Root cause analysis and resolution
- **[plex-backup.md](plex-backup.md)** - Detailed backup system documentation (legacy)
- **[plex-management.md](plex-management.md)** - Service management procedures
- **[deprecated/README.md](deprecated/README.md)** - Information about retired scripts
### Key Documentation
- **Primary Tool:** `plex-db-manager.sh` - New consolidated database management
- **Backup Strategy:** Conservative approach with manual repair intervention
- **Recovery Method:** Nuclear recovery for emergency situations
- **Monitoring:** Comprehensive logging and health tracking
## 🏷️ Version Information
- **Script Suite Version:** 2.0
- **Script Suite Version:** 3.0 (Streamlined)
- **Major Update:** June 21, 2025 - Corruption issues resolved
- **Author:** Peter Wood <peter@peterwood.dev>
- **Last Updated:** June 2025
- **Compatibility:** Ubuntu 20.04+, Debian 11+
- **Plex Version:** Compatible with Plex Media Server 1.25+
## 📞 Support
## 📞 Support & Changes
For issues, questions, or contributions:
**Recent Major Changes (June 21, 2025):**
- ✅ Fixed database corruption root cause (30-minute auto-repair loop)
- ✅ **DISABLED custom backup system** - Replaced with Plex built-in backups
- ✅ Retired redundant repair scripts to `deprecated/` folder
- ✅ Implemented Plex native scheduled backups (every 3 days)
- ✅ Created consolidated database manager (`plex-db-manager.sh`)
- ✅ Enhanced service management with safer shutdown procedures
- **Author:** Peter Wood
- **Email:** <peter@peterwood.dev>
## 💡 Best Practices
### Current Recommended Workflow (Updated)
1. **Let Plex Handle Backups:**
- ✅ Plex built-in backups run automatically every 3 days
- ✅ No manual intervention needed for routine backups
- ✅ Backups are stored in NAS location: `/mnt/share/media/backups/plex`
2. **Manual Health Monitoring (as needed):**
- Use `plex-db-manager.sh check` for occasional health checks
- Use `plex.sh status` for service status verification
- Monitor system logs if issues are suspected
3. **When Issues Occur:**
- **Always start with read-only checks** (`plex-db-manager.sh check`)
- **Service issues first:** `plex.sh restart` before database interventions
- **Nuclear recovery as last resort** with proper validation
4. **Backup Verification:**
```bash
# Check Plex built-in backup status
ls -lath "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/"*.backup.*
# Verify backup location on NAS
ls -lah /mnt/share/media/backups/plex/
```
### What NOT to Do
❌ **Don't re-enable daily custom backup scripts** (conflicts with Plex built-in)
❌ **Don't enable 30-minute auto-repair schedules** (causes corruption)
❌ **Don't use force-kill on Plex processes** (corrupts databases)
❌ **Don't bypass safety checks** in scripts
❌ **Don't run multiple repair operations simultaneously**
❌ **Don't ignore failed service shutdowns** (wait for graceful exit)
For issues or questions:
- **Author:** Peter Wood <peter@peterwood.dev>
- **Repository:** Part of comprehensive shell script collection