peterwood/shell
1
0
Fork 0
mirror of https://github.com/acedanger/shell.git synced 2025-12-06 01:10:12 -08:00
Code Issues 19 Packages Projects Releases Wiki Activity

feat: Add comprehensive Plex recovery validation script

Browse Source 
- Introduced `validate-plex-recovery.sh` for validating Plex database recovery.
- Implemented checks for service status, database integrity, web interface accessibility, API functionality, and recent logs.
- Added detailed recovery summary and next steps for users.

fix: Improve Debian patching script for compatibility

- Enhanced `debian-patches.sh` to securely download and execute bootstrap scripts.
- Updated package mapping logic and ensured proper permissions for patched files.

fix: Update Docker test scripts for better permission handling

- Modified `run-docker-tests.sh` to set appropriate permissions on logs directory.
- Ensured log files have correct permissions after test runs.

fix: Enhance setup scripts for secure installations

- Updated `setup.sh` to securely download and execute installation scripts for zoxide and nvm.
- Improved error handling for failed downloads.

fix: Refine startup script for log directory permissions

- Adjusted `startup.sh` to set proper permissions for log directories and files.

chore: Revamp update-containers.sh for better error handling and logging

- Rewrote `update-containers.sh` to include detailed logging and error handling.
- Added validation for Docker image names and improved overall script robustness.
This commit is contained in:
Peter Wood
2025-06-05 07:22:28 -04:00
parent 8b514ac0b2
commit 0123fc6007
25 changed files with 4407 additions and 608 deletions
Download Patch File Download Diff File Expand all files Collapse all files

521
plex/README.md
View File

@@ -1,293 +1,424 @@
# Plex Backup and Management Scripts
This directory contains all scripts and documentation related to Plex Media Server backup, restoration, validation, and management.
**Author:** Peter Wood <peter@peterwood.dev>
## Scripts Overview
This directory contains a comprehensive suite of scripts for Plex Media Server backup, restoration, validation, recovery, and management operations. The system provides enterprise-grade backup capabilities with automated integrity checking, multiple recovery strategies, and extensive monitoring.
### Core Backup Scripts
## 🎯 Quick Start
```bash
# Create a backup with automatic integrity checking and repair
./backup-plex.sh
# Monitor backup system health
./monitor-plex-backup.sh --watch
# Validate all existing backups
./validate-plex-backups.sh
# Restore from a specific backup
./restore-plex.sh plex-backup-20250605_143022.tar.gz
# Basic Plex service management
./plex.sh status
```
## 📁 Scripts Overview
### 🔄 Core Backup & Restoration
#### `backup-plex.sh`
**Enhanced Plex backup script with advanced features**
### Enhanced Plex backup script with advanced features
- **Full backup operations** with integrity verification
- **Performance monitoring** with JSON-based logging
- **WAL file handling** for SQLite databases
- **Database integrity checks** with automated repair options
- **Parallel processing** for improved performance
- **Multi-channel notifications** (console, webhook, email)
- **Comprehensive logging** with color-coded output
**Author:** Peter Wood <peter@peterwood.dev>
**Features:**
- Database integrity checking with automatic repair capabilities
- WAL file handling for SQLite databases
- Performance monitoring with JSON-based logging
- Parallel verification for improved performance
- Multi-channel notifications (console, webhook, email)
- Comprehensive error handling and recovery
- Automated cleanup of old backups
**Usage:**
```bash
./backup-plex.sh # Standard backup
./backup-plex.sh # Standard backup with auto-repair
./backup-plex.sh --disable-auto-repair # Backup without auto-repair
./backup-plex.sh --check-integrity # Integrity check only
./backup-plex.sh --non-interactive # Automated mode
./backup-plex.sh --auto-repair # Auto-repair database issues
./backup-plex.sh --non-interactive # Automated mode for cron jobs
./backup-plex.sh --webhook=URL # Custom webhook notifications
```
#### `restore-plex.sh`
**Safe restoration script with validation**
### Safe restoration script with validation
- **Backup validation** before restoration
- **Dry-run mode** for testing
- **Current data backup** before restoration
- **Interactive backup selection**
**Author:** Peter Wood <peter@peterwood.dev>
**Features:**
- Interactive backup selection from available archives
- Backup validation before restoration
- Dry-run mode for testing restoration process
- Automatic backup of current data before restoration
- Service management (stop/start Plex during restoration)
- File ownership and permission restoration
**Usage:**
```bash
./restore-plex.sh # List available backups
./restore-plex.sh plex-backup-20250125_143022.tar.gz # Restore specific backup
./restore-plex.sh --dry-run backup-file.tar.gz # Test restoration
./restore-plex.sh plex-backup-20250605_143022.tar.gz # Restore specific backup
./restore-plex.sh --dry-run backup-file.tar.gz # Test restoration process
./restore-plex.sh --list # List all available backups
```
### 🔍 Validation & Monitoring
#### `validate-plex-backups.sh`
**Backup validation and health monitoring**
### Backup validation and health monitoring
- **Archive integrity checking**
- **Backup freshness validation**
- **Comprehensive reporting**
- **Automated fix suggestions**
**Author:** Peter Wood <peter@peterwood.dev>
**Features:**
- Archive integrity verification (checksum validation)
- Database integrity checking within backups
- Backup completeness validation
- Automated repair suggestions and fixes
- Historical backup analysis
- Performance metrics and reporting
**Usage:**
```bash
./validate-plex-backups.sh # Validate all backups
./validate-plex-backups.sh --report # Generate detailed report
./validate-plex-backups.sh --fix # Auto-fix issues where possible
./validate-plex-backups.sh # Validate all backups
./validate-plex-backups.sh --fix # Validate and fix issues
./validate-plex-backups.sh --report # Generate detailed report
./validate-plex-backups.sh --latest # Validate only latest backup
```
### Testing and Monitoring
#### `monitor-plex-backup.sh`
### Real-time backup system monitoring dashboard
**Author:** Peter Wood <peter@peterwood.dev>
**Features:**
- Real-time backup system health monitoring
- Performance metrics and trending
- Backup schedule and execution tracking
- Disk space monitoring and alerts
- Service status verification
- Watch mode with auto-refresh
**Usage:**
```bash
./monitor-plex-backup.sh # Single status check
./monitor-plex-backup.sh --watch # Continuous monitoring
./monitor-plex-backup.sh --help # Show help information
```
### 🛠️ Database Recovery Scripts
#### `recover-plex-database.sh`
### Advanced database recovery with multiple strategies
**Author:** Peter Wood <peter@peterwood.dev>
**Features:**
- Progressive recovery strategy (gentle to aggressive)
- Multiple repair techniques (VACUUM, dump/restore, rebuild)
- Automatic backup before recovery attempts
- Database integrity verification at each step
- Rollback capability if recovery fails
- Comprehensive logging and reporting
**Usage:**
```bash
./recover-plex-database.sh # Interactive recovery
./recover-plex-database.sh --auto # Automated recovery
./recover-plex-database.sh --dry-run # Show recovery plan
./recover-plex-database.sh --gentle # Gentle repair only
```
#### `icu-aware-recovery.sh`
### ICU-aware database recovery for Unicode issues
**Author:** Peter Wood <peter@peterwood.dev>
**Features:**
- ICU collation sequence detection and repair
- Unicode-aware database reconstruction
- Advanced SQLite recovery techniques
- Plex service management during recovery
**Usage:**
```bash
./icu-aware-recovery.sh # Interactive recovery
./icu-aware-recovery.sh --auto # Automated recovery
./icu-aware-recovery.sh --check-only # Check ICU status only
```
#### `nuclear-plex-recovery.sh`
### Last-resort complete database replacement
**Author:** Peter Wood <peter@peterwood.dev>
**⚠️ WARNING:** This script completely replaces existing databases!
**Features:**
- Complete database replacement from backups
- Automatic backup of current (corrupted) databases
- Rollback capability if replacement fails
- Verification of restored database integrity
**Usage:**
```bash
./nuclear-plex-recovery.sh # Interactive recovery
./nuclear-plex-recovery.sh --auto # Automated recovery
./nuclear-plex-recovery.sh --dry-run # Show what would be done
```
#### `validate-plex-recovery.sh`
### Recovery validation and verification
**Author:** Peter Wood <peter@peterwood.dev>
**Features:**
- Database integrity verification
- Service functionality testing
- Library accessibility checks
- Performance validation
- Web interface connectivity testing
**Usage:**
```bash
./validate-plex-recovery.sh # Full validation suite
./validate-plex-recovery.sh --quick # Quick validation checks
./validate-plex-recovery.sh --detailed # Detailed analysis and reporting
```
### 🧪 Testing Framework
#### `test-plex-backup.sh`
**Comprehensive testing framework**
### Comprehensive testing suite
- **Unit tests** for core functionality
- **Integration tests** for full system testing
- **Performance benchmarks**
**Author:** Peter Wood <peter@peterwood.dev>
**Features:**
- Unit testing for individual backup components
- Integration testing for full backup workflows
- Database integrity test scenarios
- Performance benchmarking
- Error condition simulation and recovery testing
**Usage:**
```bash
./test-plex-backup.sh all # Run all tests
./test-plex-backup.sh unit # Unit tests only
./test-plex-backup.sh performance # Performance benchmarks
./test-plex-backup.sh # Run full test suite
./test-plex-backup.sh --unit # Unit tests only
./test-plex-backup.sh --integration # Integration tests only
./test-plex-backup.sh --quick # Quick smoke tests
```
#### `integration-test-plex.sh`
**Integration testing for Plex backup system**
### End-to-end integration testing
- **End-to-end testing**
- **System integration validation**
- **Environment compatibility checks**
**Author:** Peter Wood <peter@peterwood.dev>
#### `monitor-plex-backup.sh`
**Features:**
**Real-time backup monitoring**
- Full workflow integration testing
- Isolated test environment creation
- Production-safe testing procedures
- Multi-scenario testing (normal, error, edge cases)
- Cross-script compatibility testing
- **Live backup status**
- **Performance metrics**
- **Error detection and alerting**
**Usage:**
### Utility Scripts
```bash
./integration-test-plex.sh # Full integration test suite
./integration-test-plex.sh --quick # Quick smoke tests
./integration-test-plex.sh --performance # Performance benchmarks
```
### 🎮 Management & Utilities
#### `plex.sh`
**Plex Media Server service management**
### Modern Plex service management
- **Service start/stop/restart**
- **Status monitoring**
- **Safe service management**
**Author:** Peter Wood <peter@peterwood.dev>
**Features:**
- Service start/stop/restart/status operations
- Web interface launcher
- Styled console output with Unicode symbols
- Service health monitoring
- Interactive menu system
**Usage:**
```bash
./plex.sh start # Start Plex service
./plex.sh stop # Stop Plex service
./plex.sh restart # Restart Plex service
./plex.sh status # Show service status
./plex.sh web # Open web interface
./plex.sh # Interactive menu
```
#### `plex-recent-additions.sh`
**Recent media additions reporting**
### Recent media additions reporting
- **New content detection**
- **Addition summaries**
- **Media library analytics**
**Author:** Peter Wood <peter@peterwood.dev>
## Configuration
**Features:**
### Environment Variables
- Recent additions reporting (configurable time range)
- Library section filtering
- Formatted output with headers and columns
- Direct SQLite database querying
Key configuration parameters in `backup-plex.sh`:
**Usage:**
```bash
# Retention settings
MAX_BACKUP_AGE_DAYS=30 # Remove backups older than 30 days
MAX_BACKUPS_TO_KEEP=10 # Keep maximum of 10 backup archives
# Directory settings
BACKUP_ROOT="/mnt/share/media/backups/plex"
LOG_ROOT="/mnt/share/media/backups/logs"
# Feature toggles
PARALLEL_VERIFICATION=true # Enable parallel verification
PERFORMANCE_MONITORING=true # Track performance metrics
AUTO_REPAIR=false # Automatic database repair
./plex-recent-additions.sh # Show additions from last 7 days
./plex-recent-additions.sh 30 # Show additions from last 30 days
```
### Backup Strategy
## 🏗️ System Architecture
The enhanced backup system implements:
### Script Relationships
- **Archive-only structure**: Direct `.tar.gz` storage
- **Timestamp naming**: `plex-backup-YYYYMMDD_HHMMSS.tar.gz`
- **Automatic cleanup**: Age and count-based retention
- **Integrity validation**: Comprehensive archive verification
## Directory Structure
```
/mnt/share/media/backups/plex/
├── plex-backup-20250125_143022.tar.gz # Latest backup
├── plex-backup-20250124_143011.tar.gz # Previous backup
├── plex-backup-20250123_143008.tar.gz # Older backup
└── logs/
├── backup_log_20250125_143022.md
└── plex-backup-performance.json
```mermaid
graph TD
A[backup-plex.sh] --> B[validate-plex-backups.sh]
A --> C[monitor-plex-backup.sh]
B --> D[restore-plex.sh]
D --> E[validate-plex-recovery.sh]
F[recover-plex-database.sh] --> E
G[icu-aware-recovery.sh] --> E
H[nuclear-plex-recovery.sh] --> E
I[test-plex-backup.sh] --> A
J[integration-test-plex.sh] --> A
K[plex.sh] --> A
L[plex-recent-additions.sh] --> A
```
## Enhanced Features
### Data Flow
### Performance Monitoring
1. **Backup Creation:** `backup-plex.sh` creates validated backups
2. **Monitoring:** `monitor-plex-backup.sh` tracks system health
3. **Validation:** `validate-plex-backups.sh` ensures backup integrity
4. **Recovery:** Multiple recovery scripts handle different failure scenarios
5. **Restoration:** `restore-plex.sh` safely restores from backups
6. **Verification:** `validate-plex-recovery.sh` confirms successful recovery
- **JSON performance logs**: All operations timed and logged
- **Performance reports**: Automatic generation of metrics
- **Operation tracking**: Backup, verification, service management times
## 🔧 Configuration
### Database Management
### Environment Setup
- **Integrity checking**: Comprehensive SQLite database validation
- **Automated repair**: Optional auto-repair of corruption
- **WAL file handling**: Proper SQLite Write-Ahead Logging management
All scripts share common configuration patterns:
### Notification System
- **Backup Location:** `/mnt/share/media/backups/plex`
- **Log Location:** `./logs/` (local) and `/mnt/share/media/backups/logs` (shared)
- **Plex Database Path:** `/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/`
- **Service Name:** `plexmediaserver`
- **Console output**: Color-coded status messages
- **Webhook notifications**: Custom webhook URL support
- **Email notifications**: SMTP-based email alerts
- **Default webhook**: Automatic notifications to configured endpoint
### Notification Configuration
### Safety Features
Scripts support multiple notification channels:
- **Pre-flight checks**: Disk space and system validation
- **Service management**: Safe Plex service start/stop
- **Backup verification**: Checksum and integrity validation
- **Error handling**: Comprehensive error detection and recovery
- **Webhook notifications:** Custom webhook URL support
- **Email notifications:** Via sendmail (if configured)
- **Console output:** Color-coded status messages
## Automation and Scheduling
### Performance Tuning
### Cron Integration
- **Parallel verification:** Enabled by default for faster operations
- **Performance monitoring:** JSON-based metrics collection
- **Automatic cleanup:** Configurable retention policies
Example crontab entries for automated operations:
## 📊 Monitoring & Alerting
```bash
# Daily Plex backup at 04:15
15 4 * * * /home/acedanger/shell/plex/backup-plex.sh --non-interactive --auto-repair 2>&1 | logger -t plex-backup -p user.info
### Health Checks
# Daily validation at 07:00
0 7 * * * /home/acedanger/shell/plex/validate-plex-backups.sh --fix 2>&1 | logger -t plex-validation -p user.info
```
The monitoring system tracks:
### Log Monitoring
- Backup success/failure rates
- Database integrity status
- Service uptime and performance
- Disk space utilization
- Recovery operation success
Monitor backup operations with:
### Performance Metrics
```bash
# Real-time monitoring
sudo journalctl -f -t plex-backup -t plex-validation
- Backup duration and size trends
- Database operation performance
- Service start/stop times
- Recovery operation benchmarks
# Historical analysis
sudo journalctl --since '24 hours ago' -t plex-backup
## 🚨 Emergency Procedures
# Performance analysis
jq '.[] | select(.operation == "backup") | .duration_seconds' logs/plex-backup-performance.json
```
### Database Corruption
## Troubleshooting
1. **First Response:** Run `backup-plex.sh --check-integrity`
2. **Gentle Recovery:** Try `recover-plex-database.sh --gentle`
3. **Advanced Recovery:** Use `icu-aware-recovery.sh` for Unicode issues
4. **Last Resort:** Execute `nuclear-plex-recovery.sh` with known good backup
5. **Validation:** Always run `validate-plex-recovery.sh` after recovery
### Common Issues
### Service Issues
1. **Database corruption**: Use `--auto-repair` flag or manual repair
2. **Insufficient disk space**: Check space requirements (2x backup size)
3. **Service management**: Ensure Plex service accessibility
4. **Archive validation**: Use validation script for integrity checks
1. **Check Status:** `./plex.sh status`
2. **Restart Service:** `./plex.sh restart`
3. **Monitor Logs:** Check system logs and script logs
4. **Validate Database:** Run integrity checks if service fails to start
### Debug Mode
## 📚 Additional Documentation
Enable verbose logging:
- **[Plex Backup System Guide](plex-backup.md)** - Detailed backup system documentation
- **[Plex Management Guide](plex-management.md)** - Service management procedures
- **[Troubleshooting Guide](troubleshooting.md)** - Common issues and solutions
```bash
# Add environment variable for debug output
PLEX_DEBUG=true ./backup-plex.sh
```
## 🏷️ Version Information
### Log Analysis
- **Script Suite Version:** 2.0
- **Author:** Peter Wood <peter@peterwood.dev>
- **Last Updated:** June 2025
- **Compatibility:** Ubuntu 20.04+, Debian 11+
- **Plex Version:** Compatible with Plex Media Server 1.25+
```bash
# Check backup success rate
grep "SUCCESS" logs/plex-backup-*.log | wc -l
## 📞 Support
# Analyze errors
grep "ERROR" logs/plex-backup-*.log | tail -10
For issues, questions, or contributions:
# Performance trends
jq '[.[] | select(.operation == "backup") | .duration_seconds] | add/length' logs/plex-backup-performance.json
```
## Security Considerations
### File Permissions
- Backup files created with appropriate permissions
- Sensitive files maintain original ownership
- Temporary files properly cleaned up
### Access Control
- Script requires appropriate sudo permissions
- Backup locations should have restricted access
- Log files contain operational data only
### Network Security
- Webhook notifications use HTTPS when possible
- No sensitive data included in notifications
- Email notifications respect system configuration
## Documentation
### Detailed Documentation
- **[plex-backup.md](./plex-backup.md)**: Comprehensive backup script documentation
- **[plex-management.md](./plex-management.md)**: Plex management and administration guide
### Integration Notes
- All scripts follow repository coding standards
- Consistent logging and error handling
- Color-coded output for readability
- Comprehensive help systems
## Migration Notes
When migrating from legacy backup scripts:
1. **Backup current configuration**: Save any custom modifications
2. **Test new scripts**: Run with `--check-integrity` first
3. **Update automation**: Modify cron jobs to use new options
4. **Monitor performance**: Check performance logs for optimization
The enhanced scripts maintain backward compatibility while adding significant new capabilities.
---
*For additional support and advanced configuration options, refer to the detailed documentation files in this directory.*
- **Author:** Peter Wood
- **Email:** <peter@peterwood.dev>
- **Repository:** Part of comprehensive shell script collection