mirror of
https://github.com/acedanger/shell.git
synced 2025-12-06 04:30:13 -08:00
Add monitoring dashboard for Plex backup system
- Created a new script `monitor-plex-backup.sh` for real-time status and health monitoring of the Plex backup system. - Implemented features to check system status, backup status, performance metrics, recent activity, scheduling status, and health recommendations. - Added command line options for watch mode and refresh interval. - Enhanced logging with color-coded output for better visibility. Update recent additions script to include more metadata - Modified `plex-recent-additions.sh` to retrieve additional metadata fields such as year and library section type from the Plex database. - Improved SQL query to join with library_sections for better context on added items. Introduce comprehensive test suite for Plex backup system - Added `test-plex-backup.sh` to provide automated testing for all backup-related functionality. - Implemented unit tests for JSON log initialization, performance tracking, notification system, checksum caching, backup verification, and more. - Included setup and cleanup functions for a mock test environment. - Added performance benchmarks for checksum calculation and compression. - Generated detailed test reports in JSON format for better tracking of test results.
This commit is contained in:
Peter Wood
334
docs/enhanced-plex-backup.md
Normal file
334
docs/enhanced-plex-backup.md
Normal file
@@ -0,0 +1,334 @@
|
||||
# Enhanced Plex Backup Script Documentation
|
||||
|
||||
This document provides comprehensive documentation for the enhanced `backup-plex.sh` script. This advanced backup solution includes performance monitoring, parallel processing, intelligent notifications, WAL file handling, and automated testing capabilities.
|
||||
|
||||
## Script Overview
|
||||
|
||||
The enhanced script performs the following advanced tasks:
|
||||
|
||||
1. **Performance Monitoring**: Tracks backup operations with JSON-based performance logging
|
||||
2. **Intelligent Backup Detection**: Only backs up files that have changed since last backup
|
||||
3. **WAL File Handling**: Properly handles SQLite Write-Ahead Logging files
|
||||
4. **Database Integrity Verification**: Comprehensive integrity checks with automated repair options
|
||||
5. **Parallel Processing**: Concurrent verification for improved performance
|
||||
6. **Multi-Channel Notifications**: Console, webhook, and email notification support
|
||||
7. **Checksum Caching**: Intelligent caching to avoid recalculating unchanged file checksums
|
||||
8. **Enhanced Service Management**: Safe Plex service management with progress indicators
|
||||
9. **Comprehensive Logging**: Detailed logs with color-coded output and timestamps
|
||||
10. **Automated Cleanup**: Configurable retention policies for old backups
|
||||
|
||||
## Enhanced Features
|
||||
|
||||
### Performance Tracking
|
||||
|
||||
- **JSON Performance Logs**: All operations are timed and logged to `logs/plex-backup-performance.json`
|
||||
- **Performance Reports**: Automatic generation of average performance metrics
|
||||
- **Operation Monitoring**: Tracks backup, verification, service management, and overall script execution times
|
||||
|
||||
### Notification System
|
||||
|
||||
The script supports multiple notification channels:
|
||||
|
||||
#### Console Notifications
|
||||
|
||||
- Color-coded status messages (Success: Green, Error: Red, Warning: Yellow, Info: Blue)
|
||||
- Timestamped log entries with clear formatting
|
||||
|
||||
#### Webhook Notifications
|
||||
|
||||
```bash
|
||||
./backup-plex.sh --webhook=https://your-webhook-url.com/endpoint
|
||||
```
|
||||
|
||||
Sends JSON payloads with backup status, hostname, and timestamps.
|
||||
|
||||
#### Email Notifications
|
||||
|
||||
```bash
|
||||
./backup-plex.sh --email=admin@example.com
|
||||
```
|
||||
|
||||
Requires `sendmail` to be configured on the system.
|
||||
|
||||
### WAL File Management
|
||||
|
||||
The script now properly handles SQLite Write-Ahead Logging files:
|
||||
|
||||
- **Automatic Detection**: Identifies and backs up `.db-wal` and `.db-shm` files when present
|
||||
- **WAL Checkpointing**: Performs `PRAGMA wal_checkpoint(FULL)` before integrity checks
|
||||
- **Safe Backup**: Ensures WAL files are properly backed up alongside main database files
|
||||
|
||||
### Database Integrity & Repair
|
||||
|
||||
Enhanced database management features:
|
||||
|
||||
- **Pre-backup Integrity Checks**: Verifies database health before backup operations
|
||||
- **Automated Repair**: Optional automatic repair of corrupted databases using advanced techniques
|
||||
- **Interactive Repair Mode**: Prompts for repair decisions when issues are detected
|
||||
- **Post-repair Verification**: Re-checks integrity after repair operations
|
||||
|
||||
### Parallel Processing
|
||||
|
||||
- **Concurrent Verification**: Parallel backup verification for improved performance
|
||||
- **Fallback Safety**: Automatically falls back to sequential processing if parallel mode fails
|
||||
- **Configurable**: Can be disabled with `--no-parallel` for maximum safety
|
||||
|
||||
## Command Line Options
|
||||
|
||||
```bash
|
||||
Usage: ./backup-plex.sh [OPTIONS]
|
||||
|
||||
Options:
|
||||
--auto-repair Automatically attempt to repair corrupted databases
|
||||
--check-integrity Only check database integrity, don't backup
|
||||
--non-interactive Run in non-interactive mode (for automation)
|
||||
--no-parallel Disable parallel verification (slower but safer)
|
||||
--no-performance Disable performance monitoring
|
||||
--webhook=URL Send notifications to webhook URL
|
||||
--email=ADDRESS Send notifications to email address
|
||||
-h, --help Show help message
|
||||
```
|
||||
|
||||
## Configuration Files
|
||||
|
||||
### Performance Log Format
|
||||
|
||||
The performance log (`logs/plex-backup-performance.json`) contains entries like:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"operation": "backup",
|
||||
"duration_seconds": 45.3,
|
||||
"timestamp": "2025-05-25T19:45:23-05:00"
|
||||
},
|
||||
{
|
||||
"operation": "verification",
|
||||
"duration_seconds": 12.8,
|
||||
"timestamp": "2025-05-25T19:46:08-05:00"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
### Backup Tracking Log
|
||||
|
||||
The backup tracking log (`logs/plex-backup.json`) tracks last backup times:
|
||||
|
||||
```json
|
||||
{
|
||||
"/var/lib/plexmediaserver/.../com.plexapp.plugins.library.db": 1732567523,
|
||||
"/var/lib/plexmediaserver/.../Preferences.xml": 1732567523
|
||||
}
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Basic Backup
|
||||
|
||||
```bash
|
||||
./backup-plex.sh
|
||||
```
|
||||
|
||||
Performs a standard backup with all enhanced features enabled.
|
||||
|
||||
### Integrity Check Only
|
||||
|
||||
```bash
|
||||
./backup-plex.sh --check-integrity
|
||||
```
|
||||
|
||||
Only checks database integrity without performing backup.
|
||||
|
||||
### Automated Backup with Notifications
|
||||
|
||||
```bash
|
||||
./backup-plex.sh --non-interactive --auto-repair --webhook=https://notify.example.com/backup
|
||||
```
|
||||
|
||||
Runs in automated mode with auto-repair and webhook notifications.
|
||||
|
||||
### Performance-Optimized Backup
|
||||
|
||||
```bash
|
||||
./backup-plex.sh --no-parallel --no-performance
|
||||
```
|
||||
|
||||
Runs with parallel processing and performance monitoring disabled for maximum compatibility.
|
||||
|
||||
## Automation and Scheduling
|
||||
|
||||
### Cron Job Setup
|
||||
|
||||
For daily automated backups at 2 AM:
|
||||
|
||||
```bash
|
||||
# Edit crontab
|
||||
crontab -e
|
||||
|
||||
# Add this line for daily backup
|
||||
0 2 * * * /home/acedanger/shell/backup-plex.sh --non-interactive --auto-repair --email=admin@example.com 2>&1 | logger -t plex-backup
|
||||
```
|
||||
|
||||
### Systemd Service
|
||||
|
||||
Create a systemd service for more control:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Plex Backup Service
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=root
|
||||
ExecStart=/home/acedanger/shell/backup-plex.sh --non-interactive --auto-repair
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
### Systemd Timer
|
||||
|
||||
Create a timer for regular execution:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Daily Plex Backup
|
||||
Requires=plex-backup.service
|
||||
|
||||
[Timer]
|
||||
OnCalendar=daily
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
```
|
||||
|
||||
## Monitoring and Alerts
|
||||
|
||||
### Performance Monitoring
|
||||
|
||||
The script automatically tracks:
|
||||
|
||||
- Backup operation duration
|
||||
- Verification times
|
||||
- Service start/stop times
|
||||
- Overall script execution time
|
||||
|
||||
### Health Checks
|
||||
|
||||
Regular health monitoring can be implemented by checking:
|
||||
|
||||
```bash
|
||||
# Check last backup success
|
||||
jq -r '.[-1] | select(.operation == "total_script") | .timestamp' logs/plex-backup-performance.json
|
||||
|
||||
# Check average backup performance
|
||||
jq '[.[] | select(.operation == "backup") | .duration_seconds] | add/length' logs/plex-backup-performance.json
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Permission Denied Errors**
|
||||
- Ensure script runs with appropriate sudo permissions
|
||||
- Check Plex file ownership and permissions
|
||||
|
||||
2. **WAL File Warnings**
|
||||
- Now handled automatically by the enhanced script
|
||||
- WAL checkpointing ensures data consistency
|
||||
|
||||
3. **Performance Issues**
|
||||
- Use `--no-parallel` if concurrent operations cause problems
|
||||
- Monitor performance logs for bottlenecks
|
||||
|
||||
4. **Notification Failures**
|
||||
- Verify webhook URLs are accessible
|
||||
- Check sendmail configuration for email notifications
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Enable verbose logging by modifying the script or using:
|
||||
|
||||
```bash
|
||||
bash -x ./backup-plex.sh --check-integrity
|
||||
```
|
||||
|
||||
## Testing Framework
|
||||
|
||||
The script includes a comprehensive testing framework (`test-plex-backup.sh`):
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
./test-plex-backup.sh all
|
||||
|
||||
# Run only unit tests
|
||||
./test-plex-backup.sh unit
|
||||
|
||||
# Run performance benchmarks
|
||||
./test-plex-backup.sh performance
|
||||
```
|
||||
|
||||
### Test Categories
|
||||
|
||||
- **Unit Tests**: Core functionality verification
|
||||
- **Integration Tests**: Full system testing (requires Plex installation)
|
||||
- **Performance Tests**: Benchmarking and performance validation
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### File Permissions
|
||||
|
||||
- Backup files are created with appropriate permissions
|
||||
- Sensitive files maintain original ownership and permissions
|
||||
- Temporary files are properly cleaned up
|
||||
|
||||
### Network Security
|
||||
|
||||
- Webhook notifications use HTTPS when possible
|
||||
- Email notifications respect system sendmail configuration
|
||||
- No sensitive data is included in notifications
|
||||
|
||||
### Access Control
|
||||
|
||||
- Script requires appropriate sudo permissions
|
||||
- Backup locations should have restricted access
|
||||
- Log files contain operational data, not sensitive information
|
||||
|
||||
## Backup Strategy
|
||||
|
||||
The enhanced script implements a robust backup strategy:
|
||||
|
||||
### 3-2-1 Backup Rule
|
||||
|
||||
1. **3 Copies**: Original data + local backup + compressed archive
|
||||
2. **2 Different Media**: Local disk + network storage capability
|
||||
3. **1 Offsite**: Ready for remote synchronization
|
||||
|
||||
### Retention Policy
|
||||
|
||||
- Configurable maximum backup age (default: 30 days)
|
||||
- Configurable maximum backup count (default: 10 backups)
|
||||
- Automatic cleanup of old backups
|
||||
|
||||
### Verification Strategy
|
||||
|
||||
- Checksum verification for all backed up files
|
||||
- Database integrity checks before and after operations
|
||||
- Optional parallel verification for improved performance
|
||||
|
||||
## Migration from Legacy Script
|
||||
|
||||
To migrate from the original backup script:
|
||||
|
||||
1. **Backup Current Configuration**: Save any custom modifications
|
||||
2. **Test New Script**: Run with `--check-integrity` first
|
||||
3. **Update Automation**: Modify cron jobs to use new options
|
||||
4. **Monitor Performance**: Check performance logs for optimization opportunities
|
||||
|
||||
The enhanced script maintains backward compatibility while adding significant new capabilities.
|
||||
Reference in New Issue
Block a user