peterwood/shell
1
0
Fork 0
mirror of https://github.com/acedanger/shell.git synced 2025-12-06 06:40:13 -08:00
Code Issues 19 Packages Projects Releases Wiki Activity

feat: Add enhanced backup-media script and documentation

Browse Source 
- Introduced demo-enhanced-backup.sh to showcase new features.
- Created backup-media-enhancement-summary.md for a side-by-side comparison of original and enhanced scripts.
- Developed enhanced-media-backup.md detailing features, usage, configuration, and error handling of the new backup script.
- Enhanced logging, error handling, and performance monitoring capabilities.
- Added support for multiple media services with improved safety and maintenance features.
This commit is contained in:
Peter Wood
2025-05-25 23:35:47 -04:00
parent 7e2bfd451b
commit b8a3c98297
5 changed files with 1448 additions and 75 deletions
Download Patch File Download Diff File Expand all files Collapse all files

195
README.md
View File

@@ -4,114 +4,229 @@ This repository contains various shell scripts for managing media-related tasks
## Available Scripts
- [Backup Media Script](docs/backup-media.md): Documentation for the `backup-media.sh` script.
- `plex.sh`: Script to manage the Plex Media Server (start, stop, restart, status).
- `backup-plex.sh`: Enhanced Plex backup script with integrity verification, incremental backups, and advanced features.
- `restore-plex.sh`: Script to restore Plex data from backups with safety checks.
- `validate-plex-backups.sh`: Script to validate backup integrity and monitor backup health.
- `folder-metrics.sh`: Script to calculate disk usage and file count for a directory and its subdirectories.
### Backup Scripts
## Enhanced Plex Backup System
- **`backup-media.sh`**: Enterprise-grade media backup script with parallel processing, comprehensive logging, and verification features.
- **`backup-plex.sh`**: Enhanced Plex backup script with integrity verification, incremental backups, and advanced features.
- **`restore-plex.sh`**: Script to restore Plex data from backups with safety checks.
- **`validate-plex-backups.sh`**: Script to validate backup integrity and monitor backup health.
This repository includes an enhanced backup system for Plex Media Server with multiple components:
### Management Scripts
### Scripts
- **`plex.sh`**: Script to manage the Plex Media Server (start, stop, restart, status).
- **`folder-metrics.sh`**: Script to calculate disk usage and file count for a directory and its subdirectories.
### Testing Scripts
- **`test-setup.sh`**: Validates the bootstrap and setup process.
- **`run-docker-tests.sh`**: Runner script that executes tests in Docker containers.
## Enhanced Media Backup System
This repository includes enterprise-grade backup solutions for both general media files and Plex Media Server with comprehensive features for reliability, performance, and monitoring.
### Media Backup Script (`backup-media.sh`)
The enhanced media backup script provides enterprise-grade features for backing up large media collections:
#### Key Features
- **Parallel Processing**: Multi-threaded operations with configurable worker pools
- **Comprehensive Logging**: Multiple formats (text, JSON, markdown) with detailed metrics
- **Backup Verification**: SHA-256 checksum validation and integrity checks
- **Performance Monitoring**: Real-time progress tracking and transfer statistics
- **Automatic Cleanup**: Configurable retention policies with space management
- **Smart Notifications**: Detailed completion reports with statistics
- **Safety Features**: Dry-run mode, pre-flight checks, and graceful error handling
- **Interactive Mode**: Manual control with real-time feedback
#### Usage Examples
```bash
# Standard parallel backup (recommended)
./backup-media.sh
# Sequential backup for better compatibility
./backup-media.sh --sequential
# Test run without making changes
./backup-media.sh --dry-run
# Interactive mode with manual control
./backup-media.sh --interactive
# Verbose logging with performance metrics
./backup-media.sh --verbose
# Custom source and destination
./backup-media.sh --source /path/to/media --destination /path/to/backup
```
### Configuration Options
The script includes configurable parameters:
- `PARALLEL_JOBS=4`: Number of parallel rsync processes
- `MAX_BACKUP_AGE_DAYS=90`: Retention period for old backups
- `BACKUP_ROOT`: Default backup destination
- `LOG_ROOT`: Location for backup logs
### Performance Features
- **Progress Tracking**: Real-time file transfer progress
- **Transfer Statistics**: Bandwidth, file counts, and timing metrics
- **Resource Monitoring**: CPU and memory usage tracking
- **Optimization**: Intelligent file handling and compression options
### Advanced Plex Backup System
Specialized backup system for Plex Media Server with database-aware features:
### Components
- **`backup-plex.sh`**: Advanced backup script with integrity verification, incremental backups, and automatic cleanup
- **`restore-plex.sh`**: Safe restoration script with dry-run mode and current data backup
- **`validate-plex-backups.sh`**: Backup validation and health monitoring script
### Key Features
### Plex-Specific Features
- **Incremental backups**: Only backs up files that have changed since last backup
- **File integrity verification**: Uses MD5 checksums to verify backup integrity
- **Database integrity verification**: Uses MD5 checksums to verify backup integrity
- **Automatic cleanup**: Configurable retention policies for old backups
- **Disk space monitoring**: Checks available space before starting backup
- **Safe restoration**: Backs up current data before restoring from backup
- **Comprehensive logging**: Detailed logs with color-coded output
- **Service management**: Safely stops/starts Plex during backup operations
### Usage Examples
## Backup Usage Examples
#### Enhanced Backup Script
### Media Backup Operations
```bash
# Run the enhanced backup (recommended)
./backup-plex.sh
# Quick media backup with default settings
./backup-media.sh
# High-performance parallel backup
./backup-media.sh --parallel --workers 8
# Test backup strategy without making changes
./backup-media.sh --dry-run --verbose
# Custom backup with specific paths
./backup-media.sh --source /mnt/movies --destination /backup/movies
```
#### Backup Validation
### Advanced Plex Operations
```bash
# Run enhanced Plex backup (recommended)
./backup-plex.sh
# Validate all backups and generate report
./validate-plex-backups.sh --report
# Validate backups and attempt to fix common issues
./validate-plex-backups.sh --fix
# Quick validation check
./validate-plex-backups.sh
```
#### Restore from Backup
```bash
# List available backups
./restore-plex.sh
# Test restore without making changes (dry run)
./restore-plex.sh 20250125 --dry-run
# Actually restore from a specific backup
# Restore from specific backup
./restore-plex.sh 20250125
```
### Automation Examples
## Automation and Scheduling
#### Daily Backup with Validation
### Daily Media Backup
```bash
# Add to crontab for daily backup at 3 AM
# Add to crontab for daily media backup at 2 AM
0 2 * * * /home/acedanger/shell/backup-media.sh --parallel
# Alternative: Sequential backup for systems with limited resources
0 2 * * * /home/acedanger/shell/backup-media.sh --sequential
```
### Automated Plex Backup with Validation
### Daily Plex Backup with Validation
```bash
# Add to crontab for daily Plex backup at 3 AM
0 3 * * * /home/acedanger/shell/backup-plex.sh
# Add daily validation at 7 AM
0 7 * * * /home/acedanger/shell/validate-plex-backups.sh --fix
```
#### Weekly Full Validation Report
### Weekly Comprehensive Validation Report
```bash
# Generate detailed weekly report (Sundays at 8 AM)
0 8 * * 0 /home/acedanger/shell/validate-plex-backups.sh --report
```
### Configuration
## Backup Configuration and Strategy
The enhanced backup script includes configurable parameters at the top of the file:
### Media Backup Configuration
The enhanced media backup script includes configurable parameters at the top of the file:
- `PARALLEL_JOBS=4`: Number of parallel rsync processes
- `MAX_BACKUP_AGE_DAYS=90`: Remove backups older than 90 days
- `LOG_RETENTION_DAYS=30`: Keep logs for 30 days
- `BACKUP_ROOT`: Default location for backup storage
- `LOG_ROOT`: Location for backup logs and reports
### Plex Backup Configuration
The Plex backup script configuration parameters:
- `MAX_BACKUP_AGE_DAYS=30`: Remove backups older than 30 days
- `MAX_BACKUPS_TO_KEEP=10`: Keep maximum of 10 backup sets
- `BACKUP_ROOT`: Location for backup storage
- `LOG_ROOT`: Location for backup logs
### Backup Strategy
### Recommended Backup Strategy
The system implements a robust 3-2-1 backup strategy:
Both systems implement a robust backup strategy following industry best practices:
1. **3 copies**: Original data + local backup + compressed archive
2. **2 different media**: Local disk + network storage
3. **1 offsite**: Consider syncing to remote location
**For Media Files:**
For offsite backup, add to cron:
1. **Daily incremental backups** with parallel processing for speed
2. **Weekly verification** of backup integrity
3. **Monthly cleanup** of old backups based on retention policies
4. **Quarterly offsite sync** for disaster recovery
**For Plex Database:**
1. **Daily full backups** with service-aware operations
2. **Immediate validation** after each backup
3. **Weekly comprehensive reports** on backup health
4. **Monthly testing** of restore procedures
### Offsite Backup Integration
For comprehensive disaster recovery, sync backups to remote locations:
```bash
# Sync backups to remote server daily at 6 AM
# Sync media backups to remote server daily at 5 AM
0 5 * * * rsync -av /mnt/share/media/backups/media/ user@remote-server:/backups/media/
# Sync Plex backups to remote server daily at 6 AM
0 6 * * * rsync -av /mnt/share/media/backups/plex/ user@remote-server:/backups/plex/
```
## Documentation
### Backup System Documentation
- [Enhanced Media Backup Documentation](./docs/enhanced-media-backup.md): Comprehensive guide for the enhanced `backup-media.sh` script with enterprise features.
- [Media Backup Enhancement Summary](./docs/backup-media-enhancement-summary.md): Summary of enhancements and feature comparisons.
- [Plex Backup Script Documentation](./docs/plex-backup.md): Detailed documentation for the `backup-plex.sh` script.
### Script Documentation
- [Plex Management Script Documentation](./docs/plex-management.md): Detailed documentation for the `plex.sh` script.
- [Folder Metrics Script Documentation](./docs/folder-metrics.md): Detailed documentation for the `folder-metrics.sh` script.
- [Testing Framework Documentation](./docs/testing.md): Detailed documentation for the Docker-based testing system.