peterwood/shell
1
0
Fork 0
mirror of https://github.com/acedanger/shell.git synced 2025-12-06 07:50:11 -08:00
Code Issues 19 Packages Projects Releases Wiki Activity

Refactor documentation and enhance Immich backup system

Browse Source 
- Updated README.md to streamline content and improve navigation with a new Quick Navigation section.
- Consolidated detailed Plex backup descriptions into dedicated documentation files.
- Added comprehensive Docker Bootstrap Testing Framework documentation.
- Created Immich backup enhancement summary and migration summary documents.
- Implemented webhook notifications and Backblaze B2 integration in Immich backup scripts.
- Centralized logging for Immich backup operations and updated configuration requirements.
- Restructured Telegram bot project documentation for better clarity and organization.
- Enhanced .gitignore to include environment files and Backblaze CLI tool.
- Updated dotfiles README to reference new testing documentation.

Resolves Documentation review #11.
This commit is contained in:
Peter Wood
2025-05-27 12:51:00 -04:00
parent 3da0e07548
commit 2540c2d50b
11 changed files with 892 additions and 194 deletions
Download Patch File Download Diff File Expand all files Collapse all files

237
README.md
View File

@@ -2,6 +2,14 @@
This repository contains various shell scripts for managing media-related tasks and dotfiles for system configuration.
## Quick Navigation
- **[Backup Scripts](#backup-scripts)** - Enterprise-grade backup solutions
- **[Management Scripts](#management-scripts)** - System and service management
- **[Documentation](#documentation)** - Complete guides and references
- **[Testing](#testing)** - Docker-based validation framework
- **[Dotfiles](#dotfiles)** - System configuration files
## Available Scripts
### Backup Scripts
@@ -16,6 +24,10 @@ This repository contains various shell scripts for managing media-related tasks
- **`plex.sh`**: Script to manage the Plex Media Server (start, stop, restart, status).
- **`folder-metrics.sh`**: Script to calculate disk usage and file count for a directory and its subdirectories.
### Development Projects
- **[Telegram Backup Monitoring Bot](./telegram/github-issues/README.md)**: Comprehensive Telegram bot project for monitoring and managing all backup systems with real-time notifications and control capabilities.
### Testing Scripts
- **`test-setup.sh`**: Validates the bootstrap and setup process.
@@ -80,23 +92,7 @@ The script includes configurable parameters:
### Advanced Plex Backup System
Specialized backup system for Plex Media Server with database-aware features:
### Components
- **`backup-plex.sh`**: Advanced backup script with integrity verification, incremental backups, and automatic cleanup
- **`restore-plex.sh`**: Safe restoration script with dry-run mode and current data backup
- **`validate-plex-backups.sh`**: Backup validation and health monitoring script
### Plex-Specific Features
- **Incremental backups**: Only backs up files that have changed since last backup
- **Database integrity verification**: Uses MD5 checksums to verify backup integrity
- **Automatic cleanup**: Configurable retention policies for old backups
- **Disk space monitoring**: Checks available space before starting backup
- **Safe restoration**: Backs up current data before restoring from backup
- **Comprehensive logging**: Detailed logs with color-coded output
- **Service management**: Safely stops/starts Plex during backup operations
Specialized backup system for Plex Media Server with database-aware features. For complete documentation, see [Plex Scripts Documentation](./plex/README.md).
## Backup Usage Examples
@@ -168,93 +164,32 @@ Specialized backup system for Plex Media Server with database-aware features:
## Backup Configuration and Strategy
### Media Backup Configuration
For detailed configuration information and backup strategies, see:
The enhanced media backup script includes configurable parameters at the top of the file:
- `PARALLEL_JOBS=4`: Number of parallel rsync processes
- `MAX_BACKUP_AGE_DAYS=90`: Remove backups older than 90 days
- `LOG_RETENTION_DAYS=30`: Keep logs for 30 days
- `BACKUP_ROOT`: Default location for backup storage
- `LOG_ROOT`: Location for backup logs and reports
### Plex Backup Configuration
The Plex backup script configuration parameters:
- `MAX_BACKUP_AGE_DAYS=30`: Remove backups older than 30 days
- `MAX_BACKUPS_TO_KEEP=10`: Keep maximum of 10 backup archives
- `BACKUP_ROOT`: Location for compressed backup archives
- `LOG_ROOT`: Location for backup logs
### Final Backup Directory Structure
The enhanced Plex backup system creates a streamlined archive-only structure:
```bash
/mnt/share/media/backups/plex/
├── plex-backup-20250125_143022.tar.gz # Latest backup
├── plex-backup-20250124_143011.tar.gz # Previous backup
├── plex-backup-20250123_143008.tar.gz # Older backup
└── logs/
├── backup_log_20250125_143022.md
├── plex-backup-performance.json
└── plex-backup.json
```
**Key Benefits:**
- **Direct Archive Storage**: No intermediate directories required
- **Efficient Space Usage**: Only compressed files stored permanently
- **Easy Management**: Timestamp-based naming for clear identification
- **Automatic Cleanup**: Legacy dated directories removed automatically
### Recommended Backup Strategy
Both systems implement a robust backup strategy following industry best practices:
**For Media Files:**
1. **Daily incremental backups** with parallel processing for speed
2. **Weekly verification** of backup integrity
3. **Monthly cleanup** of old backups based on retention policies
4. **Quarterly offsite sync** for disaster recovery
**For Plex Database:**
1. **Daily full backups** with service-aware operations
2. **Immediate validation** after each backup
3. **Weekly comprehensive reports** on backup health
4. **Monthly testing** of restore procedures
### Offsite Backup Integration
For comprehensive disaster recovery, sync backups to remote locations:
```bash
# Sync media backups to remote server daily at 5 AM
0 5 * * * rsync -av /mnt/share/media/backups/media/ user@remote-server:/backups/media/
# Sync Plex backups to remote server daily at 6 AM
0 6 * * * rsync -av /mnt/share/media/backups/plex/ user@remote-server:/backups/plex/
```
- **[Plex Backup Configuration](./plex/README.md#configuration)**: Plex-specific settings and options
- **[Immich Backup Configuration](./immich/README.md#configuration)**: Database and upload directory settings
- **[Enhanced Media Backup](./docs/enhanced-media-backup.md)**: Media backup script configuration
- **[Crontab Management](./crontab/README.md)**: Automated scheduling and system-specific configurations
## Documentation
### Directory-Specific Documentation
### Component-Specific Documentation
- **[Plex Scripts Documentation](./plex/README.md)**: Comprehensive documentation for all Plex backup, restoration, validation, and management scripts.
- **[Crontab Management Documentation](./crontab/README.md)**: Complete guide for crontab management, backup systems, and multi-system administration.
- **[Plex Scripts Documentation](./plex/README.md)**: Comprehensive documentation for Plex backup, restoration, validation, and management scripts
- **[Immich Scripts Documentation](./immich/README.md)**: Complete guide for Immich backup and management system with PostgreSQL and B2 integration
- **[Crontab Management Documentation](./crontab/README.md)**: Complete guide for crontab management, backup systems, and multi-system administration
- **[Dotfiles Documentation](./dotfiles/README.md)**: System configuration files and setup instructions
### Backup System Documentation
### Technical Documentation
- [Enhanced Media Backup Documentation](./docs/enhanced-media-backup.md): Comprehensive guide for the enhanced `backup-media.sh` script with enterprise features.
- [Media Backup Enhancement Summary](./docs/backup-media-enhancement-summary.md): Summary of enhancements and feature comparisons.
### Script Documentation
- [Folder Metrics Script Documentation](./docs/folder-metrics.md): Detailed documentation for the `folder-metrics.sh` script.
- [Testing Framework Documentation](./docs/testing.md): Detailed documentation for the Docker-based testing system.
- **[Enhanced Media Backup Documentation](./docs/enhanced-media-backup.md)**: Comprehensive guide for the enhanced `backup-media.sh` script with enterprise features
- **[Basic Media Backup Documentation](./docs/backup-media.md)**: Documentation for the original `backup-media.sh` script
- **[Media Backup Enhancement Summary](./docs/backup-media-enhancement-summary.md)**: Summary of enhancements and feature comparisons
- **[Immich Backup Enhancement Summary](./docs/immich-backup-enhancement-summary.md)**: Details of Immich backup system improvements
- **[Docker Bootstrap Testing Framework](./docs/docker-bootstrap-testing-framework.md)**: Detailed documentation for the Docker-based bootstrap validation system
- **[Folder Metrics Script Documentation](./docs/folder-metrics.md)**: Detailed documentation for the `folder-metrics.sh` script
- **[Production Deployment Guide](./docs/production-deployment-guide.md)**: Complete deployment and operations guide
- **[Project Completion Summary](./docs/project-completion-summary.md)**: Overview of completed features and project milestones
## Dotfiles
@@ -268,59 +203,21 @@ For more information about the dotfiles, see [Dotfiles README](./dotfiles/README
## Testing
This repository includes Docker-based testing to validate the setup process across different environments:
This repository includes Docker-based testing to validate the setup process across different environments. For complete testing documentation, see [Docker Bootstrap Testing Framework](./docs/docker-bootstrap-testing-framework.md).
- **test-setup.sh**: Script that validates the bootstrap and setup process
- **run-docker-tests.sh**: Runner script that executes tests in Docker containers
- **Dockerfile**: Defines test environments (Ubuntu, Debian)
### Running Tests
Test your setup in isolated Docker containers with:
### Quick Testing
```bash
# Test in Ubuntu container
./run-docker-tests.sh ubuntu
./setup/run-docker-tests.sh ubuntu
# Test in Debian container
./run-docker-tests.sh debian
# Test in Debian container
./setup/run-docker-tests.sh debian
# Run full bootstrap test in Ubuntu
./run-docker-tests.sh full-ubuntu
# Run full bootstrap test in Debian
./run-docker-tests.sh full-debian
# Test in both Ubuntu and Debian
./run-docker-tests.sh all
# Test in both environments
./setup/run-docker-tests.sh all
```
#### When to Use Each Testing Option
- **Use `./run-docker-tests.sh ubuntu` or `./run-docker-tests.sh debian`** when you want to:
- Quickly validate if packages in packages.list are available and installed
- Test if your test-setup.sh script is working correctly
- Check for issues with specific components without performing a full setup
- **Use `./run-docker-tests.sh full-ubuntu` or `./run-docker-tests.sh full-debian`** when you want to:
- Test the complete bootstrap installation process end-to-end
- Validate that all installation steps work correctly on a fresh system
- Simulate what users will experience when running the bootstrap script
- **Use `./run-docker-tests.sh all`** when you want to:
- Ensure your test-setup.sh works across both Ubuntu and Debian
- Run comprehensive checks before committing changes
The test environment checks:
- Package availability and installation
- Core components (git, curl, wget, etc.)
- Additional packages from `setup/packages.list`
- Oh My Zsh and plugin installation
- Dotfile symlinks
Tests will continue even when some packages fail to install, reporting all issues in a comprehensive summary.
## plex.sh
This script is used to manage the Plex Media Server service on a systemd-based Linux distribution. It provides the following functionalities:
@@ -340,61 +237,21 @@ Note that these commands will run as `root`.
## Enhanced Crontab System Migration
### System Status
For complete crontab management documentation, see [Crontab Management Documentation](./crontab/README.md).
The crontab system has been migrated from a universal `crontab.txt` to system-specific files:
### Current Status
The crontab system uses hostname-specific configuration files:
- **crontab-europa.txt** - Media server configuration
- **crontab-io.txt** - Download/acquisition server configuration
- **crontab-racknerd.txt** - Backup server configuration
### Legacy File Status
The original `crontab.txt` has been:
- **Backed up** to `crontab.txt.bak`
- **Role**: Previously served as fallback, now obsolete
- **Content**: Contains mixed configuration that was split into system-specific files
### Current Management
Use the system-specific approach:
### Quick Management
```bash
# Install system-specific crontab
./manage-enhanced-crontab.sh install
# Install system-specific crontab (auto-detects hostname)
./crontab/manage-enhanced-crontab.sh install
# The script automatically detects hostname and uses appropriate file
# europa -> crontab-europa.txt
# io -> crontab-io.txt
# racknerd -> crontab-racknerd.txt
# View current system's crontab configuration
./crontab/manage-enhanced-crontab.sh show
```
### Fallback Behavior
If a system-specific file is missing, the management script will:
1. **Warning**: Display that system-specific file is not found
2. **Recommendation**: Create appropriate system-specific file
### Creating New System Files
For new systems, create system-specific files based on the templates:
```bash
# Copy and customize for new system
cp crontab-europa.txt crontab-newsystem.txt
# Edit for system-specific tasks
```
### Benefits of System-Specific Approach
- **Clear separation** of concerns between systems
- **Reduced conflicts** from universal configurations
- **System-appropriate** task scheduling
- **Better maintenance** and troubleshooting
- **Scalable** for additional systems
#### Migration Notes
Migration completed: May 26, 2025