peterwood/shell
1
0
Fork 0
mirror of https://github.com/acedanger/shell.git synced 2025-12-05 22:50:18 -08:00
Code Issues 19 Packages Projects Releases Wiki Activity

Commit local changes before merging with remote

Browse Source
This commit is contained in:
Peter Wood
2025-05-29 11:25:02 -04:00
parent 868b340fb5
commit be4f6a8d8c
75 changed files with 14107 additions and 562 deletions
Download Patch File Download Diff File Expand all files Collapse all files

310
README.md
View File

@@ -2,19 +2,195 @@
This repository contains various shell scripts for managing media-related tasks and dotfiles for system configuration.
## Quick Navigation
- **[Backup Scripts](#backup-scripts)** - Enterprise-grade backup solutions
- **[Management Scripts](#management-scripts)** - System and service management
- **[Tab Completion](#tab-completion)** - Intelligent command-line completion
- **[Documentation](#documentation)** - Complete guides and references
- **[Testing](#testing)** - Docker-based validation framework
- **[Dotfiles](#dotfiles)** - System configuration files
## Available Scripts
- [Backup Media Script](docs/backup-media.md): Documentation for the `backup-media.sh` script.
- `plex.sh`: Script to manage the Plex Media Server (start, stop, restart, status).
- `backup-plex.sh`: Script to back up Plex Media Server databases and related files.
- `folder-metrics.sh`: Script to calculate disk usage and file count for a directory and its subdirectories.
### Backup Scripts
- **`backup-media.sh`**: Enterprise-grade media backup script with parallel processing, comprehensive logging, and verification features.
- **`backup-plex.sh`**: Enhanced Plex backup script with integrity verification, incremental backups, and advanced features.
- **`restore-plex.sh`**: Script to restore Plex data from backups with safety checks.
- **`validate-plex-backups.sh`**: Script to validate backup integrity and monitor backup health.
### Management Scripts
- **`plex.sh`**: Script to manage the Plex Media Server (start, stop, restart, status).
- **`folder-metrics.sh`**: Script to calculate disk usage and file count for a directory and its subdirectories.
### Development Projects
- **[Telegram Backup Monitoring Bot](./telegram/github-issues/README.md)**: Comprehensive Telegram bot project for monitoring and managing all backup systems with real-time notifications and control capabilities.
### Testing Scripts
- **`test-setup.sh`**: Validates the bootstrap and setup process.
- **`run-docker-tests.sh`**: Runner script that executes tests in Docker containers.
## Enhanced Media Backup System
This repository includes enterprise-grade backup solutions for both general media files and Plex Media Server with comprehensive features for reliability, performance, and monitoring.
### Media Backup Script (`backup-media.sh`)
The enhanced media backup script provides enterprise-grade features for backing up large media collections:
#### Key Features
- **Parallel Processing**: Multi-threaded operations with configurable worker pools
- **Comprehensive Logging**: Multiple formats (text, JSON, markdown) with detailed metrics
- **Backup Verification**: SHA-256 checksum validation and integrity checks
- **Performance Monitoring**: Real-time progress tracking and transfer statistics
- **Automatic Cleanup**: Configurable retention policies with space management
- **Smart Notifications**: Detailed completion reports with statistics
- **Safety Features**: Dry-run mode, pre-flight checks, and graceful error handling
- **Interactive Mode**: Manual control with real-time feedback
#### Usage Examples
```bash
# Standard parallel backup (recommended)
./backup-media.sh
# Sequential backup for better compatibility
./backup-media.sh --sequential
# Test run without making changes
./backup-media.sh --dry-run
# Interactive mode with manual control
./backup-media.sh --interactive
# Verbose logging with performance metrics
./backup-media.sh --verbose
# Custom source and destination
./backup-media.sh --source /path/to/media --destination /path/to/backup
```
### Configuration Options
The script includes configurable parameters:
- `PARALLEL_JOBS=4`: Number of parallel rsync processes
- `MAX_BACKUP_AGE_DAYS=90`: Retention period for old backups
- `BACKUP_ROOT`: Default backup destination
- `LOG_ROOT`: Location for backup logs
### Performance Features
- **Progress Tracking**: Real-time file transfer progress
- **Transfer Statistics**: Bandwidth, file counts, and timing metrics
- **Resource Monitoring**: CPU and memory usage tracking
- **Optimization**: Intelligent file handling and compression options
### Advanced Plex Backup System
Specialized backup system for Plex Media Server with database-aware features. For complete documentation, see [Plex Scripts Documentation](./plex/README.md).
## Backup Usage Examples
### Media Backup Operations
```bash
# Quick media backup with default settings
./backup-media.sh
# High-performance parallel backup
./backup-media.sh --parallel --workers 8
# Test backup strategy without making changes
./backup-media.sh --dry-run --verbose
# Custom backup with specific paths
./backup-media.sh --source /mnt/movies --destination /backup/movies
```
### Advanced Plex Operations
```bash
# Run enhanced Plex backup (recommended)
./backup-plex.sh
# Validate all backups and generate report
./validate-plex-backups.sh --report
# Quick validation check
./validate-plex-backups.sh
# Test restore without making changes (dry run)
./restore-plex.sh plex-backup-20250125_143022.tar.gz --dry-run
# Restore from specific backup archive
./restore-plex.sh plex-backup-20250125_143022.tar.gz
```
## Automation and Scheduling
### Daily Media Backup
```bash
# Add to crontab for daily media backup at 2 AM
0 2 * * * /home/acedanger/shell/backup-media.sh --parallel
# Alternative: Sequential backup for systems with limited resources
0 2 * * * /home/acedanger/shell/backup-media.sh --sequential
```
### Automated Plex Backup with Validation
### Daily Plex Backup with Validation
```bash
# Add to crontab for daily Plex backup at 3 AM
0 3 * * * /home/acedanger/shell/backup-plex.sh
# Add daily validation at 7 AM
0 7 * * * /home/acedanger/shell/validate-plex-backups.sh --fix
```
### Weekly Comprehensive Validation Report
```bash
# Generate detailed weekly report (Sundays at 8 AM)
0 8 * * 0 /home/acedanger/shell/validate-plex-backups.sh --report
```
## Backup Configuration and Strategy
For detailed configuration information and backup strategies, see:
- **[Plex Backup Configuration](./plex/README.md#configuration)**: Plex-specific settings and options
- **[Immich Backup Configuration](./immich/README.md#configuration)**: Database and upload directory settings
- **[Enhanced Media Backup](./docs/enhanced-media-backup.md)**: Media backup script configuration
- **[Crontab Management](./crontab/README.md)**: Automated scheduling and system-specific configurations
## Documentation
- [Plex Backup Script Documentation](./docs/plex-backup.md): Detailed documentation for the `backup-plex.sh` script.
- [Plex Management Script Documentation](./docs/plex-management.md): Detailed documentation for the `plex.sh` script.
- [Folder Metrics Script Documentation](./docs/folder-metrics.md): Detailed documentation for the `folder-metrics.sh` script.
- [Testing Framework Documentation](./docs/testing.md): Detailed documentation for the Docker-based testing system.
### Component-Specific Documentation
- **[Plex Scripts Documentation](./plex/README.md)**: Comprehensive documentation for Plex backup, restoration, validation, and management scripts
- **[Immich Scripts Documentation](./immich/README.md)**: Complete guide for Immich backup and management system with PostgreSQL and B2 integration
- **[Crontab Management Documentation](./crontab/README.md)**: Complete guide for crontab management, backup systems, and multi-system administration
- **[Dotfiles Documentation](./dotfiles/README.md)**: System configuration files and setup instructions
### Technical Documentation
- **[Enhanced Media Backup Documentation](./docs/enhanced-media-backup.md)**: Comprehensive guide for the enhanced `backup-media.sh` script with enterprise features
- **[Basic Media Backup Documentation](./docs/backup-media.md)**: Documentation for the original `backup-media.sh` script
- **[Media Backup Enhancement Summary](./docs/backup-media-enhancement-summary.md)**: Summary of enhancements and feature comparisons
- **[Immich Backup Enhancement Summary](./docs/immich-backup-enhancement-summary.md)**: Details of Immich backup system improvements
- **[Docker Bootstrap Testing Framework](./docs/docker-bootstrap-testing-framework.md)**: Detailed documentation for the Docker-based bootstrap validation system
- **[Folder Metrics Script Documentation](./docs/folder-metrics.md)**: Detailed documentation for the `folder-metrics.sh` script
- **[Production Deployment Guide](./docs/production-deployment-guide.md)**: Complete deployment and operations guide
- **[Project Completion Summary](./docs/project-completion-summary.md)**: Overview of completed features and project milestones
## Dotfiles
@@ -26,61 +202,69 @@ curl -fsSL https://raw.githubusercontent.com/acedanger/shell/main/bootstrap.sh |
For more information about the dotfiles, see [Dotfiles README](./dotfiles/README.md).
## Tab Completion
This repository includes intelligent bash completion for all backup scripts, providing tab completion for command-line flags and options.
### Features
- **Intelligent flag completion**: Tab completion for all backup script options
- **Webhook URL suggestions**: Auto-complete common webhook endpoints
- **Path-aware completion**: Works with relative, absolute, and PATH-based script execution
- **Cross-shell support**: Compatible with both bash and zsh
### Supported Scripts
- **backup-immich.sh**: `--help`, `--dry-run`, `--no-upload`, `--verbose`
- **backup-plex.sh**: `--help`, `--auto-repair`, `--check-integrity`, `--non-interactive`, `--no-parallel`, `--no-performance`, `--webhook`, `--email`
- **backup-media.sh**: `--help`, `--dry-run`, `--no-verify`, `--sequential`, `--interactive`, `--webhook`
- **Generic backup scripts**: Common flags for all backup utilities
### Usage Examples
```bash
# Tab completion for backup script flags
~/shell/immich/backup-immich.sh --<TAB>
# Shows: --help --dry-run --no-upload --verbose
# Tab completion for webhook URLs
~/shell/plex/backup-plex.sh --webhook <TAB>
# Shows: https://notify.peterwood.rocks/lab
# Works with any script path
backup-immich.sh --<TAB> # From PATH
./backup-immich.sh --<TAB> # Relative path
/full/path/backup-immich.sh --<TAB> # Absolute path
```
### Installation
Tab completion is automatically installed when you run the setup scripts:
```bash
./setup/bootstrap.sh # Installs completion automatically
```
For manual installation or more details, see [Completions README](./completions/README.md).
## Testing
This repository includes Docker-based testing to validate the setup process across different environments:
This repository includes Docker-based testing to validate the setup process across different environments. For complete testing documentation, see [Docker Bootstrap Testing Framework](./docs/docker-bootstrap-testing-framework.md).
- **test-setup.sh**: Script that validates the bootstrap and setup process
- **run-docker-tests.sh**: Runner script that executes tests in Docker containers
- **Dockerfile**: Defines test environments (Ubuntu, Debian)
### Running Tests
Test your setup in isolated Docker containers with:
### Quick Testing
```bash
# Test in Ubuntu container
./run-docker-tests.sh ubuntu
./setup/run-docker-tests.sh ubuntu
# Test in Debian container
./run-docker-tests.sh debian
./setup/run-docker-tests.sh debian
# Run full bootstrap test in Ubuntu
./run-docker-tests.sh full-ubuntu
# Run full bootstrap test in Debian
./run-docker-tests.sh full-debian
# Test in both Ubuntu and Debian
./run-docker-tests.sh all
# Test in both environments
./setup/run-docker-tests.sh all
```
#### When to Use Each Testing Option
- **Use `./run-docker-tests.sh ubuntu` or `./run-docker-tests.sh debian`** when you want to:
- Quickly validate if packages in packages.list are available and installed
- Test if your test-setup.sh script is working correctly
- Check for issues with specific components without performing a full setup
- **Use `./run-docker-tests.sh full-ubuntu` or `./run-docker-tests.sh full-debian`** when you want to:
- Test the complete bootstrap installation process end-to-end
- Validate that all installation steps work correctly on a fresh system
- Simulate what users will experience when running the bootstrap script
- **Use `./run-docker-tests.sh all`** when you want to:
- Ensure your test-setup.sh works across both Ubuntu and Debian
- Run comprehensive checks before committing changes
The test environment checks:
- Package availability and installation
- Core components (git, curl, wget, etc.)
- Additional packages from `setup/packages.list`
- Oh My Zsh and plugin installation
- Dotfile symlinks
Tests will continue even when some packages fail to install, reporting all issues in a comprehensive summary.
# plex.sh
## plex.sh
This script is used to manage the Plex Media Server service on a systemd-based Linux distribution. It provides the following functionalities:
@@ -96,3 +280,25 @@ Note that these commands will run as `root`.
```shell
./shell/plex.sh {start|stop|restart|status}
```
## Enhanced Crontab System Migration
For complete crontab management documentation, see [Crontab Management Documentation](./crontab/README.md).
### Current Status
The crontab system uses hostname-specific configuration files:
- **crontab-europa.txt** - Media server configuration
- **crontab-io.txt** - Download/acquisition server configuration
- **crontab-racknerd.txt** - Backup server configuration
### Quick Management
```bash
# Install system-specific crontab (auto-detects hostname)
./crontab/manage-enhanced-crontab.sh install
# View current system's crontab configuration
./crontab/manage-enhanced-crontab.sh show
```