shell/README.md

Shell Scripts and Dotfiles Repository

This repository contains various shell scripts for managing media-related tasks and dotfiles for system configuration.

Available Scripts

Backup Scripts

• backup-media.sh: Enterprise-grade media backup script with parallel processing, comprehensive logging, and verification features.
• backup-plex.sh: Enhanced Plex backup script with integrity verification, incremental backups, and advanced features.
• restore-plex.sh: Script to restore Plex data from backups with safety checks.
• validate-plex-backups.sh: Script to validate backup integrity and monitor backup health.

Management Scripts

• plex.sh: Script to manage the Plex Media Server (start, stop, restart, status).
• folder-metrics.sh: Script to calculate disk usage and file count for a directory and its subdirectories.

Testing Scripts

• test-setup.sh: Validates the bootstrap and setup process.
• run-docker-tests.sh: Runner script that executes tests in Docker containers.

Enhanced Media Backup System

This repository includes enterprise-grade backup solutions for both general media files and Plex Media Server with comprehensive features for reliability, performance, and monitoring.

Media Backup Script (backup-media.sh)

The enhanced media backup script provides enterprise-grade features for backing up large media collections:

Key Features

• Parallel Processing: Multi-threaded operations with configurable worker pools
• Comprehensive Logging: Multiple formats (text, JSON, markdown) with detailed metrics
• Backup Verification: SHA-256 checksum validation and integrity checks
• Performance Monitoring: Real-time progress tracking and transfer statistics
• Automatic Cleanup: Configurable retention policies with space management
• Smart Notifications: Detailed completion reports with statistics
• Safety Features: Dry-run mode, pre-flight checks, and graceful error handling
• Interactive Mode: Manual control with real-time feedback

Usage Examples

# Standard parallel backup (recommended)
./backup-media.sh

# Sequential backup for better compatibility
./backup-media.sh --sequential

# Test run without making changes
./backup-media.sh --dry-run

# Interactive mode with manual control
./backup-media.sh --interactive

# Verbose logging with performance metrics
./backup-media.sh --verbose

# Custom source and destination
./backup-media.sh --source /path/to/media --destination /path/to/backup

Configuration Options

The script includes configurable parameters:

• PARALLEL_JOBS=4: Number of parallel rsync processes
• MAX_BACKUP_AGE_DAYS=90: Retention period for old backups
• BACKUP_ROOT: Default backup destination
• LOG_ROOT: Location for backup logs

Performance Features

• Progress Tracking: Real-time file transfer progress
• Transfer Statistics: Bandwidth, file counts, and timing metrics
• Resource Monitoring: CPU and memory usage tracking
• Optimization: Intelligent file handling and compression options

Advanced Plex Backup System

Specialized backup system for Plex Media Server with database-aware features:

Components

• backup-plex.sh: Advanced backup script with integrity verification, incremental backups, and automatic cleanup
• restore-plex.sh: Safe restoration script with dry-run mode and current data backup
• validate-plex-backups.sh: Backup validation and health monitoring script

Plex-Specific Features

• Incremental backups: Only backs up files that have changed since last backup
• Database integrity verification: Uses MD5 checksums to verify backup integrity
• Automatic cleanup: Configurable retention policies for old backups
• Disk space monitoring: Checks available space before starting backup
• Safe restoration: Backs up current data before restoring from backup
• Comprehensive logging: Detailed logs with color-coded output
• Service management: Safely stops/starts Plex during backup operations

Backup Usage Examples

Media Backup Operations

# Quick media backup with default settings
./backup-media.sh

# High-performance parallel backup
./backup-media.sh --parallel --workers 8

# Test backup strategy without making changes
./backup-media.sh --dry-run --verbose

# Custom backup with specific paths
./backup-media.sh --source /mnt/movies --destination /backup/movies

Advanced Plex Operations

# Run enhanced Plex backup (recommended)
./backup-plex.sh

# Validate all backups and generate report
./validate-plex-backups.sh --report

# Quick validation check
./validate-plex-backups.sh

# Test restore without making changes (dry run)
./restore-plex.sh 20250125 --dry-run

# Restore from specific backup
./restore-plex.sh 20250125

Automation and Scheduling

Daily Media Backup

# Add to crontab for daily media backup at 2 AM
0 2 * * * /home/acedanger/shell/backup-media.sh --parallel

# Alternative: Sequential backup for systems with limited resources
0 2 * * * /home/acedanger/shell/backup-media.sh --sequential

Automated Plex Backup with Validation

Daily Plex Backup with Validation

# Add to crontab for daily Plex backup at 3 AM
0 3 * * * /home/acedanger/shell/backup-plex.sh

# Add daily validation at 7 AM
0 7 * * * /home/acedanger/shell/validate-plex-backups.sh --fix

Weekly Comprehensive Validation Report

# Generate detailed weekly report (Sundays at 8 AM)
0 8 * * 0 /home/acedanger/shell/validate-plex-backups.sh --report

Backup Configuration and Strategy

Media Backup Configuration

The enhanced media backup script includes configurable parameters at the top of the file:

• PARALLEL_JOBS=4: Number of parallel rsync processes
• MAX_BACKUP_AGE_DAYS=90: Remove backups older than 90 days
• LOG_RETENTION_DAYS=30: Keep logs for 30 days
• BACKUP_ROOT: Default location for backup storage
• LOG_ROOT: Location for backup logs and reports

Plex Backup Configuration

The Plex backup script configuration parameters:

• MAX_BACKUP_AGE_DAYS=30: Remove backups older than 30 days
• MAX_BACKUPS_TO_KEEP=10: Keep maximum of 10 backup sets
• BACKUP_ROOT: Location for backup storage
• LOG_ROOT: Location for backup logs

Recommended Backup Strategy

Both systems implement a robust backup strategy following industry best practices:

For Media Files:

1. Daily incremental backups with parallel processing for speed
2. Weekly verification of backup integrity
3. Monthly cleanup of old backups based on retention policies
4. Quarterly offsite sync for disaster recovery

For Plex Database:

1. Daily full backups with service-aware operations
2. Immediate validation after each backup
3. Weekly comprehensive reports on backup health
4. Monthly testing of restore procedures

Offsite Backup Integration

For comprehensive disaster recovery, sync backups to remote locations:

# Sync media backups to remote server daily at 5 AM
0 5 * * * rsync -av /mnt/share/media/backups/media/ user@remote-server:/backups/media/

# Sync Plex backups to remote server daily at 6 AM
0 6 * * * rsync -av /mnt/share/media/backups/plex/ user@remote-server:/backups/plex/

Documentation

Backup System Documentation

• Enhanced Media Backup Documentation: Comprehensive guide for the enhanced backup-media.sh script with enterprise features.
• Media Backup Enhancement Summary: Summary of enhancements and feature comparisons.
• Plex Backup Script Documentation: Detailed documentation for the backup-plex.sh script.

Script Documentation

• Plex Management Script Documentation: Detailed documentation for the plex.sh script.
• Folder Metrics Script Documentation: Detailed documentation for the folder-metrics.sh script.
• Testing Framework Documentation: Detailed documentation for the Docker-based testing system.

Dotfiles

The repository includes dotfiles for system configuration in the dotfiles directory. These can be automatically set up using the bootstrap script:

curl -fsSL https://raw.githubusercontent.com/acedanger/shell/main/bootstrap.sh | bash

For more information about the dotfiles, see Dotfiles README.

Testing

This repository includes Docker-based testing to validate the setup process across different environments:

• test-setup.sh: Script that validates the bootstrap and setup process
• run-docker-tests.sh: Runner script that executes tests in Docker containers
• Dockerfile: Defines test environments (Ubuntu, Debian)

Running Tests

Test your setup in isolated Docker containers with:

# Test in Ubuntu container
./run-docker-tests.sh ubuntu

# Test in Debian container
./run-docker-tests.sh debian

# Run full bootstrap test in Ubuntu
./run-docker-tests.sh full-ubuntu

# Run full bootstrap test in Debian
./run-docker-tests.sh full-debian

# Test in both Ubuntu and Debian
./run-docker-tests.sh all

When to Use Each Testing Option

• Use ./run-docker-tests.sh ubuntu or ./run-docker-tests.sh debian when you want to:

  • Quickly validate if packages in packages.list are available and installed
  • Test if your test-setup.sh script is working correctly
  • Check for issues with specific components without performing a full setup

• Use ./run-docker-tests.sh full-ubuntu or ./run-docker-tests.sh full-debian when you want to:

  • Test the complete bootstrap installation process end-to-end
  • Validate that all installation steps work correctly on a fresh system
  • Simulate what users will experience when running the bootstrap script

• Use ./run-docker-tests.sh all when you want to:

  • Ensure your test-setup.sh works across both Ubuntu and Debian
  • Run comprehensive checks before committing changes

The test environment checks:

• Package availability and installation
• Core components (git, curl, wget, etc.)
• Additional packages from setup/packages.list
• Oh My Zsh and plugin installation
• Dotfile symlinks

Tests will continue even when some packages fail to install, reporting all issues in a comprehensive summary.

plex.sh

This script is used to manage the Plex Media Server service on a systemd-based Linux distribution. It provides the following functionalities:

• start: Starts the Plex Media Server.
• stop: Stops the Plex Media Server.
• restart: Restarts the Plex Media Server.
• status: Displays the current status of the Plex Media Server.

Usage

Note that these commands will run as root.

./shell/plex.sh {start|stop|restart|status}