shell/README.md

# Shell Scripts and Dotfiles Repository

This repository contains various shell scripts for managing media-related tasks and dotfiles for system configuration.

## Quick Navigation

- **[Backup Scripts](#backup-scripts)** - Enterprise-grade backup solutions
- **[Management Scripts](#management-scripts)** - System and service management
- **[Tab Completion](#tab-completion)** - Intelligent command-line completion
- **[Documentation](#documentation)** - Complete guides and references
- **[Testing](#testing)** - Docker-based validation framework
- **[Dotfiles](#dotfiles)** - System configuration files

## Available Scripts

### Backup Scripts

- **`backup-media.sh`**: Enterprise-grade media backup script with parallel processing, comprehensive logging, and verification features.
- **`backup-plex.sh`**: Enhanced Plex backup script with integrity verification, incremental backups, and advanced features.
- **`restore-plex.sh`**: Script to restore Plex data from backups with safety checks.
- **`validate-plex-backups.sh`**: Script to validate backup integrity and monitor backup health.

### Management Scripts

- **`plex.sh`**: Script to manage the Plex Media Server (start, stop, restart, status).
- **`folder-metrics.sh`**: Script to calculate disk usage and file count for a directory and its subdirectories.

### Development Projects

- **[Telegram Backup Monitoring Bot](./telegram/github-issues/README.md)**: Comprehensive Telegram bot project for monitoring and managing all backup systems with real-time notifications and control capabilities.

### Testing Scripts

- **`test-setup.sh`**: Validates the bootstrap and setup process.
- **`run-docker-tests.sh`**: Runner script that executes tests in Docker containers.

## Enhanced Media Backup System

This repository includes enterprise-grade backup solutions for both general media files and Plex Media Server with comprehensive features for reliability, performance, and monitoring.

### Media Backup Script (`backup-media.sh`)

The enhanced media backup script provides enterprise-grade features for backing up large media collections:

#### Key Features

- **Parallel Processing**: Multi-threaded operations with configurable worker pools
- **Comprehensive Logging**: Multiple formats (text, JSON, markdown) with detailed metrics
- **Backup Verification**: SHA-256 checksum validation and integrity checks
- **Performance Monitoring**: Real-time progress tracking and transfer statistics
- **Automatic Cleanup**: Configurable retention policies with space management
- **Smart Notifications**: Detailed completion reports with statistics
- **Safety Features**: Dry-run mode, pre-flight checks, and graceful error handling
- **Interactive Mode**: Manual control with real-time feedback

#### Usage Examples

```bash
# Standard parallel backup (recommended)
./backup-media.sh

# Sequential backup for better compatibility
./backup-media.sh --sequential

# Test run without making changes
./backup-media.sh --dry-run

# Interactive mode with manual control
./backup-media.sh --interactive

# Verbose logging with performance metrics
./backup-media.sh --verbose

# Custom source and destination
./backup-media.sh --source /path/to/media --destination /path/to/backup
```

### Configuration Options

The script includes configurable parameters:

- `PARALLEL_JOBS=4`: Number of parallel rsync processes
- `MAX_BACKUP_AGE_DAYS=90`: Retention period for old backups
- `BACKUP_ROOT`: Default backup destination
- `LOG_ROOT`: Location for backup logs

### Performance Features

- **Progress Tracking**: Real-time file transfer progress
- **Transfer Statistics**: Bandwidth, file counts, and timing metrics
- **Resource Monitoring**: CPU and memory usage tracking
- **Optimization**: Intelligent file handling and compression options

### Advanced Plex Backup System

Specialized backup system for Plex Media Server with database-aware features. For complete documentation, see [Plex Scripts Documentation](./plex/README.md).

## Backup Usage Examples

### Media Backup Operations

```bash
# Quick media backup with default settings
./backup-media.sh

# High-performance parallel backup
./backup-media.sh --parallel --workers 8

# Test backup strategy without making changes
./backup-media.sh --dry-run --verbose

# Custom backup with specific paths
./backup-media.sh --source /mnt/movies --destination /backup/movies
```

### Advanced Plex Operations

```bash
# Run enhanced Plex backup (recommended)
./backup-plex.sh

# Validate all backups and generate report
./validate-plex-backups.sh --report

# Quick validation check
./validate-plex-backups.sh

# Test restore without making changes (dry run)
./restore-plex.sh plex-backup-20250125_143022.tar.gz --dry-run

# Restore from specific backup archive
./restore-plex.sh plex-backup-20250125_143022.tar.gz
```

## Automation and Scheduling

### Daily Media Backup

```bash
# Add to crontab for daily media backup at 2 AM
0 2 * * * /home/acedanger/shell/backup-media.sh --parallel

# Alternative: Sequential backup for systems with limited resources
0 2 * * * /home/acedanger/shell/backup-media.sh --sequential
```

### Automated Plex Backup with Validation

### Daily Plex Backup with Validation

```bash
# Add to crontab for daily Plex backup at 3 AM
0 3 * * * /home/acedanger/shell/backup-plex.sh

# Add daily validation at 7 AM
0 7 * * * /home/acedanger/shell/validate-plex-backups.sh --fix
```

### Weekly Comprehensive Validation Report

```bash
# Generate detailed weekly report (Sundays at 8 AM)
0 8 * * 0 /home/acedanger/shell/validate-plex-backups.sh --report
```

## Backup Configuration and Strategy

For detailed configuration information and backup strategies, see:

- **[Plex Backup Configuration](./plex/README.md#configuration)**: Plex-specific settings and options
- **[Immich Backup Configuration](./immich/README.md#configuration)**: Database and upload directory settings
- **[Enhanced Media Backup](./docs/enhanced-media-backup.md)**: Media backup script configuration
- **[Crontab Management](./crontab/README.md)**: Automated scheduling and system-specific configurations

## Documentation

### Component-Specific Documentation

- **[Plex Scripts Documentation](./plex/README.md)**: Comprehensive documentation for Plex backup, restoration, validation, and management scripts
- **[Immich Scripts Documentation](./immich/README.md)**: Complete guide for Immich backup and management system with PostgreSQL and B2 integration
- **[Crontab Management Documentation](./crontab/README.md)**: Complete guide for crontab management, backup systems, and multi-system administration
- **[Dotfiles Documentation](./dotfiles/README.md)**: System configuration files and setup instructions

### Technical Documentation

- **[Enhanced Media Backup Documentation](./docs/enhanced-media-backup.md)**: Comprehensive guide for the enhanced `backup-media.sh` script with enterprise features
- **[Basic Media Backup Documentation](./docs/backup-media.md)**: Documentation for the original `backup-media.sh` script
- **[Media Backup Enhancement Summary](./docs/backup-media-enhancement-summary.md)**: Summary of enhancements and feature comparisons
- **[Immich Backup Enhancement Summary](./docs/immich-backup-enhancement-summary.md)**: Details of Immich backup system improvements
- **[Docker Bootstrap Testing Framework](./docs/docker-bootstrap-testing-framework.md)**: Detailed documentation for the Docker-based bootstrap validation system
- **[Folder Metrics Script Documentation](./docs/folder-metrics.md)**: Detailed documentation for the `folder-metrics.sh` script
- **[Production Deployment Guide](./docs/production-deployment-guide.md)**: Complete deployment and operations guide
- **[Project Completion Summary](./docs/project-completion-summary.md)**: Overview of completed features and project milestones

## Dotfiles

The repository includes dotfiles for system configuration in the `dotfiles` directory. These can be automatically set up using the bootstrap script:

```bash
curl -fsSL https://raw.githubusercontent.com/acedanger/shell/main/bootstrap.sh | bash
```

For more information about the dotfiles, see [Dotfiles README](./dotfiles/README.md).

## Tab Completion

This repository includes intelligent bash completion for all backup scripts, providing tab completion for command-line flags and options.

### Features

- **Intelligent flag completion**: Tab completion for all backup script options
- **Webhook URL suggestions**: Auto-complete common webhook endpoints
- **Path-aware completion**: Works with relative, absolute, and PATH-based script execution
- **Cross-shell support**: Compatible with both bash and zsh

### Supported Scripts

- **backup-immich.sh**: `--help`, `--dry-run`, `--no-upload`, `--verbose`
- **backup-plex.sh**: `--help`, `--auto-repair`, `--check-integrity`, `--non-interactive`, `--no-parallel`, `--no-performance`, `--webhook`, `--email`
- **backup-media.sh**: `--help`, `--dry-run`, `--no-verify`, `--sequential`, `--interactive`, `--webhook`
- **Generic backup scripts**: Common flags for all backup utilities

### Usage Examples

```bash
# Tab completion for backup script flags
~/shell/immich/backup-immich.sh --<TAB>
# Shows: --help --dry-run --no-upload --verbose

# Tab completion for webhook URLs
~/shell/plex/backup-plex.sh --webhook <TAB>
# Shows: https://notify.peterwood.rocks/lab

# Works with any script path
backup-immich.sh --<TAB>          # From PATH
./backup-immich.sh --<TAB>        # Relative path
/full/path/backup-immich.sh --<TAB>  # Absolute path
```

### Installation

Tab completion is automatically installed when you run the setup scripts:

```bash
./setup/bootstrap.sh  # Installs completion automatically
```

For manual installation or more details, see [Completions README](./completions/README.md).

## Testing

This repository includes Docker-based testing to validate the setup process across different environments. For complete testing documentation, see [Docker Bootstrap Testing Framework](./docs/docker-bootstrap-testing-framework.md).

### Quick Testing

```bash
# Test in Ubuntu container
./setup/run-docker-tests.sh ubuntu

# Test in Debian container
./setup/run-docker-tests.sh debian

# Test in both environments
./setup/run-docker-tests.sh all
```

## plex.sh

This script is used to manage the Plex Media Server service on a systemd-based Linux distribution. It provides the following functionalities:

- **start**: Starts the Plex Media Server.
- **stop**: Stops the Plex Media Server.
- **restart**: Restarts the Plex Media Server.
- **status**: Displays the current status of the Plex Media Server.

## Usage

Note that these commands will run as `root`.

```shell
./shell/plex.sh {start|stop|restart|status}
```

## Enhanced Crontab System Migration

For complete crontab management documentation, see [Crontab Management Documentation](./crontab/README.md).

### Current Status

The crontab system uses hostname-specific configuration files:

- **crontab-europa.txt** - Media server configuration
- **crontab-io.txt** - Download/acquisition server configuration
- **crontab-racknerd.txt** - Backup server configuration

### Quick Management

```bash
# Install system-specific crontab (auto-detects hostname)
./crontab/manage-enhanced-crontab.sh install

# View current system's crontab configuration
./crontab/manage-enhanced-crontab.sh show
```