shell/README.md

# Shell Scripts and Dotfiles Repository

This repository contains various shell scripts for managing media-related tasks and dotfiles for system configuration.

## Available Scripts

### Backup Scripts

- **`backup-media.sh`**: Enterprise-grade media backup script with parallel processing, comprehensive logging, and verification features.
- **`backup-plex.sh`**: Enhanced Plex backup script with integrity verification, incremental backups, and advanced features.
- **`restore-plex.sh`**: Script to restore Plex data from backups with safety checks.
- **`validate-plex-backups.sh`**: Script to validate backup integrity and monitor backup health.

### Management Scripts

- **`plex.sh`**: Script to manage the Plex Media Server (start, stop, restart, status).
- **`folder-metrics.sh`**: Script to calculate disk usage and file count for a directory and its subdirectories.

### Testing Scripts

- **`test-setup.sh`**: Validates the bootstrap and setup process.
- **`run-docker-tests.sh`**: Runner script that executes tests in Docker containers.

## Enhanced Media Backup System

This repository includes enterprise-grade backup solutions for both general media files and Plex Media Server with comprehensive features for reliability, performance, and monitoring.

### Media Backup Script (`backup-media.sh`)

The enhanced media backup script provides enterprise-grade features for backing up large media collections:

#### Key Features

- **Parallel Processing**: Multi-threaded operations with configurable worker pools
- **Comprehensive Logging**: Multiple formats (text, JSON, markdown) with detailed metrics
- **Backup Verification**: SHA-256 checksum validation and integrity checks
- **Performance Monitoring**: Real-time progress tracking and transfer statistics
- **Automatic Cleanup**: Configurable retention policies with space management
- **Smart Notifications**: Detailed completion reports with statistics
- **Safety Features**: Dry-run mode, pre-flight checks, and graceful error handling
- **Interactive Mode**: Manual control with real-time feedback

#### Usage Examples

```bash
# Standard parallel backup (recommended)
./backup-media.sh

# Sequential backup for better compatibility
./backup-media.sh --sequential

# Test run without making changes
./backup-media.sh --dry-run

# Interactive mode with manual control
./backup-media.sh --interactive

# Verbose logging with performance metrics
./backup-media.sh --verbose

# Custom source and destination
./backup-media.sh --source /path/to/media --destination /path/to/backup
```

### Configuration Options

The script includes configurable parameters:

- `PARALLEL_JOBS=4`: Number of parallel rsync processes
- `MAX_BACKUP_AGE_DAYS=90`: Retention period for old backups
- `BACKUP_ROOT`: Default backup destination
- `LOG_ROOT`: Location for backup logs

### Performance Features

- **Progress Tracking**: Real-time file transfer progress
- **Transfer Statistics**: Bandwidth, file counts, and timing metrics
- **Resource Monitoring**: CPU and memory usage tracking
- **Optimization**: Intelligent file handling and compression options

### Advanced Plex Backup System

Specialized backup system for Plex Media Server with database-aware features:

### Components

- **`backup-plex.sh`**: Advanced backup script with integrity verification, incremental backups, and automatic cleanup
- **`restore-plex.sh`**: Safe restoration script with dry-run mode and current data backup
- **`validate-plex-backups.sh`**: Backup validation and health monitoring script

### Plex-Specific Features

- **Incremental backups**: Only backs up files that have changed since last backup
- **Database integrity verification**: Uses MD5 checksums to verify backup integrity
- **Automatic cleanup**: Configurable retention policies for old backups
- **Disk space monitoring**: Checks available space before starting backup
- **Safe restoration**: Backs up current data before restoring from backup
- **Comprehensive logging**: Detailed logs with color-coded output
- **Service management**: Safely stops/starts Plex during backup operations

## Backup Usage Examples

### Media Backup Operations

```bash
# Quick media backup with default settings
./backup-media.sh

# High-performance parallel backup
./backup-media.sh --parallel --workers 8

# Test backup strategy without making changes
./backup-media.sh --dry-run --verbose

# Custom backup with specific paths
./backup-media.sh --source /mnt/movies --destination /backup/movies
```

### Advanced Plex Operations

```bash
# Run enhanced Plex backup (recommended)
./backup-plex.sh

# Validate all backups and generate report
./validate-plex-backups.sh --report

# Quick validation check
./validate-plex-backups.sh

# Test restore without making changes (dry run)
./restore-plex.sh plex-backup-20250125_143022.tar.gz --dry-run

# Restore from specific backup archive
./restore-plex.sh plex-backup-20250125_143022.tar.gz
```

## Automation and Scheduling

### Daily Media Backup

```bash
# Add to crontab for daily media backup at 2 AM
0 2 * * * /home/acedanger/shell/backup-media.sh --parallel

# Alternative: Sequential backup for systems with limited resources
0 2 * * * /home/acedanger/shell/backup-media.sh --sequential
```

### Automated Plex Backup with Validation

### Daily Plex Backup with Validation

```bash
# Add to crontab for daily Plex backup at 3 AM
0 3 * * * /home/acedanger/shell/backup-plex.sh

# Add daily validation at 7 AM
0 7 * * * /home/acedanger/shell/validate-plex-backups.sh --fix
```

### Weekly Comprehensive Validation Report

```bash
# Generate detailed weekly report (Sundays at 8 AM)
0 8 * * 0 /home/acedanger/shell/validate-plex-backups.sh --report
```

## Backup Configuration and Strategy

### Media Backup Configuration

The enhanced media backup script includes configurable parameters at the top of the file:

- `PARALLEL_JOBS=4`: Number of parallel rsync processes
- `MAX_BACKUP_AGE_DAYS=90`: Remove backups older than 90 days
- `LOG_RETENTION_DAYS=30`: Keep logs for 30 days
- `BACKUP_ROOT`: Default location for backup storage
- `LOG_ROOT`: Location for backup logs and reports

### Plex Backup Configuration

The Plex backup script configuration parameters:

- `MAX_BACKUP_AGE_DAYS=30`: Remove backups older than 30 days
- `MAX_BACKUPS_TO_KEEP=10`: Keep maximum of 10 backup archives
- `BACKUP_ROOT`: Location for compressed backup archives
- `LOG_ROOT`: Location for backup logs

### Final Backup Directory Structure

The enhanced Plex backup system creates a streamlined archive-only structure:

```bash
/mnt/share/media/backups/plex/
├── plex-backup-20250125_143022.tar.gz    # Latest backup
├── plex-backup-20250124_143011.tar.gz    # Previous backup
├── plex-backup-20250123_143008.tar.gz    # Older backup
└── logs/
    ├── backup_log_20250125_143022.md
    ├── plex-backup-performance.json
    └── plex-backup.json
```

**Key Benefits:**

- **Direct Archive Storage**: No intermediate directories required
- **Efficient Space Usage**: Only compressed files stored permanently
- **Easy Management**: Timestamp-based naming for clear identification
- **Automatic Cleanup**: Legacy dated directories removed automatically

### Recommended Backup Strategy

Both systems implement a robust backup strategy following industry best practices:

**For Media Files:**

1. **Daily incremental backups** with parallel processing for speed
2. **Weekly verification** of backup integrity
3. **Monthly cleanup** of old backups based on retention policies
4. **Quarterly offsite sync** for disaster recovery

**For Plex Database:**

1. **Daily full backups** with service-aware operations
2. **Immediate validation** after each backup
3. **Weekly comprehensive reports** on backup health
4. **Monthly testing** of restore procedures

### Offsite Backup Integration

For comprehensive disaster recovery, sync backups to remote locations:

```bash
# Sync media backups to remote server daily at 5 AM
0 5 * * * rsync -av /mnt/share/media/backups/media/ user@remote-server:/backups/media/

# Sync Plex backups to remote server daily at 6 AM
0 6 * * * rsync -av /mnt/share/media/backups/plex/ user@remote-server:/backups/plex/
```

## Documentation

### Backup System Documentation

- [Enhanced Media Backup Documentation](./docs/enhanced-media-backup.md): Comprehensive guide for the enhanced `backup-media.sh` script with enterprise features.
- [Media Backup Enhancement Summary](./docs/backup-media-enhancement-summary.md): Summary of enhancements and feature comparisons.
- [Plex Backup Script Documentation](./docs/plex-backup.md): Detailed documentation for the `backup-plex.sh` script.

### Script Documentation

- [Plex Management Script Documentation](./docs/plex-management.md): Detailed documentation for the `plex.sh` script.
- [Folder Metrics Script Documentation](./docs/folder-metrics.md): Detailed documentation for the `folder-metrics.sh` script.
- [Testing Framework Documentation](./docs/testing.md): Detailed documentation for the Docker-based testing system.

## Dotfiles

The repository includes dotfiles for system configuration in the `dotfiles` directory. These can be automatically set up using the bootstrap script:

```bash
curl -fsSL https://raw.githubusercontent.com/acedanger/shell/main/bootstrap.sh | bash
```

For more information about the dotfiles, see [Dotfiles README](./dotfiles/README.md).

## Testing

This repository includes Docker-based testing to validate the setup process across different environments:

- **test-setup.sh**: Script that validates the bootstrap and setup process
- **run-docker-tests.sh**: Runner script that executes tests in Docker containers
- **Dockerfile**: Defines test environments (Ubuntu, Debian)

### Running Tests

Test your setup in isolated Docker containers with:

```bash
# Test in Ubuntu container
./run-docker-tests.sh ubuntu

# Test in Debian container
./run-docker-tests.sh debian

# Run full bootstrap test in Ubuntu
./run-docker-tests.sh full-ubuntu

# Run full bootstrap test in Debian
./run-docker-tests.sh full-debian

# Test in both Ubuntu and Debian
./run-docker-tests.sh all
```

#### When to Use Each Testing Option

- **Use `./run-docker-tests.sh ubuntu` or `./run-docker-tests.sh debian`** when you want to:
  - Quickly validate if packages in packages.list are available and installed
  - Test if your test-setup.sh script is working correctly
  - Check for issues with specific components without performing a full setup

- **Use `./run-docker-tests.sh full-ubuntu` or `./run-docker-tests.sh full-debian`** when you want to:
  - Test the complete bootstrap installation process end-to-end
  - Validate that all installation steps work correctly on a fresh system
  - Simulate what users will experience when running the bootstrap script

- **Use `./run-docker-tests.sh all`** when you want to:
  - Ensure your test-setup.sh works across both Ubuntu and Debian
  - Run comprehensive checks before committing changes

The test environment checks:

- Package availability and installation
- Core components (git, curl, wget, etc.)
- Additional packages from `setup/packages.list`
- Oh My Zsh and plugin installation
- Dotfile symlinks

Tests will continue even when some packages fail to install, reporting all issues in a comprehensive summary.

## plex.sh

This script is used to manage the Plex Media Server service on a systemd-based Linux distribution. It provides the following functionalities:

- **start**: Starts the Plex Media Server.
- **stop**: Stops the Plex Media Server.
- **restart**: Restarts the Plex Media Server.
- **status**: Displays the current status of the Plex Media Server.

## Usage

Note that these commands will run as `root`.

```shell
./shell/plex.sh {start|stop|restart|status}
```

## Enhanced Crontab System Migration

### System Status

The crontab system has been migrated from a universal `crontab.txt` to system-specific files:

- **crontab-europa.txt** - Media server configuration
- **crontab-io.txt** - Download/acquisition server configuration  
- **crontab-racknerd.txt** - Backup server configuration

### Legacy File Status

The original `crontab.txt` has been:

- **Backed up** to `crontab.txt.bak`
- **Role**: Previously served as fallback, now obsolete
- **Content**: Contains mixed configuration that was split into system-specific files

### Current Management

Use the system-specific approach:

```bash
# Install system-specific crontab
./manage-enhanced-crontab.sh install

# The script automatically detects hostname and uses appropriate file
# europa -> crontab-europa.txt
# io -> crontab-io.txt  
# racknerd -> crontab-racknerd.txt
```

### Fallback Behavior

If a system-specific file is missing, the management script will:

1. **Warning**: Display that system-specific file is not found
2. **Recommendation**: Create appropriate system-specific file

### Creating New System Files

For new systems, create system-specific files based on the templates:

```bash
# Copy and customize for new system
cp crontab-europa.txt crontab-newsystem.txt
# Edit for system-specific tasks
```

### Benefits of System-Specific Approach

- **Clear separation** of concerns between systems
- **Reduced conflicts** from universal configurations
- **System-appropriate** task scheduling
- **Better maintenance** and troubleshooting
- **Scalable** for additional systems

#### Migration Notes

Migration completed: May 26, 2025