Shell Scripts and Dotfiles Repository

This repository contains various shell scripts for managing media-related tasks and dotfiles for system configuration.

Quick Navigation

• Backup Scripts - Enterprise-grade backup solutions
• Management Scripts - System and service management
• Security - Comprehensive security framework and standards
• AI Integration - Ollama and Fabric setup for AI-assisted development
• Tab Completion - Intelligent command-line completion
• Documentation - Complete guides and references
• Testing - Docker-based validation framework
• Dotfiles - System configuration files

Available Scripts

Backup Scripts

• backup-media.sh: Enterprise-grade media backup script with parallel processing, comprehensive logging, and verification features.
• backup-plex.sh: Enhanced Plex backup script with integrity verification, incremental backups, and advanced features.
• restore-plex.sh: Script to restore Plex data from backups with safety checks.
• validate-plex-backups.sh: Script to validate backup integrity and monitor backup health.
• backup-immich.sh: Comprehensive Immich backup script with PostgreSQL database and upload directory backup to B2 storage.
• restore-immich.sh: Complete Immich restoration script with database and upload directory restoration from backups.
• validate-immich-backups.sh: Immich backup validation script with integrity checks and restoration command generation.

Management Scripts

• plex.sh: Script to manage the Plex Media Server (start, stop, restart, status).
• folder-metrics.sh: Script to calculate disk usage and file count for a directory and its subdirectories.

Security

This repository implements comprehensive security standards and practices for all shell scripts.

Security Framework

• Security Review Summary: Comprehensive security assessment results and risk analysis
• Security Checklist: Complete security validation checklist for development
• Security Remediation Plan: Prioritized security improvement roadmap
• Security Implementation Report: Detailed report of completed security enhancements

Security Standards

✅ Implemented Security Controls:

• All variables properly quoted to prevent injection attacks
• No direct remote code execution (curl | bash patterns eliminated)
• Appropriate file permissions (no 777 usage)
• Comprehensive input validation for user-provided data
• Secure temporary file handling with proper cleanup
• Robust error handling and logging

Security Rating: A- (Excellent - Industry standard security practices)

Key Security Features

• Command Injection Protection: All variables properly quoted in command contexts
• Remote Code Safety: Secure download and validation before script execution
• Privilege Management: Minimal privilege usage with appropriate permissions
• Input Validation: Comprehensive validation of paths, image names, and user inputs
• Error Handling: Secure error handling with proper cleanup procedures

Security Testing

All scripts undergo comprehensive security validation:

• Syntax validation with bash -n
• Variable quoting verification
• Privilege requirement analysis
• Input validation testing
• Security pattern compliance checking

For security-related changes, refer to the security documentation and follow the established security checklist.

AI Integration

This repository includes a complete AI development environment with Ollama and Fabric integration for AI-assisted development tasks.

Ollama + Fabric Setup

The system includes:

• Ollama Docker container running on port 11434 with phi3:mini model (3.8B parameters)
• Fabric v1.4.195 with 216+ AI patterns for text processing
• Google Gemini 2.5 Pro as primary AI provider
• Local Ollama models as secondary AI provider
• Custom shell aliases for easy container management

Basic Fabric Usage

# List all available patterns
fabric -l

# Use a pattern (uses Gemini 2.5 Pro by default)
echo "Your text here" | fabric -p summarize

# Use with specific model
echo "Your text here" | fabric -p summarize -m gemini-2.0-flash-exp

# Use local Ollama models
echo "Your text here" | fabric -p summarize -m ollama:phi3:mini

# Update patterns
fabric -U

Ollama Management Aliases

ollama-start      # Start Ollama container
ollama-stop       # Stop Ollama container
ollama-restart    # Restart Ollama container
ollama-logs       # View container logs
ollama-status     # Check container status

Popular AI Patterns

• summarize - Summarize text content
• explain_code - Explain code snippets and logic
• improve_writing - Enhance writing quality and clarity
• extract_wisdom - Extract key insights from content
• create_quiz - Generate quiz questions from text
• analyze_claims - Analyze and fact-check claims

Configuration Files

• Fabric config: ~/.config/fabric/.env - AI provider settings and API keys
• Ollama aliases: ~/.oh-my-zsh/custom/ollama-aliases.zsh - Container management commands
• Shell config: ~/.zshrc - Main shell configuration

For complete setup instructions, see the setup documentation.

Development Projects

• Telegram Backup Monitoring Bot: Comprehensive Telegram bot project for monitoring and managing all backup systems with real-time notifications and control capabilities.
• Docker Deployment Management: Centralized Docker stack deployment system for managing containers across multiple servers with intelligent assignment and environment integration.

Testing Scripts

• test-setup.sh: Validates the bootstrap and setup process.
• run-docker-tests.sh: Runner script that executes tests in Docker containers.

Enhanced Media Backup System

This repository includes enterprise-grade backup solutions for both general media files and Plex Media Server with comprehensive features for reliability, performance, and monitoring.

Media Backup Script (backup-media.sh)

The enhanced media backup script provides enterprise-grade features for backing up large media collections:

Key Features

• Parallel Processing: Multi-threaded operations with configurable worker pools
• Comprehensive Logging: Multiple formats (text, JSON, markdown) with detailed metrics
• Backup Verification: SHA-256 checksum validation and integrity checks
• Performance Monitoring: Real-time progress tracking and transfer statistics
• Automatic Cleanup: Configurable retention policies with space management
• Smart Notifications: Detailed completion reports with statistics
• Safety Features: Dry-run mode, pre-flight checks, and graceful error handling
• Interactive Mode: Manual control with real-time feedback

Usage Examples

# Standard parallel backup (recommended)
./backup-media.sh

# Sequential backup for better compatibility
./backup-media.sh --sequential

# Test run without making changes
./backup-media.sh --dry-run

# Interactive mode with manual control
./backup-media.sh --interactive

# Verbose logging with performance metrics
./backup-media.sh --verbose

# Custom source and destination
./backup-media.sh --source /path/to/media --destination /path/to/backup

Configuration Options

The script includes configurable parameters:

• PARALLEL_JOBS=4: Number of parallel rsync processes
• MAX_BACKUP_AGE_DAYS=90: Retention period for old backups
• BACKUP_ROOT: Default backup destination
• LOG_ROOT: Location for backup logs

Performance Features

• Progress Tracking: Real-time file transfer progress
• Transfer Statistics: Bandwidth, file counts, and timing metrics
• Resource Monitoring: CPU and memory usage tracking
• Optimization: Intelligent file handling and compression options

Advanced Plex Backup System

Specialized backup system for Plex Media Server with database-aware features. For complete documentation, see Plex Scripts Documentation.

Backup Usage Examples

Media Backup Operations

# Quick media backup with default settings
./backup-media.sh

# High-performance parallel backup
./backup-media.sh --parallel --workers 8

# Test backup strategy without making changes
./backup-media.sh --dry-run --verbose

# Custom backup with specific paths
./backup-media.sh --source /mnt/movies --destination /backup/movies

Advanced Plex Operations

# Run enhanced Plex backup (recommended)
./backup-plex.sh

# Validate all backups and generate report
./validate-plex-backups.sh --report

# Quick validation check
./validate-plex-backups.sh

# Test restore without making changes (dry run)
./restore-plex.sh plex-backup-20250125_143022.tar.gz --dry-run

# Restore from specific backup archive
./restore-plex.sh plex-backup-20250125_143022.tar.gz

Immich Photo Management Operations

# Create comprehensive Immich backup (database + uploads)
./immich/backup-immich.sh

# Test backup process without uploading to B2
./immich/backup-immich.sh --dry-run

# Backup without uploading to cloud storage
./immich/backup-immich.sh --no-upload

# Validate existing Immich backups
./immich/validate-immich-backups.sh

# Test restoration with dry-run (recommended first)
./immich/restore-immich.sh \
  --db-backup ./immich_backups/immich_db_backup_20250603_140000.sql.gz \
  --uploads-backup ./immich_backups/immich_uploads_20250603_140000.tar.gz \
  --dry-run

# Complete restoration from backup files
./immich/restore-immich.sh \
  --db-backup ./immich_backups/immich_db_backup_20250603_140000.sql.gz \
  --uploads-backup ./immich_backups/immich_uploads_20250603_140000.tar.gz

# Restore database only (skip uploads)
./immich/restore-immich.sh \
  --db-backup ./immich_backups/immich_db_backup_20250603_140000.sql.gz \
  --uploads-backup ./immich_backups/immich_uploads_20250603_140000.tar.gz \
  --skip-uploads

# Restore uploads only (skip database)
./immich/restore-immich.sh \
  --db-backup ./immich_backups/immich_db_backup_20250603_140000.sql.gz \
  --uploads-backup ./immich_backups/immich_uploads_20250603_140000.tar.gz \
  --skip-db

Automation and Scheduling

Daily Media Backup

# Add to crontab for daily media backup at 2 AM
0 2 * * * /home/acedanger/shell/backup-media.sh --parallel

# Alternative: Sequential backup for systems with limited resources
0 2 * * * /home/acedanger/shell/backup-media.sh --sequential

Automated Plex Backup with Validation

Daily Plex Backup with Validation

# Add to crontab for daily Plex backup at 3 AM
0 3 * * * /home/acedanger/shell/backup-plex.sh

# Add daily validation at 7 AM
0 7 * * * /home/acedanger/shell/validate-plex-backups.sh --fix

Automated Immich Backup

# Add to crontab for daily Immich backup at 4 AM
0 4 * * * /home/acedanger/shell/immich/backup-immich.sh

# Add weekly validation (Sundays at 9 AM)
0 9 * * 0 /home/acedanger/shell/immich/validate-immich-backups.sh

Weekly Comprehensive Validation Report

# Generate detailed weekly report (Sundays at 8 AM)
0 8 * * 0 /home/acedanger/shell/validate-plex-backups.sh --report

Backup Configuration and Strategy

For detailed configuration information and backup strategies, see:

• Plex Backup Configuration: Plex-specific settings and options
• Immich Backup Configuration: Database and upload directory settings
• Enhanced Media Backup: Media backup script configuration
• Crontab Management: Automated scheduling and system-specific configurations

Comprehensive Documentation

This repository includes extensive documentation organized by component and functionality:

📁 Documentation Structure

📁 docs/          - Technical documentation and guides
📁 setup/         - Setup and configuration files
📁 plex/          - Plex Media Server scripts and docs
📁 immich/        - Immich photo management scripts
📁 crontab/       - Crontab management system
📁 dotfiles/      - System configuration files
📁 completions/   - Tab completion scripts
📁 telegram/      - Telegram bot integration

🛠️ Component Documentation

Backup Systems

• Enhanced Media Backup: Advanced media backup features and configuration
• Basic Media Backup: Original backup system documentation
• Environment Backup System: Environment and configuration backup
• Backup Environment Safety Guide: Safety guidelines for environment backups

Media Server Management

• Plex Scripts Documentation: Comprehensive Plex backup, restoration, and management
• Plex Backup Guide: Detailed Plex backup system documentation
• Plex Management Guide: Server management and operations

Photo Management

• Immich Scripts Documentation: Complete Immich backup and management system
• Immich Enhancement Summary: Recent improvements and features
• Immich Migration Guide: Migration procedures and best practices

System Administration

• Crontab Management: Multi-system crontab coordination and backup
• Enhanced Crontab System: Advanced scheduling features
• Dotfiles Documentation: System configuration and setup
• Tab Completion Guide: Intelligent command completion system

Development & Deployment

• Docker Deployment Management: Centralized multi-server Docker stack deployment system
• Docker Bootstrap Testing: Comprehensive testing framework
• Production Deployment Guide: Complete deployment procedures
• Docker Stack Strategy: Container deployment strategy

📊 Project Documentation

• Project Completion Summary: Overall project status and milestones
• Documentation Review: Comprehensive documentation overview
• Technical Implementation Summaries: Various technical guides and implementation details

Documentation

Component-Specific Documentation

• Plex Scripts Documentation: Comprehensive documentation for Plex backup, restoration, validation, and management scripts
• Immich Scripts Documentation: Complete guide for Immich backup and management system with PostgreSQL and B2 integration
• Crontab Management Documentation: Complete guide for crontab management, backup systems, and multi-system administration
• Dotfiles Documentation: System configuration files and setup instructions

Technical Documentation

• Enhanced Media Backup Documentation: Comprehensive guide for the enhanced backup-media.sh script with enterprise features
• Basic Media Backup Documentation: Documentation for the original backup-media.sh script
• Media Backup Enhancement Summary: Summary of enhancements and feature comparisons
• Immich Backup Enhancement Summary: Details of Immich backup system improvements
• Docker Bootstrap Testing Framework: Detailed documentation for the Docker-based bootstrap validation system
• Folder Metrics Script Documentation: Detailed documentation for the folder-metrics.sh script
• Production Deployment Guide: Complete deployment and operations guide
• Project Completion Summary: Overview of completed features and project milestones

Environment & System Documentation

• Environment Backup System: Complete environment backup documentation
• Environment Backup Integration Guide: Integration procedures and best practices
• Backup Environment Safety Guide: Safety guidelines and procedures
• Tab Completion Implementation: Command completion system details
• Docker Stack Deployment Strategy: Container deployment and management

Project Reviews & Summaries

• Documentation Review Summary: Comprehensive documentation audit and review
• Issue Resolution Summaries: Technical implementation details and problem resolution guides

Setup and Getting Started

Quick Setup

The repository includes automated setup via bootstrap script:

curl -fsSL https://raw.githubusercontent.com/acedanger/shell/main/bootstrap.sh | bash

This installs:

• Zsh with Oh My Zsh and custom plugins
• All packages from setup/packages.list
• Dotfiles and system configuration
• Tab completion for all scripts
• Development tools (Node.js via nvm, VS Code, etc.)

AI Development Environment

For AI-assisted development, the system includes:

• Ollama running in Docker with local models
• Fabric with 216+ AI patterns for text processing
• Google Gemini integration as primary AI provider
• Custom aliases for easy management

Test the AI setup:

# Test Gemini integration
echo "Test text" | fabric -p summarize

# Test local Ollama integration
echo "Test text" | fabric -p summarize -m ollama:phi3:mini

Dotfiles

The repository includes dotfiles for system configuration in the dotfiles directory. These can be automatically set up using the bootstrap script:

curl -fsSL https://raw.githubusercontent.com/acedanger/shell/main/bootstrap.sh | bash

For more information about the dotfiles, see Dotfiles README.

Tab Completion

This repository includes intelligent bash completion for all backup scripts, providing tab completion for command-line flags and options.

Features

• Intelligent flag completion: Tab completion for all backup script options
• Webhook URL suggestions: Auto-complete common webhook endpoints
• Path-aware completion: Works with relative, absolute, and PATH-based script execution
• Cross-shell support: Compatible with both bash and zsh

Supported Scripts

• backup-immich.sh: --help, --dry-run, --no-upload, --verbose
• backup-plex.sh: --help, --auto-repair, --check-integrity, --non-interactive, --no-parallel, --no-performance, --webhook, --email
• backup-media.sh: --help, --dry-run, --no-verify, --sequential, --interactive, --webhook
• Generic backup scripts: Common flags for all backup utilities

Usage Examples

# Tab completion for backup script flags
~/shell/immich/backup-immich.sh --<TAB>
# Shows: --help --dry-run --no-upload --verbose

# Tab completion for webhook URLs
~/shell/plex/backup-plex.sh --webhook <TAB>
# Shows: https://notify.peterwood.rocks/lab

# Works with any script path
backup-immich.sh --<TAB>          # From PATH
./backup-immich.sh --<TAB>        # Relative path
/full/path/backup-immich.sh --<TAB>  # Absolute path

Installation

Tab completion is automatically installed when you run the setup scripts:

./setup/bootstrap.sh  # Installs completion automatically

For manual installation or more details, see Completions README.

Testing

This repository includes Docker-based testing to validate the setup process across different environments. For complete testing documentation, see Docker Bootstrap Testing Framework.

Quick Testing

# Test in Ubuntu container
./setup/run-docker-tests.sh ubuntu

# Test in Debian container
./setup/run-docker-tests.sh debian

# Test in both environments
./setup/run-docker-tests.sh all

plex.sh

This script is used to manage the Plex Media Server service on a systemd-based Linux distribution. It provides the following functionalities:

• start: Starts the Plex Media Server.
• stop: Stops the Plex Media Server.
• restart: Restarts the Plex Media Server.
• status: Displays the current status of the Plex Media Server.

Usage

Note that these commands will run as root.

./shell/plex.sh {start|stop|restart|status}

Enhanced Crontab System Migration

For complete crontab management documentation, see Crontab Management Documentation.

Current Status

The crontab system uses hostname-specific configuration files:

• crontab-europa.txt - Media server configuration
• crontab-io.txt - Download/acquisition server configuration
• crontab-racknerd.txt - Backup server configuration

Quick Management

# Install system-specific crontab (auto-detects hostname)
./crontab/manage-enhanced-crontab.sh install

# View current system's crontab configuration
./crontab/manage-enhanced-crontab.sh show