Files
shell/immich
Peter Wood 6d726cb015 feat: Add base HTML template and implement dashboard, logs, and service views
- Created a base HTML template for consistent layout across pages.
- Developed a dashboard page to display backup service metrics and statuses.
- Implemented a log viewer for detailed log file inspection.
- Added error handling page for better user experience during failures.
- Introduced service detail page to show specific service metrics and actions.
- Enhanced log filtering and viewing capabilities.
- Integrated auto-refresh functionality for real-time updates on metrics.
- Created integration and unit test scripts for backup metrics functionality.
2025-06-18 08:06:08 -04:00
..

Immich Management Scripts

This directory contains scripts for managing and backing up Immich photo management system.

Scripts

backup-immich.sh

Complete backup script for Immich installation that creates backups of:

  • PostgreSQL database (using pg_dumpall as recommended by Immich)
  • User upload directories (photos, videos, and metadata)

Requirements:

  • .env file in the parent directory (/home/acedanger/shell/.env) with:
    • DB_USERNAME - PostgreSQL username
    • DB_DATABASE_NAME - Database name
    • UPLOAD_LOCATION - Path to Immich upload directory
  • Docker containers: immich_postgres and immich_server

Usage:

./backup-immich.sh [OPTIONS]

Command-Line Options:

  • --help, -h - Show help message and exit
  • --dry-run - Show what would be backed up without performing actual backup
  • --no-upload - Skip B2 upload (local backup only)
  • --verbose - Enable verbose logging

Examples:

# Standard backup (default behavior)
./backup-immich.sh

# Show help and usage information
./backup-immich.sh --help

# Preview what would be backed up without executing
./backup-immich.sh --dry-run

# Backup locally only (skip B2 upload)
./backup-immich.sh --no-upload

# Run with verbose logging
./backup-immich.sh --verbose

# Combine options
./backup-immich.sh --no-upload --verbose

Backup Location:

Primary Storage: /mnt/share/media/backups/immich/ (shared storage)

  • Database: immich_db_backup_YYYYMMDD_HHMMSS.sql.gz
  • Uploads: immich_uploads_YYYYMMDD_HHMMSS.tar.gz

Temporary Location: ../immich_backups/ (cleaned up after copy to shared storage)

Backup Workflow:

  1. Create local backups in temporary directory (../immich_backups/)
  2. Copy to shared storage (/mnt/share/media/backups/immich/)
  3. Upload to Backblaze B2 (if configured)
  4. Delete local copies (shared storage copies retained)

Features:

  • Smart backup workflow: Creates → Copies to shared storage → Uploads to B2 → Cleans up locally
  • Command-line options for flexible operation (--help, --dry-run, --no-upload, --verbose)
  • Dry-run mode to preview operations without executing
  • Option to skip B2 upload for local-only backups
  • Shared storage integration: Automatically copies backups to /mnt/share/media/backups/immich/
  • Local cleanup: Removes temporary files after successful copy to shared storage
  • Automatic container pausing/resuming during backup
  • Comprehensive error handling and cleanup
  • Graceful degradation: Retains local backups if shared storage copy fails
  • Backup validation and health checks
  • Automatic compression
  • Old backup cleanup (configurable)
  • Centralized logging to /home/acedanger/shell/logs/
  • Detailed progress reporting and timestamped logs
  • 🔔 Webhook notifications to notify.peterwood.rocks/lab
  • ☁️ Backblaze B2 integration for off-site backup storage
  • 📊 File size reporting in notifications

Usage Examples

Basic Operations

Standard Backup (Default)

./backup-immich.sh
# Performs complete backup with all default settings:
# - Backs up database and upload directory
# - Uploads to B2 if configured
# - Sends webhook notifications
# - Logs to /home/acedanger/shell/logs/immich-backup.log

Getting Help

./backup-immich.sh --help
# Shows complete usage information including:
# - All available command-line options
# - Configuration requirements
# - Examples and restore instructions

Preview and Testing

Dry Run (Preview Mode)

./backup-immich.sh --dry-run
# Shows what would be backed up without executing:
# - Checks all prerequisites and container status
# - Displays backup file paths and estimated sizes
# - Validates B2 configuration if present
# - Reports any issues that would prevent backup
# - No files are created or modified

Example dry-run output:

=== DRY RUN MODE - NO ACTUAL BACKUP WILL BE PERFORMED ===

Configuration:
  - Database: immich
  - Username: postgres
  - Upload Location: /opt/immich/upload
  - Container: immich_postgres
  - Backup Directory: /home/acedanger/shell/immich_backups

Would create:
  - Database backup: /home/acedanger/shell/immich_backups/immich_db_backup_20250527_140000.sql.gz
  - Upload backup: /home/acedanger/shell/immich_backups/immich_uploads_20250527_140000.tar.gz

Workflow:
  1. Create local backups
  2. Copy to shared storage: /mnt/share/media/backups/immich/
  3. Upload to B2 (if configured)
  4. Delete local backups (keep shared copies)

Container Status Check:
  ✓ immich_server: Running (would pause during backup)
  ✓ immich_postgres: Running
  ✓ Upload directory: /opt/immich/upload (42GB)

B2 Upload Configuration:
  ✓ B2 configured - would upload to bucket: my-immich-backups
  ✓ B2 CLI found at: /home/acedanger/shell/immich/b2-linux

Shared Storage Check:
  ✓ Shared storage accessible: /mnt/share/media/backups
  ✓ Shared storage writable - would copy backups before B2 upload

=== DRY RUN COMPLETE - No files were created or modified ===

Local Backup Only

Skip B2 Upload

./backup-immich.sh --no-upload
# Performs backup but skips B2 upload:
# - Creates local backup files
# - Validates backup integrity
# - Sends notifications (without B2 status)
# - Useful for testing or when B2 is unavailable

Verbose Logging

Detailed Output

./backup-immich.sh --verbose
# Enables detailed logging for troubleshooting:
# - Shows additional progress information
# - Includes Docker command output
# - Provides more detailed error messages
# - Helpful for debugging issues

Combined Options

Local Backup with Verbose Output

./backup-immich.sh --no-upload --verbose
# Combines multiple options:
# - Creates local backup only (no B2 upload)
# - Shows detailed progress and logging
# - Useful for testing or troubleshooting

Preview with Verbose Details

./backup-immich.sh --dry-run --verbose
# Shows detailed preview information:
# - Extended configuration validation
# - More detailed container status
# - Comprehensive B2 configuration check
# - Additional filesystem checks

Automation Examples

Scheduled Backup (Crontab)

# Daily backup at 2:00 AM with logging
0 2 * * * /home/acedanger/shell/immich/backup-immich.sh >> /home/acedanger/shell/logs/immich-backup.log 2>&1

# Weekly local-only backup (no B2 upload) at 3:00 AM on Sundays
0 3 * * 0 /home/acedanger/shell/immich/backup-immich.sh --no-upload

# Daily validation run (dry-run) at 1:55 AM to check system health
55 1 * * * /home/acedanger/shell/immich/backup-immich.sh --dry-run >> /home/acedanger/shell/logs/immich-validation.log 2>&1

Manual Backup Scripts

#!/bin/bash
# emergency-backup.sh - Quick local backup without B2
echo "Starting emergency Immich backup..."
/home/acedanger/shell/immich/backup-immich.sh --no-upload --verbose

#!/bin/bash
# weekly-validation.sh - Comprehensive system check
echo "Validating Immich backup system..."
/home/acedanger/shell/immich/backup-immich.sh --dry-run --verbose

Troubleshooting Examples

Check System Status

# Quick system validation without backup
./backup-immich.sh --dry-run

# If containers are not running:
docker ps | grep immich  # Check container status
docker start immich_server immich_postgres  # Start if needed

# If upload directory missing:
ls -la /opt/immich/upload  # Verify path exists

Test B2 Configuration

# Backup without B2 to test local functionality
./backup-immich.sh --no-upload

# Check B2 CLI manually
./b2-linux version  # Verify B2 CLI works
./b2-linux authorize-account YOUR_KEY_ID YOUR_KEY  # Test authorization

Debug Backup Issues

# Run with maximum detail for troubleshooting
./backup-immich.sh --verbose --no-upload

# Check logs for errors
tail -f /home/acedanger/shell/logs/immich-backup.log

# Validate backup files
ls -la /home/acedanger/shell/immich_backups/

Configuration

The scripts expect a .env file in the parent directory with the following variables:

# Database configuration
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
UPLOAD_LOCATION=/path/to/immich/uploads

# Notification settings
WEBHOOK_URL="https://notify.peterwood.rocks/lab"

# Backblaze B2 settings (optional)
# Get these from your B2 account: https://secure.backblaze.com/app_keys.htm
# B2_APPLICATION_KEY_ID=your_key_id_here
# B2_APPLICATION_KEY=your_application_key_here
# B2_BUCKET_NAME=your_bucket_name_here

# Optional: Backup retention (days)
BACKUP_RETENTION_DAYS=30

Shared Storage Configuration

The backup script automatically uses shared storage at /mnt/share/media/backups/immich/ for permanent backup storage. This provides:

  • Centralized storage: All backups stored in one accessible location
  • Network accessibility: Backups available across your network
  • Space efficiency: Temporary local files are cleaned up after copying

Requirements:

  • /mnt/share/media/backups/ directory must be accessible and writable
  • Sufficient disk space for backup files (database + uploads archive)

Fallback behavior:

  • If shared storage is inaccessible, backups remain in local temporary directory
  • Script logs warnings but continues operation
  • B2 upload still functions from local files if configured

Backup Strategy

Workflow Process:

  1. Local Creation: Backups created in temporary directory (../immich_backups/)
  2. Shared Storage Copy: Files copied to /mnt/share/media/backups/immich/
  3. B2 Upload: Local files uploaded to Backblaze B2 (if configured)
  4. Local Cleanup: Temporary files deleted after successful copy to shared storage

Backup Content (based on Immich's official recommendations):

  1. Database Backup: Uses pg_dumpall with --clean and --if-exists flags

  2. Upload Directory: Complete archive of upload location including:

    • upload/ - Original photos and videos
    • profile/ - User profile images
    • thumbs/ - Generated thumbnails
    • encoded-video/ - Transcoded videos
    • library/ - Library metadata
    • backups/ - Existing backup files (excluded from new backups)

Safety Features:

  • Graceful degradation: If shared storage fails, keeps local backups
  • Error isolation: B2 upload failure doesn't affect local or shared storage
  • Container safety: Immich server paused during backup to ensure consistency

Notifications 🔔

The backup script sends notifications to your webhook URL with:

  • 🚀 Start notification: When backup begins
  • Success notification: When backup completes successfully with file sizes
  • ⚠️ Warning notification: When backup succeeds but B2 upload fails
  • 🚨 Error notification: When backup fails

Example notification:

📦 Database: immich_db_backup_20250526_215913.sql.gz (150MB)
📁 Uploads: immich_uploads_20250526_215913.tar.gz (25GB)
💾 Stored in: /mnt/share/media/backups/immich (local files cleaned up)
☁️ Successfully uploaded to B2 bucket: my-immich-backups

Backblaze B2 Integration ☁️

Setup B2 Account

  1. Create a Backblaze B2 account
  2. Create a new bucket for Immich backups
  3. Generate application keys:

Configure B2 in .env

Add these variables to your .env file:

B2_APPLICATION_KEY_ID=your_key_id_here
B2_APPLICATION_KEY=your_application_key_here
B2_BUCKET_NAME=your_bucket_name_here

B2 Features

  • Automatic upload: Backup files are uploaded to B2 after creation
  • Organized storage: Files stored in immich-backups/ folder in your bucket
  • Error handling: Script continues if B2 upload fails (local backup preserved)
  • Progress tracking: Upload status included in notifications

The B2 CLI tool (b2-linux) is included in this directory and doesn't require separate installation.

Restore Process

For complete restore instructions, see: https://immich.app/docs/administration/backup-and-restore/

Backup File Locations:

  • Primary: /mnt/share/media/backups/immich/ (shared storage)
  • Fallback: ../immich_backups/ (if shared storage copy failed)
  • Cloud: Backblaze B2 bucket (if B2 upload was configured)

Restore Steps:

  1. Database Restore:

    # From shared storage (primary location)
    docker exec -i immich_postgres psql -U postgres < /mnt/share/media/backups/immich/immich_db_backup_YYYYMMDD_HHMMSS.sql
    
    # OR from B2 download
    ./b2-linux download-file-by-name your-bucket immich-backups/immich_db_backup_YYYYMMDD_HHMMSS.sql.gz
    gunzip immich_db_backup_YYYYMMDD_HHMMSS.sql.gz
    docker exec -i immich_postgres psql -U postgres < immich_db_backup_YYYYMMDD_HHMMSS.sql
    
  2. Upload Directory Restore:

    # From shared storage (primary location)
    tar -xzf /mnt/share/media/backups/immich/immich_uploads_YYYYMMDD_HHMMSS.tar.gz -C /target/location
    
    # OR from B2 download
    ./b2-linux download-file-by-name your-bucket immich-backups/immich_uploads_YYYYMMDD_HHMMSS.tar.gz
    tar -xzf immich_uploads_YYYYMMDD_HHMMSS.tar.gz -C /target/location
    

restore-immich.sh

Complete restoration script for Immich installation that restores from backups created by backup-immich.sh:

  • PostgreSQL database restoration (with decompression and integrity checks)
  • User upload directories restoration (photos, videos, and metadata)
  • Container management (stop/start services during restoration)
  • Comprehensive validation and error handling

Requirements:

  • .env file in the parent directory (/home/acedanger/shell/.env) with:
    • DB_USERNAME - PostgreSQL username
    • DB_DATABASE_NAME - Database name
    • UPLOAD_LOCATION - Path to Immich upload directory
  • Docker containers: immich_postgres and immich_server
  • Valid backup files created by backup-immich.sh

Usage:

./restore-immich.sh --db-backup <path> --uploads-backup <path> [OPTIONS]

Required Arguments:

  • --db-backup PATH - Path to database backup file (.sql.gz)
  • --uploads-backup PATH - Path to uploads backup file (.tar.gz)

Optional Arguments:

  • --dry-run - Show what would be restored without making changes
  • --skip-db - Skip database restoration (uploads only)
  • --skip-uploads - Skip uploads restoration (database only)
  • --help - Show help message and exit

Examples:

# Complete restoration (recommended to run dry-run first)
./restore-immich.sh \
  --db-backup ../immich_backups/immich_db_backup_20250603_120000.sql.gz \
  --uploads-backup ../immich_backups/immich_uploads_20250603_120000.tar.gz \
  --dry-run

# Actual restoration after verifying dry-run
./restore-immich.sh \
  --db-backup ../immich_backups/immich_db_backup_20250603_120000.sql.gz \
  --uploads-backup ../immich_backups/immich_uploads_20250603_120000.tar.gz

# Restore database only
./restore-immich.sh \
  --db-backup ../immich_backups/immich_db_backup_20250603_120000.sql.gz \
  --uploads-backup ../immich_backups/immich_uploads_20250603_120000.tar.gz \
  --skip-uploads

# Restore uploads only
./restore-immich.sh \
  --db-backup ../immich_backups/immich_db_backup_20250603_120000.sql.gz \
  --uploads-backup ../immich_backups/immich_uploads_20250603_120000.tar.gz \
  --skip-db

Safety Features:

  • Dry-run mode: Preview all operations without making changes
  • File validation: Verifies backup file integrity before restoration
  • Container management: Safely stops/starts services during restoration
  • Cleanup on failure: Automatically restores services if restoration fails
  • Comprehensive logging: All operations logged to ../logs/immich-restore.log
  • Notification support: Sends webhook notifications for restoration status

validate-immich-backups.sh

Validation script that checks the integrity and completeness of Immich backup files:

  • File integrity validation (gzip and tar.gz corruption detection)
  • Backup pairing verification (matching database and upload backups)
  • Size and content validation
  • Restoration command generation for valid backups

Usage:

./validate-immich-backups.sh [backup_directory]

Arguments:

  • backup_directory - Directory containing backup files (default: ../immich_backups)

Examples:

# Validate backups in default directory
./validate-immich-backups.sh

# Validate backups in specific directory
./validate-immich-backups.sh /path/to/backups

# Example output for valid backups
=== IMMICH BACKUP VALIDATION ===
Backup directory: ../immich_backups

=== DATABASE BACKUPS ===
Validating: immich_db_backup_20250603_120000.sql.gz
  ✓ File size OK (45M)
  ✓ Gzip file integrity OK
  ✓ SQL content appears valid

=== UPLOAD BACKUPS ===
Validating: immich_uploads_20250603_120000.tar.gz
  ✓ File size OK (2.1G)
  ✓ Tar.gz file integrity OK
  ✓ Archive contains 15,234 files/directories

=== RESTORATION COMMANDS ===
To restore the most recent complete backup:

# Dry run (recommended first):
./restore-immich.sh \
  --db-backup "../immich_backups/immich_db_backup_20250603_120000.sql.gz" \
  --uploads-backup "../immich_backups/immich_uploads_20250603_120000.tar.gz" \
  --dry-run

Backup and Restore Workflow

1. Regular Backup

# Run daily backup (typically via cron)
./backup-immich.sh

2. Validate Backups

# Check backup integrity periodically
./validate-immich-backups.sh

3. Restoration Process

# Step 1: Validate available backups and get commands
./validate-immich-backups.sh

# Step 2: Test restoration with dry-run
./restore-immich.sh \
  --db-backup "path/to/db_backup.sql.gz" \
  --uploads-backup "path/to/uploads_backup.tar.gz" \
  --dry-run

# Step 3: Perform actual restoration
./restore-immich.sh \
  --db-backup "path/to/db_backup.sql.gz" \
  --uploads-backup "path/to/uploads_backup.tar.gz"

Advanced Usage

Partial Restoration

For specific restoration scenarios:

# Database only (preserve existing uploads)
./restore-immich.sh \
  --db-backup "db_backup.sql.gz" \
  --uploads-backup "uploads_backup.tar.gz" \
  --skip-uploads

# Uploads only (preserve existing database)
./restore-immich.sh \
  --db-backup "db_backup.sql.gz" \
  --uploads-backup "uploads_backup.tar.gz" \
  --skip-db

Emergency Recovery

In case of data loss or corruption:

  1. Stop Immich services: docker stop immich_server immich_postgres
  2. Validate backups: ./validate-immich-backups.sh
  3. Dry-run restoration: Use --dry-run first
  4. Restore data: Remove --dry-run flag
  5. Verify functionality: Check Immich web interface

Monitoring and Logs

All scripts log to the centralized logging directory:

  • Backup logs: ../logs/immich-backup.log
  • Restoration logs: ../logs/immich-restore.log
  • Validation logs: ../logs/immich-validation.log

Troubleshooting

Common Issues

  1. Container not found: Ensure Docker containers are running
  2. Permission denied: Check file permissions and Docker access
  3. Database connection failed: Verify PostgreSQL credentials in .env
  4. Insufficient disk space: Check available space before restoration
  5. Backup file corruption: Use validate-immich-backups.sh to check integrity

Recovery Steps

  1. Check logs in ../logs/ directory
  2. Verify .env file configuration
  3. Ensure Docker containers are accessible
  4. Validate backup file integrity
  5. Use dry-run mode to test operations

Environment Configuration

Required .env file structure:

# Database Configuration
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

# Upload Directory
UPLOAD_LOCATION=/path/to/immich/uploads

# Optional: Notification Settings
NTFY_URL=https://ntfy.sh/your-topic