shell/immich/README.md

# Immich Management Scripts

This directory contains scripts for managing and backing up Immich photo management system.

## Scripts

### backup-immich.sh

Complete backup script for Immich installation that creates backups of:

- PostgreSQL database (using pg_dumpall as recommended by Immich)
- User upload directories (photos, videos, and metadata)

**Requirements:**

- `.env` file in the parent directory (`/home/acedanger/shell/.env`) with:
  - `DB_USERNAME` - PostgreSQL username
  - `DB_DATABASE_NAME` - Database name
  - `UPLOAD_LOCATION` - Path to Immich upload directory
- Docker containers: `immich_postgres` and `immich_server`

**Usage:**

```bash
./backup-immich.sh [OPTIONS]
```

**Command-Line Options:**

- `--help, -h` - Show help message and exit
- `--dry-run` - Show what would be backed up without performing actual backup
- `--no-upload` - Skip B2 upload (local backup only)
- `--verbose` - Enable verbose logging

**Examples:**

```bash
# Standard backup (default behavior)
./backup-immich.sh

# Show help and usage information
./backup-immich.sh --help

# Preview what would be backed up without executing
./backup-immich.sh --dry-run

# Backup locally only (skip B2 upload)
./backup-immich.sh --no-upload

# Run with verbose logging
./backup-immich.sh --verbose

# Combine options
./backup-immich.sh --no-upload --verbose
```

**Backup Location:**

**Primary Storage:** `/mnt/share/media/backups/immich/` (shared storage)

- Database: `immich_db_backup_YYYYMMDD_HHMMSS.sql.gz`
- Uploads: `immich_uploads_YYYYMMDD_HHMMSS.tar.gz`

**Temporary Location:** `../immich_backups/` (cleaned up after copy to shared storage)

**Backup Workflow:**

1. **Create local backups** in temporary directory (`../immich_backups/`)
2. **Copy to shared storage** (`/mnt/share/media/backups/immich/`)
3. **Upload to Backblaze B2** (if configured)
4. **Delete local copies** (shared storage copies retained)

**Features:**

- **Smart backup workflow**: Creates → Copies to shared storage → Uploads to B2 → Cleans up locally
- Command-line options for flexible operation (--help, --dry-run, --no-upload, --verbose)
- Dry-run mode to preview operations without executing
- Option to skip B2 upload for local-only backups
- **Shared storage integration**: Automatically copies backups to `/mnt/share/media/backups/immich/`
- **Local cleanup**: Removes temporary files after successful copy to shared storage
- Automatic container pausing/resuming during backup
- Comprehensive error handling and cleanup
- **Graceful degradation**: Retains local backups if shared storage copy fails
- Backup validation and health checks
- Automatic compression
- Old backup cleanup (configurable)
- Centralized logging to `/home/acedanger/shell/logs/`
- Detailed progress reporting and timestamped logs
- 🔔 **Webhook notifications** to notify.peterwood.rocks/lab
- ☁️ **Backblaze B2 integration** for off-site backup storage
- 📊 **File size reporting** in notifications

## Usage Examples

### Basic Operations

**Standard Backup (Default)**

```bash
./backup-immich.sh
# Performs complete backup with all default settings:
# - Backs up database and upload directory
# - Uploads to B2 if configured
# - Sends webhook notifications
# - Logs to /home/acedanger/shell/logs/immich-backup.log
```

**Getting Help**

```bash
./backup-immich.sh --help
# Shows complete usage information including:
# - All available command-line options
# - Configuration requirements
# - Examples and restore instructions
```

### Preview and Testing

**Dry Run (Preview Mode)**

```bash
./backup-immich.sh --dry-run
# Shows what would be backed up without executing:
# - Checks all prerequisites and container status
# - Displays backup file paths and estimated sizes
# - Validates B2 configuration if present
# - Reports any issues that would prevent backup
# - No files are created or modified
```

Example dry-run output:

```text
=== DRY RUN MODE - NO ACTUAL BACKUP WILL BE PERFORMED ===

Configuration:
  - Database: immich
  - Username: postgres
  - Upload Location: /opt/immich/upload
  - Container: immich_postgres
  - Backup Directory: /home/acedanger/shell/immich_backups

Would create:
  - Database backup: /home/acedanger/shell/immich_backups/immich_db_backup_20250527_140000.sql.gz
  - Upload backup: /home/acedanger/shell/immich_backups/immich_uploads_20250527_140000.tar.gz

Workflow:
  1. Create local backups
  2. Copy to shared storage: /mnt/share/media/backups/immich/
  3. Upload to B2 (if configured)
  4. Delete local backups (keep shared copies)

Container Status Check:
  ✓ immich_server: Running (would pause during backup)
  ✓ immich_postgres: Running
  ✓ Upload directory: /opt/immich/upload (42GB)

B2 Upload Configuration:
  ✓ B2 configured - would upload to bucket: my-immich-backups
  ✓ B2 CLI found at: /home/acedanger/shell/immich/b2-linux

Shared Storage Check:
  ✓ Shared storage accessible: /mnt/share/media/backups
  ✓ Shared storage writable - would copy backups before B2 upload

=== DRY RUN COMPLETE - No files were created or modified ===
```

### Local Backup Only

**Skip B2 Upload**

```bash
./backup-immich.sh --no-upload
# Performs backup but skips B2 upload:
# - Creates local backup files
# - Validates backup integrity
# - Sends notifications (without B2 status)
# - Useful for testing or when B2 is unavailable
```

### Verbose Logging

**Detailed Output**

```bash
./backup-immich.sh --verbose
# Enables detailed logging for troubleshooting:
# - Shows additional progress information
# - Includes Docker command output
# - Provides more detailed error messages
# - Helpful for debugging issues
```

### Combined Options

**Local Backup with Verbose Output**

```bash
./backup-immich.sh --no-upload --verbose
# Combines multiple options:
# - Creates local backup only (no B2 upload)
# - Shows detailed progress and logging
# - Useful for testing or troubleshooting
```

**Preview with Verbose Details**

```bash
./backup-immich.sh --dry-run --verbose
# Shows detailed preview information:
# - Extended configuration validation
# - More detailed container status
# - Comprehensive B2 configuration check
# - Additional filesystem checks
```

### Automation Examples

**Scheduled Backup (Crontab)**

```bash
# Daily backup at 2:00 AM with logging
0 2 * * * /home/acedanger/shell/immich/backup-immich.sh >> /home/acedanger/shell/logs/immich-backup.log 2>&1

# Weekly local-only backup (no B2 upload) at 3:00 AM on Sundays
0 3 * * 0 /home/acedanger/shell/immich/backup-immich.sh --no-upload

# Daily validation run (dry-run) at 1:55 AM to check system health
55 1 * * * /home/acedanger/shell/immich/backup-immich.sh --dry-run >> /home/acedanger/shell/logs/immich-validation.log 2>&1
```

**Manual Backup Scripts**

```bash
#!/bin/bash
# emergency-backup.sh - Quick local backup without B2
echo "Starting emergency Immich backup..."
/home/acedanger/shell/immich/backup-immich.sh --no-upload --verbose

#!/bin/bash
# weekly-validation.sh - Comprehensive system check
echo "Validating Immich backup system..."
/home/acedanger/shell/immich/backup-immich.sh --dry-run --verbose
```

### Troubleshooting Examples

**Check System Status**

```bash
# Quick system validation without backup
./backup-immich.sh --dry-run

# If containers are not running:
docker ps | grep immich  # Check container status
docker start immich_server immich_postgres  # Start if needed

# If upload directory missing:
ls -la /opt/immich/upload  # Verify path exists
```

**Test B2 Configuration**

```bash
# Backup without B2 to test local functionality
./backup-immich.sh --no-upload

# Check B2 CLI manually
./b2-linux version  # Verify B2 CLI works
./b2-linux authorize-account YOUR_KEY_ID YOUR_KEY  # Test authorization
```

**Debug Backup Issues**

```bash
# Run with maximum detail for troubleshooting
./backup-immich.sh --verbose --no-upload

# Check logs for errors
tail -f /home/acedanger/shell/logs/immich-backup.log

# Validate backup files
ls -la /home/acedanger/shell/immich_backups/
```

## Configuration

The scripts expect a `.env` file in the parent directory with the following variables:

```bash
# Database configuration
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
UPLOAD_LOCATION=/path/to/immich/uploads

# Notification settings
WEBHOOK_URL="https://notify.peterwood.rocks/lab"

# Backblaze B2 settings (optional)
# Get these from your B2 account: https://secure.backblaze.com/app_keys.htm
# B2_APPLICATION_KEY_ID=your_key_id_here
# B2_APPLICATION_KEY=your_application_key_here
# B2_BUCKET_NAME=your_bucket_name_here

# Optional: Backup retention (days)
BACKUP_RETENTION_DAYS=30
```

## Shared Storage Configuration

The backup script automatically uses shared storage at `/mnt/share/media/backups/immich/` for permanent backup storage. This provides:

- **Centralized storage**: All backups stored in one accessible location
- **Network accessibility**: Backups available across your network
- **Space efficiency**: Temporary local files are cleaned up after copying

**Requirements:**

- `/mnt/share/media/backups/` directory must be accessible and writable
- Sufficient disk space for backup files (database + uploads archive)

**Fallback behavior:**

- If shared storage is inaccessible, backups remain in local temporary directory
- Script logs warnings but continues operation
- B2 upload still functions from local files if configured

## Backup Strategy

**Workflow Process:**

1. **Local Creation**: Backups created in temporary directory (`../immich_backups/`)
2. **Shared Storage Copy**: Files copied to `/mnt/share/media/backups/immich/`
3. **B2 Upload**: Local files uploaded to Backblaze B2 (if configured)
4. **Local Cleanup**: Temporary files deleted after successful copy to shared storage

**Backup Content** (based on Immich's official recommendations):

1. **Database Backup**: Uses `pg_dumpall` with `--clean` and `--if-exists` flags
2. **Upload Directory**: Complete archive of upload location including:

   - upload/ - Original photos and videos
   - profile/ - User profile images
   - thumbs/ - Generated thumbnails
   - encoded-video/ - Transcoded videos
   - library/ - Library metadata
   - backups/ - Existing backup files (excluded from new backups)

**Safety Features:**

- **Graceful degradation**: If shared storage fails, keeps local backups
- **Error isolation**: B2 upload failure doesn't affect local or shared storage
- **Container safety**: Immich server paused during backup to ensure consistency

## Notifications 🔔

The backup script sends notifications to your webhook URL with:

- 🚀 **Start notification**: When backup begins
- ✅ **Success notification**: When backup completes successfully with file sizes
- ⚠️ **Warning notification**: When backup succeeds but B2 upload fails
- 🚨 **Error notification**: When backup fails

Example notification:

```text
📦 Database: immich_db_backup_20250526_215913.sql.gz (150MB)
📁 Uploads: immich_uploads_20250526_215913.tar.gz (25GB)
💾 Stored in: /mnt/share/media/backups/immich (local files cleaned up)
☁️ Successfully uploaded to B2 bucket: my-immich-backups
```

## Backblaze B2 Integration ☁️

### Setup B2 Account

1. Create a [Backblaze B2 account](https://www.backblaze.com/b2/cloud-storage.html)
2. Create a new bucket for Immich backups
3. Generate application keys:
   - Go to: <https://secure.backblaze.com/app_keys.htm>
   - Create new key with read/write access to your bucket

### Configure B2 in .env

Add these variables to your `.env` file:

```bash
B2_APPLICATION_KEY_ID=your_key_id_here
B2_APPLICATION_KEY=your_application_key_here
B2_BUCKET_NAME=your_bucket_name_here
```

### B2 Features

- **Automatic upload**: Backup files are uploaded to B2 after creation
- **Organized storage**: Files stored in `immich-backups/` folder in your bucket
- **Error handling**: Script continues if B2 upload fails (local backup preserved)
- **Progress tracking**: Upload status included in notifications

The B2 CLI tool (`b2-linux`) is included in this directory and doesn't require separate installation.

## Restore Process

For complete restore instructions, see: <https://immich.app/docs/administration/backup-and-restore/>

**Backup File Locations:**

- **Primary**: `/mnt/share/media/backups/immich/` (shared storage)
- **Fallback**: `../immich_backups/` (if shared storage copy failed)
- **Cloud**: Backblaze B2 bucket (if B2 upload was configured)

**Restore Steps:**

1. **Database Restore:**

   ```bash
   # From shared storage (primary location)
   docker exec -i immich_postgres psql -U postgres < /mnt/share/media/backups/immich/immich_db_backup_YYYYMMDD_HHMMSS.sql

   # OR from B2 download
   ./b2-linux download-file-by-name your-bucket immich-backups/immich_db_backup_YYYYMMDD_HHMMSS.sql.gz
   gunzip immich_db_backup_YYYYMMDD_HHMMSS.sql.gz
   docker exec -i immich_postgres psql -U postgres < immich_db_backup_YYYYMMDD_HHMMSS.sql
   ```

2. **Upload Directory Restore:**

   ```bash
   # From shared storage (primary location)
   tar -xzf /mnt/share/media/backups/immich/immich_uploads_YYYYMMDD_HHMMSS.tar.gz -C /target/location

   # OR from B2 download
   ./b2-linux download-file-by-name your-bucket immich-backups/immich_uploads_YYYYMMDD_HHMMSS.tar.gz
   tar -xzf immich_uploads_YYYYMMDD_HHMMSS.tar.gz -C /target/location
   ```

### restore-immich.sh

Complete restoration script for Immich installation that restores from backups created by `backup-immich.sh`:

- PostgreSQL database restoration (with decompression and integrity checks)
- User upload directories restoration (photos, videos, and metadata)
- Container management (stop/start services during restoration)
- Comprehensive validation and error handling

**Requirements:**

- `.env` file in the parent directory (`/home/acedanger/shell/.env`) with:
  - `DB_USERNAME` - PostgreSQL username
  - `DB_DATABASE_NAME` - Database name
  - `UPLOAD_LOCATION` - Path to Immich upload directory
- Docker containers: `immich_postgres` and `immich_server`
- Valid backup files created by `backup-immich.sh`

**Usage:**

```bash
./restore-immich.sh --db-backup <path> --uploads-backup <path> [OPTIONS]
```

**Required Arguments:**

- `--db-backup PATH` - Path to database backup file (.sql.gz)
- `--uploads-backup PATH` - Path to uploads backup file (.tar.gz)

**Optional Arguments:**

- `--dry-run` - Show what would be restored without making changes
- `--skip-db` - Skip database restoration (uploads only)
- `--skip-uploads` - Skip uploads restoration (database only)
- `--help` - Show help message and exit

**Examples:**

```bash
# Complete restoration (recommended to run dry-run first)
./restore-immich.sh \
  --db-backup ../immich_backups/immich_db_backup_20250603_120000.sql.gz \
  --uploads-backup ../immich_backups/immich_uploads_20250603_120000.tar.gz \
  --dry-run

# Actual restoration after verifying dry-run
./restore-immich.sh \
  --db-backup ../immich_backups/immich_db_backup_20250603_120000.sql.gz \
  --uploads-backup ../immich_backups/immich_uploads_20250603_120000.tar.gz

# Restore database only
./restore-immich.sh \
  --db-backup ../immich_backups/immich_db_backup_20250603_120000.sql.gz \
  --uploads-backup ../immich_backups/immich_uploads_20250603_120000.tar.gz \
  --skip-uploads

# Restore uploads only
./restore-immich.sh \
  --db-backup ../immich_backups/immich_db_backup_20250603_120000.sql.gz \
  --uploads-backup ../immich_backups/immich_uploads_20250603_120000.tar.gz \
  --skip-db
```

**Safety Features:**

- **Dry-run mode**: Preview all operations without making changes
- **File validation**: Verifies backup file integrity before restoration
- **Container management**: Safely stops/starts services during restoration
- **Cleanup on failure**: Automatically restores services if restoration fails
- **Comprehensive logging**: All operations logged to `../logs/immich-restore.log`
- **Notification support**: Sends webhook notifications for restoration status

### validate-immich-backups.sh

Validation script that checks the integrity and completeness of Immich backup files:

- File integrity validation (gzip and tar.gz corruption detection)
- Backup pairing verification (matching database and upload backups)
- Size and content validation
- Restoration command generation for valid backups

**Usage:**

```bash
./validate-immich-backups.sh [backup_directory]
```

**Arguments:**

- `backup_directory` - Directory containing backup files (default: `../immich_backups`)

**Examples:**

```bash
# Validate backups in default directory
./validate-immich-backups.sh

# Validate backups in specific directory
./validate-immich-backups.sh /path/to/backups

# Example output for valid backups
=== IMMICH BACKUP VALIDATION ===
Backup directory: ../immich_backups

=== DATABASE BACKUPS ===
Validating: immich_db_backup_20250603_120000.sql.gz
  ✓ File size OK (45M)
  ✓ Gzip file integrity OK
  ✓ SQL content appears valid

=== UPLOAD BACKUPS ===
Validating: immich_uploads_20250603_120000.tar.gz
  ✓ File size OK (2.1G)
  ✓ Tar.gz file integrity OK
  ✓ Archive contains 15,234 files/directories

=== RESTORATION COMMANDS ===
To restore the most recent complete backup:

# Dry run (recommended first):
./restore-immich.sh \
  --db-backup "../immich_backups/immich_db_backup_20250603_120000.sql.gz" \
  --uploads-backup "../immich_backups/immich_uploads_20250603_120000.tar.gz" \
  --dry-run
```

## Backup and Restore Workflow

### 1. Regular Backup

```bash
# Run daily backup (typically via cron)
./backup-immich.sh
```

### 2. Validate Backups

```bash
# Check backup integrity periodically
./validate-immich-backups.sh
```

### 3. Restoration Process

```bash
# Step 1: Validate available backups and get commands
./validate-immich-backups.sh

# Step 2: Test restoration with dry-run
./restore-immich.sh \
  --db-backup "path/to/db_backup.sql.gz" \
  --uploads-backup "path/to/uploads_backup.tar.gz" \
  --dry-run

# Step 3: Perform actual restoration
./restore-immich.sh \
  --db-backup "path/to/db_backup.sql.gz" \
  --uploads-backup "path/to/uploads_backup.tar.gz"
```

## Advanced Usage

### Partial Restoration

For specific restoration scenarios:

```bash
# Database only (preserve existing uploads)
./restore-immich.sh \
  --db-backup "db_backup.sql.gz" \
  --uploads-backup "uploads_backup.tar.gz" \
  --skip-uploads

# Uploads only (preserve existing database)
./restore-immich.sh \
  --db-backup "db_backup.sql.gz" \
  --uploads-backup "uploads_backup.tar.gz" \
  --skip-db
```

### Emergency Recovery

In case of data loss or corruption:

1. **Stop Immich services**: `docker stop immich_server immich_postgres`
2. **Validate backups**: `./validate-immich-backups.sh`
3. **Dry-run restoration**: Use `--dry-run` first
4. **Restore data**: Remove `--dry-run` flag
5. **Verify functionality**: Check Immich web interface

### Monitoring and Logs

All scripts log to the centralized logging directory:

- **Backup logs**: `../logs/immich-backup.log`
- **Restoration logs**: `../logs/immich-restore.log`
- **Validation logs**: `../logs/immich-validation.log`

## Troubleshooting

### Common Issues

1. **Container not found**: Ensure Docker containers are running
2. **Permission denied**: Check file permissions and Docker access
3. **Database connection failed**: Verify PostgreSQL credentials in `.env`
4. **Insufficient disk space**: Check available space before restoration
5. **Backup file corruption**: Use `validate-immich-backups.sh` to check integrity

### Recovery Steps

1. Check logs in `../logs/` directory
2. Verify `.env` file configuration
3. Ensure Docker containers are accessible
4. Validate backup file integrity
5. Use dry-run mode to test operations

## Environment Configuration

Required `.env` file structure:

```bash
# Database Configuration
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

# Upload Directory
UPLOAD_LOCATION=/path/to/immich/uploads

# Optional: Notification Settings
NTFY_URL=https://ntfy.sh/your-topic
```