mirror of
https://github.com/acedanger/shell.git
synced 2025-12-06 02:20:11 -08:00
Refactor documentation and enhance Immich backup system
- Updated README.md to streamline content and improve navigation with a new Quick Navigation section. - Consolidated detailed Plex backup descriptions into dedicated documentation files. - Added comprehensive Docker Bootstrap Testing Framework documentation. - Created Immich backup enhancement summary and migration summary documents. - Implemented webhook notifications and Backblaze B2 integration in Immich backup scripts. - Centralized logging for Immich backup operations and updated configuration requirements. - Restructured Telegram bot project documentation for better clarity and organization. - Enhanced .gitignore to include environment files and Backblaze CLI tool. - Updated dotfiles README to reference new testing documentation. Resolves Documentation review #11.
This commit is contained in:
Peter Wood
14
.env.example
Normal file
14
.env.example
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
# Immich Configuration for Testing
|
||||||
|
DB_USERNAME=postgres
|
||||||
|
DB_DATABASE_NAME=immich
|
||||||
|
UPLOAD_LOCATION=/mnt/share/media/immich/uploads
|
||||||
|
|
||||||
|
# Notification settings
|
||||||
|
WEBHOOK_URL="https://notify.peterwood.rocks/lab"
|
||||||
|
|
||||||
|
# Backblaze B2 settings
|
||||||
|
# Get these from your B2 account: https://secure.backblaze.com/app_keys.htm
|
||||||
|
K005YB4icG3edh5Z9o64ieXvepEYWoA
|
||||||
|
# B2_APPLICATION_KEY_ID=your_key_id_here
|
||||||
|
# B2_APPLICATION_KEY=your_application_key_here
|
||||||
|
# B2_BUCKET_NAME=your_bucket_name_here
|
||||||
2
.github/copilot-instructions.md
vendored
2
.github/copilot-instructions.md
vendored
@@ -39,7 +39,7 @@ This repository contains:
|
|||||||
### Documentation
|
### Documentation
|
||||||
|
|
||||||
- **README.md**: Main repository documentation
|
- **README.md**: Main repository documentation
|
||||||
- **docs/testing.md**: Detailed documentation for the testing framework
|
- **docs/docker-bootstrap-testing-framework.md**: Detailed documentation for the Docker-based bootstrap validation framework
|
||||||
- **dotfiles/README.md**: Documentation for dotfiles setup and usage
|
- **dotfiles/README.md**: Documentation for dotfiles setup and usage
|
||||||
|
|
||||||
## Style Guidelines
|
## Style Guidelines
|
||||||
|
|||||||
8
.gitignore
vendored
8
.gitignore
vendored
@@ -18,8 +18,12 @@ _book
|
|||||||
# swap file
|
# swap file
|
||||||
*.swp
|
*.swp
|
||||||
|
|
||||||
|
# environment files
|
||||||
|
.env
|
||||||
|
|
||||||
# Runtime generated files
|
# Runtime generated files
|
||||||
logs/
|
logs/
|
||||||
|
immich_backups/*.gz
|
||||||
# Backup files - ignore most backups but keep current state files
|
# Backup files - ignore most backups but keep current state files
|
||||||
crontab/crontab-backups/*/archive/
|
crontab/crontab-backups/*/archive/
|
||||||
!crontab/crontab-backups/*/current-crontab.backup
|
!crontab/crontab-backups/*/current-crontab.backup
|
||||||
@@ -27,3 +31,7 @@ crontab/crontab-backups/*/archive/
|
|||||||
# Temporary files
|
# Temporary files
|
||||||
*.tmp
|
*.tmp
|
||||||
*.backup
|
*.backup
|
||||||
|
|
||||||
|
# backblaze cli <https://www.backblaze.com/docs/cloud-storage-command-line-tools>
|
||||||
|
# can be downloaded from <https://github.com/Backblaze/B2_Command_Line_Tool/releases/latest/download/b2-linux>
|
||||||
|
immich/b2-linux
|
||||||
|
|||||||
237
README.md
237
README.md
@@ -2,6 +2,14 @@
|
|||||||
|
|
||||||
This repository contains various shell scripts for managing media-related tasks and dotfiles for system configuration.
|
This repository contains various shell scripts for managing media-related tasks and dotfiles for system configuration.
|
||||||
|
|
||||||
|
## Quick Navigation
|
||||||
|
|
||||||
|
- **[Backup Scripts](#backup-scripts)** - Enterprise-grade backup solutions
|
||||||
|
- **[Management Scripts](#management-scripts)** - System and service management
|
||||||
|
- **[Documentation](#documentation)** - Complete guides and references
|
||||||
|
- **[Testing](#testing)** - Docker-based validation framework
|
||||||
|
- **[Dotfiles](#dotfiles)** - System configuration files
|
||||||
|
|
||||||
## Available Scripts
|
## Available Scripts
|
||||||
|
|
||||||
### Backup Scripts
|
### Backup Scripts
|
||||||
@@ -16,6 +24,10 @@ This repository contains various shell scripts for managing media-related tasks
|
|||||||
- **`plex.sh`**: Script to manage the Plex Media Server (start, stop, restart, status).
|
- **`plex.sh`**: Script to manage the Plex Media Server (start, stop, restart, status).
|
||||||
- **`folder-metrics.sh`**: Script to calculate disk usage and file count for a directory and its subdirectories.
|
- **`folder-metrics.sh`**: Script to calculate disk usage and file count for a directory and its subdirectories.
|
||||||
|
|
||||||
|
### Development Projects
|
||||||
|
|
||||||
|
- **[Telegram Backup Monitoring Bot](./telegram/github-issues/README.md)**: Comprehensive Telegram bot project for monitoring and managing all backup systems with real-time notifications and control capabilities.
|
||||||
|
|
||||||
### Testing Scripts
|
### Testing Scripts
|
||||||
|
|
||||||
- **`test-setup.sh`**: Validates the bootstrap and setup process.
|
- **`test-setup.sh`**: Validates the bootstrap and setup process.
|
||||||
@@ -80,23 +92,7 @@ The script includes configurable parameters:
|
|||||||
|
|
||||||
### Advanced Plex Backup System
|
### Advanced Plex Backup System
|
||||||
|
|
||||||
Specialized backup system for Plex Media Server with database-aware features:
|
Specialized backup system for Plex Media Server with database-aware features. For complete documentation, see [Plex Scripts Documentation](./plex/README.md).
|
||||||
|
|
||||||
### Components
|
|
||||||
|
|
||||||
- **`backup-plex.sh`**: Advanced backup script with integrity verification, incremental backups, and automatic cleanup
|
|
||||||
- **`restore-plex.sh`**: Safe restoration script with dry-run mode and current data backup
|
|
||||||
- **`validate-plex-backups.sh`**: Backup validation and health monitoring script
|
|
||||||
|
|
||||||
### Plex-Specific Features
|
|
||||||
|
|
||||||
- **Incremental backups**: Only backs up files that have changed since last backup
|
|
||||||
- **Database integrity verification**: Uses MD5 checksums to verify backup integrity
|
|
||||||
- **Automatic cleanup**: Configurable retention policies for old backups
|
|
||||||
- **Disk space monitoring**: Checks available space before starting backup
|
|
||||||
- **Safe restoration**: Backs up current data before restoring from backup
|
|
||||||
- **Comprehensive logging**: Detailed logs with color-coded output
|
|
||||||
- **Service management**: Safely stops/starts Plex during backup operations
|
|
||||||
|
|
||||||
## Backup Usage Examples
|
## Backup Usage Examples
|
||||||
|
|
||||||
@@ -168,93 +164,32 @@ Specialized backup system for Plex Media Server with database-aware features:
|
|||||||
|
|
||||||
## Backup Configuration and Strategy
|
## Backup Configuration and Strategy
|
||||||
|
|
||||||
### Media Backup Configuration
|
For detailed configuration information and backup strategies, see:
|
||||||
|
|
||||||
The enhanced media backup script includes configurable parameters at the top of the file:
|
- **[Plex Backup Configuration](./plex/README.md#configuration)**: Plex-specific settings and options
|
||||||
|
- **[Immich Backup Configuration](./immich/README.md#configuration)**: Database and upload directory settings
|
||||||
- `PARALLEL_JOBS=4`: Number of parallel rsync processes
|
- **[Enhanced Media Backup](./docs/enhanced-media-backup.md)**: Media backup script configuration
|
||||||
- `MAX_BACKUP_AGE_DAYS=90`: Remove backups older than 90 days
|
- **[Crontab Management](./crontab/README.md)**: Automated scheduling and system-specific configurations
|
||||||
- `LOG_RETENTION_DAYS=30`: Keep logs for 30 days
|
|
||||||
- `BACKUP_ROOT`: Default location for backup storage
|
|
||||||
- `LOG_ROOT`: Location for backup logs and reports
|
|
||||||
|
|
||||||
### Plex Backup Configuration
|
|
||||||
|
|
||||||
The Plex backup script configuration parameters:
|
|
||||||
|
|
||||||
- `MAX_BACKUP_AGE_DAYS=30`: Remove backups older than 30 days
|
|
||||||
- `MAX_BACKUPS_TO_KEEP=10`: Keep maximum of 10 backup archives
|
|
||||||
- `BACKUP_ROOT`: Location for compressed backup archives
|
|
||||||
- `LOG_ROOT`: Location for backup logs
|
|
||||||
|
|
||||||
### Final Backup Directory Structure
|
|
||||||
|
|
||||||
The enhanced Plex backup system creates a streamlined archive-only structure:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
/mnt/share/media/backups/plex/
|
|
||||||
├── plex-backup-20250125_143022.tar.gz # Latest backup
|
|
||||||
├── plex-backup-20250124_143011.tar.gz # Previous backup
|
|
||||||
├── plex-backup-20250123_143008.tar.gz # Older backup
|
|
||||||
└── logs/
|
|
||||||
├── backup_log_20250125_143022.md
|
|
||||||
├── plex-backup-performance.json
|
|
||||||
└── plex-backup.json
|
|
||||||
```
|
|
||||||
|
|
||||||
**Key Benefits:**
|
|
||||||
|
|
||||||
- **Direct Archive Storage**: No intermediate directories required
|
|
||||||
- **Efficient Space Usage**: Only compressed files stored permanently
|
|
||||||
- **Easy Management**: Timestamp-based naming for clear identification
|
|
||||||
- **Automatic Cleanup**: Legacy dated directories removed automatically
|
|
||||||
|
|
||||||
### Recommended Backup Strategy
|
|
||||||
|
|
||||||
Both systems implement a robust backup strategy following industry best practices:
|
|
||||||
|
|
||||||
**For Media Files:**
|
|
||||||
|
|
||||||
1. **Daily incremental backups** with parallel processing for speed
|
|
||||||
2. **Weekly verification** of backup integrity
|
|
||||||
3. **Monthly cleanup** of old backups based on retention policies
|
|
||||||
4. **Quarterly offsite sync** for disaster recovery
|
|
||||||
|
|
||||||
**For Plex Database:**
|
|
||||||
|
|
||||||
1. **Daily full backups** with service-aware operations
|
|
||||||
2. **Immediate validation** after each backup
|
|
||||||
3. **Weekly comprehensive reports** on backup health
|
|
||||||
4. **Monthly testing** of restore procedures
|
|
||||||
|
|
||||||
### Offsite Backup Integration
|
|
||||||
|
|
||||||
For comprehensive disaster recovery, sync backups to remote locations:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Sync media backups to remote server daily at 5 AM
|
|
||||||
0 5 * * * rsync -av /mnt/share/media/backups/media/ user@remote-server:/backups/media/
|
|
||||||
|
|
||||||
# Sync Plex backups to remote server daily at 6 AM
|
|
||||||
0 6 * * * rsync -av /mnt/share/media/backups/plex/ user@remote-server:/backups/plex/
|
|
||||||
```
|
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
### Directory-Specific Documentation
|
### Component-Specific Documentation
|
||||||
|
|
||||||
- **[Plex Scripts Documentation](./plex/README.md)**: Comprehensive documentation for all Plex backup, restoration, validation, and management scripts.
|
- **[Plex Scripts Documentation](./plex/README.md)**: Comprehensive documentation for Plex backup, restoration, validation, and management scripts
|
||||||
- **[Crontab Management Documentation](./crontab/README.md)**: Complete guide for crontab management, backup systems, and multi-system administration.
|
- **[Immich Scripts Documentation](./immich/README.md)**: Complete guide for Immich backup and management system with PostgreSQL and B2 integration
|
||||||
|
- **[Crontab Management Documentation](./crontab/README.md)**: Complete guide for crontab management, backup systems, and multi-system administration
|
||||||
|
- **[Dotfiles Documentation](./dotfiles/README.md)**: System configuration files and setup instructions
|
||||||
|
|
||||||
### Backup System Documentation
|
### Technical Documentation
|
||||||
|
|
||||||
- [Enhanced Media Backup Documentation](./docs/enhanced-media-backup.md): Comprehensive guide for the enhanced `backup-media.sh` script with enterprise features.
|
- **[Enhanced Media Backup Documentation](./docs/enhanced-media-backup.md)**: Comprehensive guide for the enhanced `backup-media.sh` script with enterprise features
|
||||||
- [Media Backup Enhancement Summary](./docs/backup-media-enhancement-summary.md): Summary of enhancements and feature comparisons.
|
- **[Basic Media Backup Documentation](./docs/backup-media.md)**: Documentation for the original `backup-media.sh` script
|
||||||
|
- **[Media Backup Enhancement Summary](./docs/backup-media-enhancement-summary.md)**: Summary of enhancements and feature comparisons
|
||||||
### Script Documentation
|
- **[Immich Backup Enhancement Summary](./docs/immich-backup-enhancement-summary.md)**: Details of Immich backup system improvements
|
||||||
|
- **[Docker Bootstrap Testing Framework](./docs/docker-bootstrap-testing-framework.md)**: Detailed documentation for the Docker-based bootstrap validation system
|
||||||
- [Folder Metrics Script Documentation](./docs/folder-metrics.md): Detailed documentation for the `folder-metrics.sh` script.
|
- **[Folder Metrics Script Documentation](./docs/folder-metrics.md)**: Detailed documentation for the `folder-metrics.sh` script
|
||||||
- [Testing Framework Documentation](./docs/testing.md): Detailed documentation for the Docker-based testing system.
|
- **[Production Deployment Guide](./docs/production-deployment-guide.md)**: Complete deployment and operations guide
|
||||||
|
- **[Project Completion Summary](./docs/project-completion-summary.md)**: Overview of completed features and project milestones
|
||||||
|
|
||||||
## Dotfiles
|
## Dotfiles
|
||||||
|
|
||||||
@@ -268,59 +203,21 @@ For more information about the dotfiles, see [Dotfiles README](./dotfiles/README
|
|||||||
|
|
||||||
## Testing
|
## Testing
|
||||||
|
|
||||||
This repository includes Docker-based testing to validate the setup process across different environments:
|
This repository includes Docker-based testing to validate the setup process across different environments. For complete testing documentation, see [Docker Bootstrap Testing Framework](./docs/docker-bootstrap-testing-framework.md).
|
||||||
|
|
||||||
- **test-setup.sh**: Script that validates the bootstrap and setup process
|
### Quick Testing
|
||||||
- **run-docker-tests.sh**: Runner script that executes tests in Docker containers
|
|
||||||
- **Dockerfile**: Defines test environments (Ubuntu, Debian)
|
|
||||||
|
|
||||||
### Running Tests
|
|
||||||
|
|
||||||
Test your setup in isolated Docker containers with:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Test in Ubuntu container
|
# Test in Ubuntu container
|
||||||
./run-docker-tests.sh ubuntu
|
./setup/run-docker-tests.sh ubuntu
|
||||||
|
|
||||||
# Test in Debian container
|
# Test in Debian container
|
||||||
./run-docker-tests.sh debian
|
./setup/run-docker-tests.sh debian
|
||||||
|
|
||||||
# Run full bootstrap test in Ubuntu
|
# Test in both environments
|
||||||
./run-docker-tests.sh full-ubuntu
|
./setup/run-docker-tests.sh all
|
||||||
|
|
||||||
# Run full bootstrap test in Debian
|
|
||||||
./run-docker-tests.sh full-debian
|
|
||||||
|
|
||||||
# Test in both Ubuntu and Debian
|
|
||||||
./run-docker-tests.sh all
|
|
||||||
```
|
```
|
||||||
|
|
||||||
#### When to Use Each Testing Option
|
|
||||||
|
|
||||||
- **Use `./run-docker-tests.sh ubuntu` or `./run-docker-tests.sh debian`** when you want to:
|
|
||||||
- Quickly validate if packages in packages.list are available and installed
|
|
||||||
- Test if your test-setup.sh script is working correctly
|
|
||||||
- Check for issues with specific components without performing a full setup
|
|
||||||
|
|
||||||
- **Use `./run-docker-tests.sh full-ubuntu` or `./run-docker-tests.sh full-debian`** when you want to:
|
|
||||||
- Test the complete bootstrap installation process end-to-end
|
|
||||||
- Validate that all installation steps work correctly on a fresh system
|
|
||||||
- Simulate what users will experience when running the bootstrap script
|
|
||||||
|
|
||||||
- **Use `./run-docker-tests.sh all`** when you want to:
|
|
||||||
- Ensure your test-setup.sh works across both Ubuntu and Debian
|
|
||||||
- Run comprehensive checks before committing changes
|
|
||||||
|
|
||||||
The test environment checks:
|
|
||||||
|
|
||||||
- Package availability and installation
|
|
||||||
- Core components (git, curl, wget, etc.)
|
|
||||||
- Additional packages from `setup/packages.list`
|
|
||||||
- Oh My Zsh and plugin installation
|
|
||||||
- Dotfile symlinks
|
|
||||||
|
|
||||||
Tests will continue even when some packages fail to install, reporting all issues in a comprehensive summary.
|
|
||||||
|
|
||||||
## plex.sh
|
## plex.sh
|
||||||
|
|
||||||
This script is used to manage the Plex Media Server service on a systemd-based Linux distribution. It provides the following functionalities:
|
This script is used to manage the Plex Media Server service on a systemd-based Linux distribution. It provides the following functionalities:
|
||||||
@@ -340,61 +237,21 @@ Note that these commands will run as `root`.
|
|||||||
|
|
||||||
## Enhanced Crontab System Migration
|
## Enhanced Crontab System Migration
|
||||||
|
|
||||||
### System Status
|
For complete crontab management documentation, see [Crontab Management Documentation](./crontab/README.md).
|
||||||
|
|
||||||
The crontab system has been migrated from a universal `crontab.txt` to system-specific files:
|
### Current Status
|
||||||
|
|
||||||
|
The crontab system uses hostname-specific configuration files:
|
||||||
- **crontab-europa.txt** - Media server configuration
|
- **crontab-europa.txt** - Media server configuration
|
||||||
- **crontab-io.txt** - Download/acquisition server configuration
|
- **crontab-io.txt** - Download/acquisition server configuration
|
||||||
- **crontab-racknerd.txt** - Backup server configuration
|
- **crontab-racknerd.txt** - Backup server configuration
|
||||||
|
|
||||||
### Legacy File Status
|
### Quick Management
|
||||||
|
|
||||||
The original `crontab.txt` has been:
|
|
||||||
|
|
||||||
- **Backed up** to `crontab.txt.bak`
|
|
||||||
- **Role**: Previously served as fallback, now obsolete
|
|
||||||
- **Content**: Contains mixed configuration that was split into system-specific files
|
|
||||||
|
|
||||||
### Current Management
|
|
||||||
|
|
||||||
Use the system-specific approach:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Install system-specific crontab
|
# Install system-specific crontab (auto-detects hostname)
|
||||||
./manage-enhanced-crontab.sh install
|
./crontab/manage-enhanced-crontab.sh install
|
||||||
|
|
||||||
# The script automatically detects hostname and uses appropriate file
|
# View current system's crontab configuration
|
||||||
# europa -> crontab-europa.txt
|
./crontab/manage-enhanced-crontab.sh show
|
||||||
# io -> crontab-io.txt
|
|
||||||
# racknerd -> crontab-racknerd.txt
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Fallback Behavior
|
|
||||||
|
|
||||||
If a system-specific file is missing, the management script will:
|
|
||||||
|
|
||||||
1. **Warning**: Display that system-specific file is not found
|
|
||||||
2. **Recommendation**: Create appropriate system-specific file
|
|
||||||
|
|
||||||
### Creating New System Files
|
|
||||||
|
|
||||||
For new systems, create system-specific files based on the templates:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Copy and customize for new system
|
|
||||||
cp crontab-europa.txt crontab-newsystem.txt
|
|
||||||
# Edit for system-specific tasks
|
|
||||||
```
|
|
||||||
|
|
||||||
### Benefits of System-Specific Approach
|
|
||||||
|
|
||||||
- **Clear separation** of concerns between systems
|
|
||||||
- **Reduced conflicts** from universal configurations
|
|
||||||
- **System-appropriate** task scheduling
|
|
||||||
- **Better maintenance** and troubleshooting
|
|
||||||
- **Scalable** for additional systems
|
|
||||||
|
|
||||||
#### Migration Notes
|
|
||||||
|
|
||||||
Migration completed: May 26, 2025
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Shell Setup Testing Framework
|
# Docker Bootstrap Testing Framework
|
||||||
|
|
||||||
This document describes the testing framework for validating the shell setup across different environments.
|
This document describes the comprehensive Docker-based testing framework for validating the bootstrap and setup process across different environments.
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
165
docs/documentation-review-summary.md
Normal file
165
docs/documentation-review-summary.md
Normal file
@@ -0,0 +1,165 @@
|
|||||||
|
# Documentation Review Summary - May 27, 2025
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Comprehensive review and cleanup of repository documentation structure to eliminate redundancy and improve cross-referencing.
|
||||||
|
|
||||||
|
## Issues Identified and Resolved
|
||||||
|
|
||||||
|
### 1. **Redundant Content Elimination**
|
||||||
|
|
||||||
|
#### Root README.md Cleanup
|
||||||
|
|
||||||
|
- **Removed**: Detailed Plex backup component descriptions (80+ lines) → Referenced `/plex/README.md`
|
||||||
|
- **Removed**: Extensive configuration parameter details (50+ lines) → Referenced component-specific docs
|
||||||
|
- **Removed**: Detailed testing instructions (40+ lines) → Referenced `/docs/docker-bootstrap-testing-framework.md`
|
||||||
|
- **Removed**: Complete crontab management section (70+ lines) → Referenced `/crontab/README.md`
|
||||||
|
|
||||||
|
#### Content Consolidation
|
||||||
|
|
||||||
|
- **Before**: 401 lines with significant duplication
|
||||||
|
- **After**: ~280 lines with proper cross-references
|
||||||
|
- **Reduction**: ~30% size reduction while maintaining all information access
|
||||||
|
|
||||||
|
### 2. **Missing Cross-References Added**
|
||||||
|
|
||||||
|
#### New Documentation Links
|
||||||
|
|
||||||
|
- ✅ **[Immich Scripts Documentation](./immich/README.md)** - Previously not referenced
|
||||||
|
- ✅ **[Telegram Bot Project](./telegram/github-issues/README.md)** - Major project missing from main docs
|
||||||
|
- ✅ **[Project Completion Summary](./docs/project-completion-summary.md)** - Achievement tracking doc
|
||||||
|
- ✅ **[Immich Backup Enhancement Summary](./docs/immich-backup-enhancement-summary.md)** - Technical improvements doc
|
||||||
|
|
||||||
|
#### Improved Navigation
|
||||||
|
|
||||||
|
- Added **Quick Navigation** section to root README
|
||||||
|
- Reorganized documentation into logical categories:
|
||||||
|
- Component-Specific Documentation
|
||||||
|
- Technical Documentation
|
||||||
|
|
||||||
|
### 3. **Structure Improvements**
|
||||||
|
|
||||||
|
#### Documentation Organization
|
||||||
|
|
||||||
|
```
|
||||||
|
Root README.md
|
||||||
|
├── Quick Navigation (NEW)
|
||||||
|
├── Available Scripts
|
||||||
|
│ ├── Backup Scripts
|
||||||
|
│ ├── Management Scripts
|
||||||
|
│ └── Development Projects (NEW)
|
||||||
|
├── Documentation (REORGANIZED)
|
||||||
|
│ ├── Component-Specific Documentation
|
||||||
|
│ └── Technical Documentation
|
||||||
|
├── Testing (STREAMLINED)
|
||||||
|
└── Dotfiles
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Cross-Reference Quality
|
||||||
|
|
||||||
|
- **Before**: Inconsistent linking, some docs orphaned
|
||||||
|
- **After**: Every major documentation file properly referenced
|
||||||
|
- **Validation**: All internal links functional and contextual
|
||||||
|
|
||||||
|
## Files Modified
|
||||||
|
|
||||||
|
### Primary Changes
|
||||||
|
|
||||||
|
1. **`/README.md`** - Major restructuring and content consolidation
|
||||||
|
2. **Created**: `/docs/documentation-review-summary.md` - This summary document
|
||||||
|
|
||||||
|
### Content Moved/Referenced
|
||||||
|
|
||||||
|
- Plex details → `/plex/README.md`
|
||||||
|
- Immich details → `/immich/README.md`
|
||||||
|
- Crontab management → `/crontab/README.md`
|
||||||
|
- Testing framework → `/docs/docker-bootstrap-testing-framework.md`
|
||||||
|
- Configuration details → Component-specific docs
|
||||||
|
|
||||||
|
## Benefits Achieved
|
||||||
|
|
||||||
|
### For Users
|
||||||
|
|
||||||
|
- **Faster navigation** with clear entry points
|
||||||
|
- **Reduced redundancy** - information appears once in authoritative location
|
||||||
|
- **Better discoverability** - all major features and docs properly linked
|
||||||
|
|
||||||
|
### For Maintainers
|
||||||
|
|
||||||
|
- **Single source of truth** for each topic
|
||||||
|
- **Easier updates** - change information in one place only
|
||||||
|
- **Consistent structure** across all documentation files
|
||||||
|
|
||||||
|
### For Contributors
|
||||||
|
|
||||||
|
- **Clear documentation hierarchy** for understanding project structure
|
||||||
|
- **Logical organization** makes finding relevant info intuitive
|
||||||
|
- **Complete cross-referencing** ensures no orphaned documentation
|
||||||
|
|
||||||
|
## Validation Results
|
||||||
|
|
||||||
|
### Link Verification
|
||||||
|
|
||||||
|
- ✅ All cross-references validated
|
||||||
|
- ✅ No broken internal links
|
||||||
|
- ✅ Proper relative path usage
|
||||||
|
|
||||||
|
### Content Coverage
|
||||||
|
|
||||||
|
- ✅ All major scripts documented
|
||||||
|
- ✅ All subdirectories with READMEs referenced
|
||||||
|
- ✅ All docs/ files properly linked
|
||||||
|
|
||||||
|
### Structure Consistency
|
||||||
|
|
||||||
|
- ✅ Consistent formatting across all documentation
|
||||||
|
- ✅ Logical information hierarchy maintained
|
||||||
|
- ✅ Clear separation between overview and detailed docs
|
||||||
|
|
||||||
|
## Post-Review Documentation Map
|
||||||
|
|
||||||
|
### Entry Points
|
||||||
|
|
||||||
|
1. **`/README.md`** - Main overview with navigation
|
||||||
|
2. **`/plex/README.md`** - Plex backup system details
|
||||||
|
3. **`/immich/README.md`** - Immich backup system details
|
||||||
|
4. **`/crontab/README.md`** - Crontab management system
|
||||||
|
5. **`/dotfiles/README.md`** - System configuration files
|
||||||
|
|
||||||
|
### Technical References
|
||||||
|
|
||||||
|
1. **`/docs/docker-bootstrap-testing-framework.md`** - Docker-based bootstrap validation system
|
||||||
|
2. **`/docs/enhanced-media-backup.md`** - Media backup technical guide
|
||||||
|
3. **`/docs/production-deployment-guide.md`** - Deployment procedures
|
||||||
|
4. **`/docs/project-completion-summary.md`** - Project achievements
|
||||||
|
|
||||||
|
### Development Projects
|
||||||
|
|
||||||
|
1. **`/telegram/github-issues/README.md`** - Telegram bot development plan
|
||||||
|
|
||||||
|
## Recommendations for Future Maintenance
|
||||||
|
|
||||||
|
### Documentation Updates
|
||||||
|
|
||||||
|
1. **When adding new scripts**: Update appropriate section in root README with brief description and link to detailed docs
|
||||||
|
2. **When creating new docs**: Add reference in root README documentation section
|
||||||
|
3. **When modifying major features**: Update both specific docs and root README references
|
||||||
|
|
||||||
|
### Structure Principles
|
||||||
|
|
||||||
|
1. **Root README**: Overview only, deep details in component docs
|
||||||
|
2. **Component READMEs**: Complete guides for their specific domain
|
||||||
|
3. **Docs folder**: Technical specifications, guides, and summaries
|
||||||
|
4. **Cross-referencing**: Always link to authoritative source rather than duplicating
|
||||||
|
|
||||||
|
### Quality Assurance
|
||||||
|
|
||||||
|
1. **Regular link validation**: Ensure all cross-references remain functional
|
||||||
|
2. **Content audit**: Quarterly review to catch new redundancies
|
||||||
|
3. **User feedback**: Monitor for navigation or information access issues
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Successfully eliminated ~120 lines of redundant content while improving discoverability and maintaining complete information access. The documentation now follows a clear hierarchy with proper cross-referencing, making it significantly easier to navigate and maintain.
|
||||||
|
|
||||||
|
**Key Achievement**: Transformed a monolithic documentation approach into a modular, well-organized system that scales better and reduces maintenance overhead.
|
||||||
132
docs/immich-backup-enhancement-summary.md
Normal file
132
docs/immich-backup-enhancement-summary.md
Normal file
@@ -0,0 +1,132 @@
|
|||||||
|
# Immich Backup Enhancement Summary
|
||||||
|
|
||||||
|
## Date: May 26, 2025
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Enhanced the Immich backup script with notification system and Backblaze B2 cloud storage integration, following the same pattern as the existing Plex backup script.
|
||||||
|
|
||||||
|
## New Features Added
|
||||||
|
|
||||||
|
### 🔔 Webhook Notifications
|
||||||
|
|
||||||
|
- **Start notification**: Sent when backup begins
|
||||||
|
- **Success notification**: Sent when backup completes with file details
|
||||||
|
- **Warning notification**: Sent when local backup succeeds but B2 upload fails
|
||||||
|
- **Error notification**: Sent when backup process fails
|
||||||
|
- **Rich content**: Includes file names, sizes, and emojis for better readability
|
||||||
|
|
||||||
|
### ☁️ Backblaze B2 Integration
|
||||||
|
|
||||||
|
- **B2 CLI tool**: Downloaded and installed `b2-linux` v4.3.2 in immich directory
|
||||||
|
- **Automatic uploads**: Both database and upload archives are uploaded to B2
|
||||||
|
- **Organized storage**: Files stored in `immich-backups/` folder in B2 bucket
|
||||||
|
- **Error resilience**: Script continues if B2 upload fails (local backup preserved)
|
||||||
|
- **Configuration**: Optional B2 settings in `.env` file
|
||||||
|
|
||||||
|
### 📊 Enhanced Reporting
|
||||||
|
|
||||||
|
- **File size reporting**: Backup sizes included in notifications
|
||||||
|
- **Upload status**: B2 upload success/failure status in notifications
|
||||||
|
- **Detailed logging**: All activities logged to centralized logs directory
|
||||||
|
|
||||||
|
## Technical Implementation
|
||||||
|
|
||||||
|
### Notification Function
|
||||||
|
|
||||||
|
```bash
|
||||||
|
send_notification() {
|
||||||
|
local title="$1"
|
||||||
|
local message="$2"
|
||||||
|
local status="${3:-info}"
|
||||||
|
# Sends to webhook with tags: backup,immich,hostname
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### B2 Upload Function
|
||||||
|
|
||||||
|
```bash
|
||||||
|
upload_to_b2() {
|
||||||
|
local file_path="$1"
|
||||||
|
# Authorizes and uploads to B2 bucket
|
||||||
|
# Returns success/failure status
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration Updates
|
||||||
|
|
||||||
|
Added to `.env` file:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Notification settings
|
||||||
|
WEBHOOK_URL="https://notify.peterwood.rocks/lab"
|
||||||
|
|
||||||
|
# Backblaze B2 settings (optional)
|
||||||
|
B2_APPLICATION_KEY_ID=your_key_id_here
|
||||||
|
B2_APPLICATION_KEY=your_application_key_here
|
||||||
|
B2_BUCKET_NAME=your_bucket_name_here
|
||||||
|
```
|
||||||
|
|
||||||
|
## Files Modified
|
||||||
|
|
||||||
|
1. **`/home/acedanger/shell/immich/backup-immich.sh`**
|
||||||
|
- Added notification functions
|
||||||
|
- Added B2 upload functionality
|
||||||
|
- Enhanced error handling with notifications
|
||||||
|
- Added file size reporting
|
||||||
|
|
||||||
|
2. **`/home/acedanger/shell/.env`**
|
||||||
|
- Added webhook URL configuration
|
||||||
|
- Added B2 configuration template
|
||||||
|
|
||||||
|
3. **`/home/acedanger/shell/immich/README.md`**
|
||||||
|
- Documented new notification features
|
||||||
|
- Added B2 setup instructions
|
||||||
|
- Enhanced feature documentation
|
||||||
|
|
||||||
|
## Files Added
|
||||||
|
|
||||||
|
1. **`/home/acedanger/shell/immich/b2-linux`**
|
||||||
|
- Backblaze B2 CLI tool v4.3.2
|
||||||
|
- Executable file for B2 operations
|
||||||
|
|
||||||
|
## Testing Results
|
||||||
|
|
||||||
|
✅ **Successful test run showed:**
|
||||||
|
|
||||||
|
- Environment variables loaded correctly
|
||||||
|
- Webhook notifications sent successfully
|
||||||
|
- Database backup created and compressed
|
||||||
|
- Container pause/unpause functionality working
|
||||||
|
- Error handling and cleanup working properly
|
||||||
|
- Notifications include proper emojis and formatting
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
### User Action Required
|
||||||
|
|
||||||
|
1. **Configure B2 (Optional)**:
|
||||||
|
- Create Backblaze B2 account and bucket
|
||||||
|
- Generate application keys
|
||||||
|
- Add credentials to `.env` file
|
||||||
|
|
||||||
|
2. **Test Full Backup**:
|
||||||
|
- Run complete backup: `./immich/backup-immich.sh`
|
||||||
|
- Verify notifications received
|
||||||
|
- Check B2 uploads (if configured)
|
||||||
|
|
||||||
|
3. **Setup Automation**:
|
||||||
|
- Add to crontab for scheduled backups
|
||||||
|
- Monitor backup logs in `/home/acedanger/shell/logs/`
|
||||||
|
|
||||||
|
## Benefits
|
||||||
|
|
||||||
|
- **Visibility**: Real-time notifications of backup status
|
||||||
|
- **Reliability**: Off-site backup storage with B2
|
||||||
|
- **Consistency**: Same notification pattern as Plex backups
|
||||||
|
- **Monitoring**: Enhanced logging and error reporting
|
||||||
|
- **Scalability**: Easy to extend with additional storage providers
|
||||||
|
|
||||||
|
## Pattern Consistency
|
||||||
|
|
||||||
|
The implementation follows the same notification and logging patterns established in the Plex backup script, ensuring consistency across the backup system.
|
||||||
133
docs/immich-backup-migration-summary.md
Normal file
133
docs/immich-backup-migration-summary.md
Normal file
@@ -0,0 +1,133 @@
|
|||||||
|
# Immich Backup System Migration Summary
|
||||||
|
|
||||||
|
## Changes Made
|
||||||
|
|
||||||
|
### 1. Directory Structure Reorganization
|
||||||
|
|
||||||
|
- **Created**: `/home/acedanger/shell/immich/` directory for all Immich-related scripts
|
||||||
|
- **Moved**: `backup-immich-db.sh` → `immich/backup-immich.sh`
|
||||||
|
- **Removed**: `immich/logs/` directory (consolidated to main logs)
|
||||||
|
- **Deleted**: Original `backup-immich-db.sh` from root directory
|
||||||
|
|
||||||
|
### 2. Centralized Logging Implementation
|
||||||
|
|
||||||
|
- **Log Directory**: All logs now go to `/home/acedanger/shell/logs/`
|
||||||
|
- **Log Files**:
|
||||||
|
- `immich-backup.log` - Main backup operations
|
||||||
|
- `immich-validation.log` - Backup validation results
|
||||||
|
- `immich-restore.log` - Restore operations (when implemented)
|
||||||
|
|
||||||
|
### 3. Script Updates
|
||||||
|
|
||||||
|
#### backup-immich.sh
|
||||||
|
|
||||||
|
- Updated paths to work from `immich/` subdirectory
|
||||||
|
- Added centralized logging with timestamps
|
||||||
|
- Enhanced with `log_message()` and `log_status()` functions
|
||||||
|
- All major operations now logged with timestamps
|
||||||
|
- Improved cleanup function with logging
|
||||||
|
|
||||||
|
#### validate-immich-backups.sh
|
||||||
|
|
||||||
|
- Added centralized logging capability
|
||||||
|
- Validation results logged to `immich-validation.log`
|
||||||
|
- Enhanced error reporting and logging
|
||||||
|
|
||||||
|
#### restore-immich.sh
|
||||||
|
|
||||||
|
- Added logging framework (template ready for implementation)
|
||||||
|
- Configured to use centralized logs directory
|
||||||
|
|
||||||
|
### 4. Configuration Updates
|
||||||
|
|
||||||
|
#### README.md
|
||||||
|
|
||||||
|
- Updated logging references to point to central logs directory
|
||||||
|
- Updated crontab example to use correct log path:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
0 2 * * * /home/acedanger/shell/immich/backup-immich.sh >> /home/acedanger/shell/logs/immich-backup.log 2>&1
|
||||||
|
```
|
||||||
|
|
||||||
|
- Enhanced features list to highlight centralized logging
|
||||||
|
|
||||||
|
#### .env.example
|
||||||
|
|
||||||
|
- Created example configuration file for easy setup
|
||||||
|
- Contains sample values for required environment variables
|
||||||
|
|
||||||
|
### 5. Directory Structure (Final)
|
||||||
|
|
||||||
|
```
|
||||||
|
/home/acedanger/shell/
|
||||||
|
├── immich/ # Immich management scripts
|
||||||
|
│ ├── backup-immich.sh # Complete backup script
|
||||||
|
│ ├── restore-immich.sh # Restore script (template)
|
||||||
|
│ ├── validate-immich-backups.sh # Backup validation
|
||||||
|
│ └── README.md # Documentation
|
||||||
|
├── logs/ # Centralized logging
|
||||||
|
│ ├── immich-backup.log # Backup operations (when created)
|
||||||
|
│ ├── immich-validation.log # Validation results (when created)
|
||||||
|
│ └── immich-restore.log # Restore operations (when created)
|
||||||
|
├── immich_backups/ # Backup storage (when created)
|
||||||
|
└── .env.example # Configuration template
|
||||||
|
```
|
||||||
|
|
||||||
|
## Benefits of Changes
|
||||||
|
|
||||||
|
### 1. **Consistency**
|
||||||
|
|
||||||
|
- Follows same organizational pattern as existing `plex/` folder
|
||||||
|
- All Immich-related scripts in one location
|
||||||
|
- Centralized logging matches shell repository standards
|
||||||
|
|
||||||
|
### 2. **Maintainability**
|
||||||
|
|
||||||
|
- Clear separation of concerns
|
||||||
|
- Documented scripts with usage examples
|
||||||
|
- Consistent logging format across all operations
|
||||||
|
|
||||||
|
### 3. **Monitoring & Debugging**
|
||||||
|
|
||||||
|
- All logs in one central location
|
||||||
|
- Timestamped log entries for better troubleshooting
|
||||||
|
- Comprehensive logging of all backup phases
|
||||||
|
|
||||||
|
### 4. **Automation Ready**
|
||||||
|
|
||||||
|
- Updated crontab examples for automated backups
|
||||||
|
- Proper logging for unattended operations
|
||||||
|
- Error tracking and reporting
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
### Manual Backup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /home/acedanger/shell/immich
|
||||||
|
./backup-immich.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### Automated Backup (Crontab)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Daily backup at 2:00 AM
|
||||||
|
0 2 * * * /home/acedanger/shell/immich/backup-immich.sh >> /home/acedanger/shell/logs/immich-backup.log 2>&1
|
||||||
|
```
|
||||||
|
|
||||||
|
### Validation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /home/acedanger/shell/immich
|
||||||
|
./validate-immich-backups.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
1. **Configure .env file** with your Immich settings
|
||||||
|
2. **Test backup script** in your environment
|
||||||
|
3. **Set up automated backups** via crontab
|
||||||
|
4. **Implement restore script** (template provided)
|
||||||
|
5. **Monitor logs** for any issues
|
||||||
|
|
||||||
|
All logging will be automatically created in `/home/acedanger/shell/logs/` when the scripts are first run.
|
||||||
@@ -127,7 +127,7 @@ The tests validate:
|
|||||||
./run-docker-tests.sh full-ubuntu
|
./run-docker-tests.sh full-ubuntu
|
||||||
```
|
```
|
||||||
|
|
||||||
For more details on testing, see [Testing Documentation](../docs/testing.md).
|
For more details on testing, see [Docker Bootstrap Testing Framework](../docs/docker-bootstrap-testing-framework.md).
|
||||||
|
|
||||||
## Manual Steps
|
## Manual Steps
|
||||||
|
|
||||||
|
|||||||
165
immich/README.md
Normal file
165
immich/README.md
Normal file
@@ -0,0 +1,165 @@
|
|||||||
|
# Immich Management Scripts
|
||||||
|
|
||||||
|
This directory contains scripts for managing and backing up Immich photo management system.
|
||||||
|
|
||||||
|
## Scripts
|
||||||
|
|
||||||
|
### backup-immich.sh
|
||||||
|
|
||||||
|
Complete backup script for Immich installation that creates backups of:
|
||||||
|
|
||||||
|
- PostgreSQL database (using pg_dumpall as recommended by Immich)
|
||||||
|
- User upload directories (photos, videos, and metadata)
|
||||||
|
|
||||||
|
**Requirements:**
|
||||||
|
|
||||||
|
- `.env` file in the parent directory (`/home/acedanger/shell/.env`) with:
|
||||||
|
- `DB_USERNAME` - PostgreSQL username
|
||||||
|
- `DB_DATABASE_NAME` - Database name
|
||||||
|
- `UPLOAD_LOCATION` - Path to Immich upload directory
|
||||||
|
- Docker containers: `immich_postgres` and `immich_server`
|
||||||
|
|
||||||
|
**Usage:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./backup-immich.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
**Backup Location:**
|
||||||
|
|
||||||
|
- Database: `../immich_backups/immich_db_backup_YYYYMMDD_HHMMSS.sql.gz`
|
||||||
|
- Uploads: `../immich_backups/immich_uploads_YYYYMMDD_HHMMSS.tar.gz`
|
||||||
|
|
||||||
|
**Features:**
|
||||||
|
|
||||||
|
- Automatic container pausing/resuming during backup
|
||||||
|
- Comprehensive error handling and cleanup
|
||||||
|
- Backup validation and health checks
|
||||||
|
- Automatic compression
|
||||||
|
- Old backup cleanup (configurable)
|
||||||
|
- Centralized logging to `/home/acedanger/shell/logs/`
|
||||||
|
- Detailed progress reporting and timestamped logs
|
||||||
|
- 🔔 **Webhook notifications** to notify.peterwood.rocks/lab
|
||||||
|
- ☁️ **Backblaze B2 integration** for off-site backup storage
|
||||||
|
- 📊 **File size reporting** in notifications
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
The scripts expect a `.env` file in the parent directory with the following variables:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Database configuration
|
||||||
|
DB_USERNAME=postgres
|
||||||
|
DB_DATABASE_NAME=immich
|
||||||
|
UPLOAD_LOCATION=/path/to/immich/uploads
|
||||||
|
|
||||||
|
# Notification settings
|
||||||
|
WEBHOOK_URL="https://notify.peterwood.rocks/lab"
|
||||||
|
|
||||||
|
# Backblaze B2 settings (optional)
|
||||||
|
# Get these from your B2 account: https://secure.backblaze.com/app_keys.htm
|
||||||
|
# B2_APPLICATION_KEY_ID=your_key_id_here
|
||||||
|
# B2_APPLICATION_KEY=your_application_key_here
|
||||||
|
# B2_BUCKET_NAME=your_bucket_name_here
|
||||||
|
|
||||||
|
# Optional: Backup retention (days)
|
||||||
|
BACKUP_RETENTION_DAYS=30
|
||||||
|
```
|
||||||
|
|
||||||
|
## Backup Strategy
|
||||||
|
|
||||||
|
Based on Immich's official backup recommendations:
|
||||||
|
|
||||||
|
1. **Database Backup**: Uses `pg_dumpall` with `--clean` and `--if-exists` flags
|
||||||
|
2. **Upload Directory**: Complete archive of upload location including:
|
||||||
|
- upload/ - Original photos and videos
|
||||||
|
- profile/ - User profile images
|
||||||
|
- thumbs/ - Generated thumbnails
|
||||||
|
- encoded-video/ - Transcoded videos
|
||||||
|
- library/ - Library metadata
|
||||||
|
- backups/ - Existing backup files (excluded from new backups)
|
||||||
|
|
||||||
|
## Notifications 🔔
|
||||||
|
|
||||||
|
The backup script sends notifications to your webhook URL with:
|
||||||
|
|
||||||
|
- 🚀 **Start notification**: When backup begins
|
||||||
|
- ✅ **Success notification**: When backup completes successfully with file sizes
|
||||||
|
- ⚠️ **Warning notification**: When backup succeeds but B2 upload fails
|
||||||
|
- 🚨 **Error notification**: When backup fails
|
||||||
|
|
||||||
|
Example notification:
|
||||||
|
|
||||||
|
```
|
||||||
|
📦 Database: immich_db_backup_20250526_215913.sql.gz (150MB)
|
||||||
|
📁 Uploads: immich_uploads_20250526_215913.tar.gz (25GB)
|
||||||
|
☁️ Successfully uploaded to B2 bucket: my-immich-backups
|
||||||
|
```
|
||||||
|
|
||||||
|
## Backblaze B2 Integration ☁️
|
||||||
|
|
||||||
|
### Setup B2 Account
|
||||||
|
|
||||||
|
1. Create a [Backblaze B2 account](https://www.backblaze.com/b2/cloud-storage.html)
|
||||||
|
2. Create a new bucket for Immich backups
|
||||||
|
3. Generate application keys:
|
||||||
|
- Go to: <https://secure.backblaze.com/app_keys.htm>
|
||||||
|
- Create new key with read/write access to your bucket
|
||||||
|
|
||||||
|
### Configure B2 in .env
|
||||||
|
|
||||||
|
Add these variables to your `.env` file:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
B2_APPLICATION_KEY_ID=your_key_id_here
|
||||||
|
B2_APPLICATION_KEY=your_application_key_here
|
||||||
|
B2_BUCKET_NAME=your_bucket_name_here
|
||||||
|
```
|
||||||
|
|
||||||
|
### B2 Features
|
||||||
|
|
||||||
|
- **Automatic upload**: Backup files are uploaded to B2 after creation
|
||||||
|
- **Organized storage**: Files stored in `immich-backups/` folder in your bucket
|
||||||
|
- **Error handling**: Script continues if B2 upload fails (local backup preserved)
|
||||||
|
- **Progress tracking**: Upload status included in notifications
|
||||||
|
|
||||||
|
The B2 CLI tool (`b2-linux`) is included in this directory and doesn't require separate installation.
|
||||||
|
|
||||||
|
## Restore Process
|
||||||
|
|
||||||
|
For complete restore instructions, see: <https://immich.app/docs/administration/backup-and-restore/>
|
||||||
|
|
||||||
|
1. **Database Restore:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker exec -i immich_postgres psql -U postgres < immich_db_backup.sql
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Upload Directory Restore:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tar -xzf immich_uploads_backup.tar.gz -C /target/location
|
||||||
|
```
|
||||||
|
|
||||||
|
## Logs
|
||||||
|
|
||||||
|
Backup logs and performance metrics are stored in the main shell logs directory (`/home/acedanger/shell/logs/`).
|
||||||
|
|
||||||
|
## Monitoring
|
||||||
|
|
||||||
|
The backup script includes:
|
||||||
|
|
||||||
|
- Progress indicators for long-running operations
|
||||||
|
- Size validation for backup files
|
||||||
|
- Container status monitoring
|
||||||
|
- Automatic cleanup procedures
|
||||||
|
- Comprehensive error reporting
|
||||||
|
|
||||||
|
## Automation
|
||||||
|
|
||||||
|
To automate backups, add to crontab:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Daily Immich backup at 2:00 AM
|
||||||
|
0 2 * * * /home/acedanger/shell/immich/backup-immich.sh >> /home/acedanger/shell/logs/immich-backup.log 2>&1
|
||||||
|
```
|
||||||
224
telegram/github-issues/README.md
Normal file
224
telegram/github-issues/README.md
Normal file
@@ -0,0 +1,224 @@
|
|||||||
|
# Telegram Backup Monitoring Bot - GitHub Issues Summary
|
||||||
|
|
||||||
|
This document provides an overview of the GitHub issues created for developing a comprehensive Telegram bot that monitors and manages the existing backup systems (Plex, Immich, Media services).
|
||||||
|
|
||||||
|
## Project Overview
|
||||||
|
|
||||||
|
The Telegram bot will integrate with three sophisticated backup systems:
|
||||||
|
|
||||||
|
1. **Plex Backup System** - Enhanced backup with validation, monitoring, restoration, and performance tracking
|
||||||
|
2. **Immich Backup System** - Database and upload directory backup with B2 cloud storage integration
|
||||||
|
3. **Media Services Backup** - Docker container backup for 7 media services with parallel execution
|
||||||
|
|
||||||
|
## Issues Created
|
||||||
|
|
||||||
|
### Core Development Issues
|
||||||
|
|
||||||
|
| Issue | Title | Complexity | Estimated Time | Dependencies |
|
||||||
|
|-------|-------|------------|----------------|--------------|
|
||||||
|
| #01 | [Telegram Bot Core Infrastructure](01-telegram-bot-core-infrastructure.md) | Medium | 2-3 days | None |
|
||||||
|
| #02 | [Plex Backup Monitoring Integration](02-plex-backup-monitoring.md) | Medium-High | 3-4 days | #01 |
|
||||||
|
| #03 | [Immich Backup Monitoring Integration](03-immich-backup-monitoring.md) | Medium | 2-3 days | #01 |
|
||||||
|
| #04 | [Media Services Backup Monitoring Integration](04-media-services-monitoring.md) | Medium | 2-3 days | #01 |
|
||||||
|
| #05 | [Advanced Monitoring Dashboard](05-advanced-monitoring-dashboard.md) | High | 4-5 days | #02, #03, #04 |
|
||||||
|
| #06 | [Notification System Integration](06-notification-system-integration.md) | Medium-High | 3-4 days | #01-05 |
|
||||||
|
| #07 | [Backup Control and Management](07-backup-control-management.md) | High | 4-5 days | #01-06 |
|
||||||
|
| #08 | [Performance Optimization and Enhancement](08-performance-optimization.md) | High | 5-6 days | #01-07 |
|
||||||
|
| #09 | [Testing, Documentation, and Deployment](09-testing-documentation-deployment.md) | High | 6-7 days | #01-08 |
|
||||||
|
|
||||||
|
**Total Estimated Development Time: 31-40 days**
|
||||||
|
|
||||||
|
## Development Phases
|
||||||
|
|
||||||
|
### Phase 1: Foundation (Issues #01-04)
|
||||||
|
**Duration**: 9-13 days
|
||||||
|
**Goal**: Establish core bot infrastructure and basic monitoring for all three backup systems
|
||||||
|
|
||||||
|
- Core bot framework with authentication
|
||||||
|
- Basic monitoring commands for each backup system
|
||||||
|
- JSON log parsing and status reporting
|
||||||
|
- Security framework implementation
|
||||||
|
|
||||||
|
### Phase 2: Advanced Features (Issues #05-07)
|
||||||
|
**Duration**: 11-14 days
|
||||||
|
**Goal**: Implement advanced monitoring, notifications, and control capabilities
|
||||||
|
|
||||||
|
- Unified monitoring dashboard with trend analysis
|
||||||
|
- Real-time notification system with webhook integration
|
||||||
|
- Administrative control interface for backup management
|
||||||
|
- Cross-system health correlation and alerting
|
||||||
|
|
||||||
|
### Phase 3: Production Ready (Issues #08-09)
|
||||||
|
**Duration**: 11-13 days
|
||||||
|
**Goal**: Optimize performance and prepare for production deployment
|
||||||
|
|
||||||
|
- Performance optimization with caching and streaming
|
||||||
|
- Interactive UI enhancements and smart features
|
||||||
|
- Comprehensive testing suite and documentation
|
||||||
|
- Production deployment infrastructure
|
||||||
|
|
||||||
|
## Key Features by Issue
|
||||||
|
|
||||||
|
### Issue #01: Core Infrastructure
|
||||||
|
- Bot registration and token management
|
||||||
|
- Command parsing and routing system
|
||||||
|
- User authentication and authorization
|
||||||
|
- Basic logging and error handling
|
||||||
|
- Configuration management
|
||||||
|
|
||||||
|
### Issue #02: Plex Integration
|
||||||
|
- Status monitoring with JSON log parsing
|
||||||
|
- Performance metrics and trend analysis
|
||||||
|
- Health checks and validation integration
|
||||||
|
- Admin controls for backup management
|
||||||
|
- Real-time monitoring dashboard integration
|
||||||
|
|
||||||
|
### Issue #03: Immich Integration
|
||||||
|
- Database backup status monitoring
|
||||||
|
- Upload directory synchronization tracking
|
||||||
|
- B2 cloud storage integration and status
|
||||||
|
- Backup validation and health reporting
|
||||||
|
- Multi-component status aggregation
|
||||||
|
|
||||||
|
### Issue #04: Media Services Integration
|
||||||
|
- Multi-service status monitoring (7 services)
|
||||||
|
- Parallel vs sequential backup mode detection
|
||||||
|
- Per-service detailed status and metrics
|
||||||
|
- Container health and backup freshness checking
|
||||||
|
- Service-specific command handling
|
||||||
|
|
||||||
|
### Issue #05: Advanced Dashboard
|
||||||
|
- Cross-system health correlation
|
||||||
|
- Performance trend analysis and forecasting
|
||||||
|
- Predictive failure detection
|
||||||
|
- Executive summary reporting
|
||||||
|
- Storage usage forecasting
|
||||||
|
|
||||||
|
### Issue #06: Notification System
|
||||||
|
- Real-time alert delivery
|
||||||
|
- Scheduled status reports (daily/weekly)
|
||||||
|
- Emergency notification prioritization
|
||||||
|
- Webhook integration with existing systems
|
||||||
|
- Customizable notification preferences
|
||||||
|
|
||||||
|
### Issue #07: Backup Control
|
||||||
|
- Manual backup triggering for all systems
|
||||||
|
- Backup schedule management
|
||||||
|
- Queue management and conflict prevention
|
||||||
|
- Maintenance mode controls
|
||||||
|
- Emergency stop capabilities
|
||||||
|
|
||||||
|
### Issue #08: Performance & UX
|
||||||
|
- Response caching and optimization
|
||||||
|
- Interactive keyboards and progress indicators
|
||||||
|
- Performance monitoring and analytics
|
||||||
|
- Smart command suggestions
|
||||||
|
- Visual chart generation
|
||||||
|
|
||||||
|
### Issue #09: Production Deployment
|
||||||
|
- Comprehensive testing suite (unit, integration, e2e)
|
||||||
|
- Complete documentation package
|
||||||
|
- Docker containerization and deployment
|
||||||
|
- CI/CD pipeline setup
|
||||||
|
- Production monitoring and health checks
|
||||||
|
|
||||||
|
## Integration with Existing Systems
|
||||||
|
|
||||||
|
### Backup Scripts Integration
|
||||||
|
```bash
|
||||||
|
# Scripts the bot will interface with:
|
||||||
|
/home/acedanger/shell/plex/backup-plex.sh # Plex backup & validation
|
||||||
|
/home/acedanger/shell/plex/monitor-plex-backup.sh # Real-time monitoring
|
||||||
|
/home/acedanger/shell/plex/validate-plex-backups.sh # Health checks
|
||||||
|
/home/acedanger/shell/immich/backup-immich.sh # Immich backup
|
||||||
|
/home/acedanger/shell/backup-media.sh # Media services
|
||||||
|
/home/acedanger/shell/backup-log-monitor.sh # System monitoring
|
||||||
|
```
|
||||||
|
|
||||||
|
### Log Files Integration
|
||||||
|
```bash
|
||||||
|
# JSON performance logs
|
||||||
|
/home/acedanger/shell/logs/plex-backup-performance.json
|
||||||
|
/home/acedanger/shell/logs/media-backup.json
|
||||||
|
/home/acedanger/shell/logs/immich-backup.log
|
||||||
|
|
||||||
|
# Validation and monitoring logs
|
||||||
|
/home/acedanger/shell/plex/logs/backup-validation-*.log
|
||||||
|
/home/acedanger/shell/logs/immich-validation.log
|
||||||
|
```
|
||||||
|
|
||||||
|
### Webhook Integration
|
||||||
|
- Enhance existing webhook support in backup scripts
|
||||||
|
- Implement structured webhook payloads
|
||||||
|
- Real-time event delivery to Telegram bot
|
||||||
|
- Integration with notification preferences
|
||||||
|
|
||||||
|
## Security Considerations
|
||||||
|
|
||||||
|
### Multi-Level Authorization
|
||||||
|
- **User Level**: Basic status and monitoring commands
|
||||||
|
- **Admin Level**: Backup controls and management
|
||||||
|
- **Super Admin Level**: Emergency controls and system management
|
||||||
|
|
||||||
|
### Security Features
|
||||||
|
- Token-based user authentication
|
||||||
|
- Command input validation and sanitization
|
||||||
|
- Rate limiting and abuse prevention
|
||||||
|
- Secure webhook signature validation
|
||||||
|
- Audit logging for all administrative actions
|
||||||
|
|
||||||
|
## Technology Stack
|
||||||
|
|
||||||
|
### Core Technologies
|
||||||
|
- **Python 3.9+** - Main development language
|
||||||
|
- **python-telegram-bot** - Telegram API framework
|
||||||
|
- **asyncio** - Asynchronous operations
|
||||||
|
- **aiohttp** - HTTP client with connection pooling
|
||||||
|
- **redis** - Caching and session storage
|
||||||
|
|
||||||
|
### Supporting Tools
|
||||||
|
- **Docker** - Containerization and deployment
|
||||||
|
- **pytest** - Testing framework
|
||||||
|
- **matplotlib** - Chart generation
|
||||||
|
- **GitHub Actions** - CI/CD pipeline
|
||||||
|
|
||||||
|
## Success Metrics
|
||||||
|
|
||||||
|
### Functional Requirements
|
||||||
|
- ✅ Monitor all 3 backup systems reliably
|
||||||
|
- ✅ Provide real-time status and health information
|
||||||
|
- ✅ Enable administrative control of backup operations
|
||||||
|
- ✅ Deliver proactive notifications and alerts
|
||||||
|
- ✅ Generate performance analytics and trends
|
||||||
|
|
||||||
|
### Performance Requirements
|
||||||
|
- Response times < 2 seconds for cached data
|
||||||
|
- Support for multiple concurrent users
|
||||||
|
- 99.9% uptime and reliability
|
||||||
|
- Efficient memory usage with large log files
|
||||||
|
- Real-time notification delivery
|
||||||
|
|
||||||
|
### Security Requirements
|
||||||
|
- Multi-level user authorization
|
||||||
|
- Secure token and credential management
|
||||||
|
- Input validation and injection prevention
|
||||||
|
- Comprehensive audit logging
|
||||||
|
- Rate limiting and abuse protection
|
||||||
|
|
||||||
|
## Getting Started
|
||||||
|
|
||||||
|
1. **Create GitHub Issues**: Copy the individual markdown files to create GitHub issues
|
||||||
|
2. **Set up Development Environment**: Follow Issue #01 for initial setup
|
||||||
|
3. **Implement in Order**: Follow the dependency chain starting with Issue #01
|
||||||
|
4. **Test Thoroughly**: Each issue includes comprehensive testing requirements
|
||||||
|
5. **Deploy Incrementally**: Use Docker deployment from Issue #09
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- Each issue is designed to be independently assignable to GitHub Copilot
|
||||||
|
- Issues include detailed technical specifications and examples
|
||||||
|
- Testing requirements are built into each issue
|
||||||
|
- Security considerations are addressed throughout
|
||||||
|
- Production deployment is fully automated
|
||||||
|
|
||||||
|
This comprehensive issue set provides a complete roadmap for developing a production-ready Telegram bot that transforms backup system monitoring from a manual process into an intelligent, proactive assistant.
|
||||||
Reference in New Issue
Block a user