peterwood/shell
1
0
Fork 0
mirror of https://github.com/acedanger/shell.git synced 2025-12-06 00:00:13 -08:00
Code Issues 19 Packages Projects Releases Wiki Activity

feat: Add Docker deployment manager and stack assignment helper scripts

Browse Source 
- Introduced `docker-deployment-manager.sh` for managing Docker stack deployments across multiple servers, including initialization, deployment, and status checks.
- Added `stack-assignment-helper.sh` to analyze Docker stacks and suggest server assignments based on predefined patterns.
- Removed outdated `SETUP_COMPLETE.md` file as it is no longer relevant to the current setup process.
- Ref - Documentation review #11
This commit is contained in:
Peter Wood
2025-05-29 18:19:37 -04:00
parent a05f5c6d9d
commit 3ce2b687ac
7 changed files with 469 additions and 116 deletions
Download Patch File Download Diff File Expand all files Collapse all files

172
README.md
View File

@@ -6,8 +6,9 @@ This repository contains various shell scripts for managing media-related tasks
- **[Backup Scripts](#backup-scripts)** - Enterprise-grade backup solutions - **[Backup Scripts](#backup-scripts)** - Enterprise-grade backup solutions
- **[Management Scripts](#management-scripts)** - System and service management - **[Management Scripts](#management-scripts)** - System and service management
- **[AI Integration](#ai-integration)** - Ollama and Fabric setup for AI-assisted development
- **[Tab Completion](#tab-completion)** - Intelligent command-line completion - **[Tab Completion](#tab-completion)** - Intelligent command-line completion
- **[Documentation](#documentation)** - Complete guides and references - **[Documentation](#comprehensive-documentation)** - Complete guides and references
- **[Testing](#testing)** - Docker-based validation framework - **[Testing](#testing)** - Docker-based validation framework
- **[Dotfiles](#dotfiles)** - System configuration files - **[Dotfiles](#dotfiles)** - System configuration files
@@ -25,9 +26,70 @@ This repository contains various shell scripts for managing media-related tasks
- **`plex.sh`**: Script to manage the Plex Media Server (start, stop, restart, status). - **`plex.sh`**: Script to manage the Plex Media Server (start, stop, restart, status).
- **`folder-metrics.sh`**: Script to calculate disk usage and file count for a directory and its subdirectories. - **`folder-metrics.sh`**: Script to calculate disk usage and file count for a directory and its subdirectories.
## AI Integration
This repository includes a complete AI development environment with Ollama and Fabric integration for AI-assisted development tasks.
### Ollama + Fabric Setup
The system includes:
- **Ollama Docker container** running on port 11434 with phi3:mini model (3.8B parameters)
- **Fabric v1.4.195** with 216+ AI patterns for text processing
- **Google Gemini 2.5 Pro** as primary AI provider
- **Local Ollama models** as secondary AI provider
- **Custom shell aliases** for easy container management
### Basic Fabric Usage
```bash
# List all available patterns
fabric -l
# Use a pattern (uses Gemini 2.5 Pro by default)
echo "Your text here" | fabric -p summarize
# Use with specific model
echo "Your text here" | fabric -p summarize -m gemini-2.0-flash-exp
# Use local Ollama models
echo "Your text here" | fabric -p summarize -m ollama:phi3:mini
# Update patterns
fabric -U
```
### Ollama Management Aliases
```bash
ollama-start # Start Ollama container
ollama-stop # Stop Ollama container
ollama-restart # Restart Ollama container
ollama-logs # View container logs
ollama-status # Check container status
```
### Popular AI Patterns
- `summarize` - Summarize text content
- `explain_code` - Explain code snippets and logic
- `improve_writing` - Enhance writing quality and clarity
- `extract_wisdom` - Extract key insights from content
- `create_quiz` - Generate quiz questions from text
- `analyze_claims` - Analyze and fact-check claims
### Configuration Files
- **Fabric config**: `~/.config/fabric/.env` - AI provider settings and API keys
- **Ollama aliases**: `~/.oh-my-zsh/custom/ollama-aliases.zsh` - Container management commands
- **Shell config**: `~/.zshrc` - Main shell configuration
For complete setup instructions, see the setup documentation.
### Development Projects ### Development Projects
- **[Telegram Backup Monitoring Bot](./telegram/github-issues/README.md)**: Comprehensive Telegram bot project for monitoring and managing all backup systems with real-time notifications and control capabilities. - **[Telegram Backup Monitoring Bot](./telegram/github-issues/README.md)**: Comprehensive Telegram bot project for monitoring and managing all backup systems with real-time notifications and control capabilities.
- **[Docker Deployment Management](./docker-deployment/README.md)**: Centralized Docker stack deployment system for managing containers across multiple servers with intelligent assignment and environment integration.
### Testing Scripts ### Testing Scripts
@@ -172,6 +234,64 @@ For detailed configuration information and backup strategies, see:
- **[Enhanced Media Backup](./docs/enhanced-media-backup.md)**: Media backup script configuration - **[Enhanced Media Backup](./docs/enhanced-media-backup.md)**: Media backup script configuration
- **[Crontab Management](./crontab/README.md)**: Automated scheduling and system-specific configurations - **[Crontab Management](./crontab/README.md)**: Automated scheduling and system-specific configurations
## Comprehensive Documentation
This repository includes extensive documentation organized by component and functionality:
### 📁 Documentation Structure
```
📁 docs/ - Technical documentation and guides
📁 setup/ - Setup and configuration files
📁 plex/ - Plex Media Server scripts and docs
📁 immich/ - Immich photo management scripts
📁 crontab/ - Crontab management system
📁 dotfiles/ - System configuration files
📁 completions/ - Tab completion scripts
📁 telegram/ - Telegram bot integration
```
### 🛠️ Component Documentation
#### Backup Systems
- **[Enhanced Media Backup](./docs/enhanced-media-backup.md)**: Advanced media backup features and configuration
- **[Basic Media Backup](./docs/backup-media.md)**: Original backup system documentation
- **[Environment Backup System](./docs/env-backup-system.md)**: Environment and configuration backup
- **[Backup Environment Safety Guide](./docs/backup-env-safety-guide.md)**: Safety guidelines for environment backups
#### Media Server Management
- **[Plex Scripts Documentation](./plex/README.md)**: Comprehensive Plex backup, restoration, and management
- **[Plex Backup Guide](./plex/plex-backup.md)**: Detailed Plex backup system documentation
- **[Plex Management Guide](./plex/plex-management.md)**: Server management and operations
#### Photo Management
- **[Immich Scripts Documentation](./immich/README.md)**: Complete Immich backup and management system
- **[Immich Enhancement Summary](./docs/immich-backup-enhancement-summary.md)**: Recent improvements and features
- **[Immich Migration Guide](./docs/immich-backup-migration-summary.md)**: Migration procedures and best practices
#### System Administration
- **[Crontab Management](./crontab/README.md)**: Multi-system crontab coordination and backup
- **[Enhanced Crontab System](./crontab/enhanced-crontab-system.md)**: Advanced scheduling features
- **[Dotfiles Documentation](./dotfiles/README.md)**: System configuration and setup
- **[Tab Completion Guide](./completions/README.md)**: Intelligent command completion system
#### Development & Deployment
- **[Docker Deployment Management](./docker-deployment/README.md)**: Centralized multi-server Docker stack deployment system
- **[Docker Bootstrap Testing](./docs/docker-bootstrap-testing-framework.md)**: Comprehensive testing framework
- **[Production Deployment Guide](./docs/production-deployment-guide.md)**: Complete deployment procedures
- **[Docker Stack Strategy](./docs/docker-stack-deployment-strategy.md)**: Container deployment strategy
### 📊 Project Documentation
- **[Project Completion Summary](./docs/project-completion-summary.md)**: Overall project status and milestones
- **[Documentation Review](./docs/documentation-review-summary.md)**: Comprehensive documentation overview
- **[Technical Implementation Summaries](./docs/)**: Various technical guides and implementation details
## Documentation ## Documentation
### Component-Specific Documentation ### Component-Specific Documentation
@@ -192,6 +312,56 @@ For detailed configuration information and backup strategies, see:
- **[Production Deployment Guide](./docs/production-deployment-guide.md)**: Complete deployment and operations guide - **[Production Deployment Guide](./docs/production-deployment-guide.md)**: Complete deployment and operations guide
- **[Project Completion Summary](./docs/project-completion-summary.md)**: Overview of completed features and project milestones - **[Project Completion Summary](./docs/project-completion-summary.md)**: Overview of completed features and project milestones
### Environment & System Documentation
- **[Environment Backup System](./docs/env-backup-system.md)**: Complete environment backup documentation
- **[Environment Backup Integration Guide](./docs/env-backup-integration-guide.md)**: Integration procedures and best practices
- **[Backup Environment Safety Guide](./docs/backup-env-safety-guide.md)**: Safety guidelines and procedures
- **[Tab Completion Implementation](./docs/tab-completion-implementation-summary.md)**: Command completion system details
- **[Docker Stack Deployment Strategy](./docs/docker-stack-deployment-strategy.md)**: Container deployment and management
### Project Reviews & Summaries
- **[Documentation Review Summary](./docs/documentation-review-summary.md)**: Comprehensive documentation audit and review
- **[Issue Resolution Summaries](./docs/)**: Technical implementation details and problem resolution guides
## Setup and Getting Started
### Quick Setup
The repository includes automated setup via bootstrap script:
```bash
curl -fsSL https://raw.githubusercontent.com/acedanger/shell/main/bootstrap.sh | bash
```
This installs:
- Zsh with Oh My Zsh and custom plugins
- All packages from `setup/packages.list`
- Dotfiles and system configuration
- Tab completion for all scripts
- Development tools (Node.js via nvm, VS Code, etc.)
### AI Development Environment
For AI-assisted development, the system includes:
- **Ollama** running in Docker with local models
- **Fabric** with 216+ AI patterns for text processing
- **Google Gemini integration** as primary AI provider
- **Custom aliases** for easy management
Test the AI setup:
```bash
# Test Gemini integration
echo "Test text" | fabric -p summarize
# Test local Ollama integration
echo "Test text" | fabric -p summarize -m ollama:phi3:mini
```
## Dotfiles ## Dotfiles
The repository includes dotfiles for system configuration in the `dotfiles` directory. These can be automatically set up using the bootstrap script: The repository includes dotfiles for system configuration in the `dotfiles` directory. These can be automatically set up using the bootstrap script:

108
TABLE_OF_CONTENTS.md Normal file
View File

@@ -0,0 +1,108 @@
# 📚 Documentation Table of Contents
This is a comprehensive index of all documentation available in this shell scripts and dotfiles repository.
## 🏠 Main Documentation
- [**README.md**](./README.md) - Main repository documentation and overview
- [**backup-wants.md**](./backup-wants.md) - Backup requirements and wishlist
## 🚀 Setup & Configuration
- [**Setup Complete Guide**](./setup/SETUP_COMPLETE.md) - Complete setup documentation for Ollama + Fabric integration
## 🛠️ Component Documentation
### Backup Systems
- [**Backup Media Documentation**](./docs/backup-media.md) - Media backup system documentation
- [**Enhanced Media Backup**](./docs/enhanced-media-backup.md) - Advanced media backup features
- [**Backup Media Enhancement Summary**](./docs/backup-media-enhancement-summary.md) - Summary of backup enhancements
- [**Environment Backup System**](./docs/env-backup-system.md) - Environment backup documentation
- [**Environment Backup Integration Guide**](./docs/env-backup-integration-guide.md) - Integration guide for env backups
- [**Backup Environment Safety Guide**](./docs/backup-env-safety-guide.md) - Safety guidelines for environment backups
### Plex Media Server
- [**Plex README**](./plex/README.md) - Plex scripts overview
- [**Plex Backup Documentation**](./plex/plex-backup.md) - Plex backup system guide
- [**Plex Management Guide**](./plex/plex-management.md) - Plex server management documentation
### Immich Photo Management
- [**Immich README**](./immich/README.md) - Immich backup scripts overview
- [**Immich Backup Prompt**](./immich/.prompts/immich-backup.prompt.md) - Immich backup automation prompt
- [**Immich Backup Enhancement Summary**](./docs/immich-backup-enhancement-summary.md) - Enhancement summary
- [**Immich Backup Migration Summary**](./docs/immich-backup-migration-summary.md) - Migration documentation
### Crontab & Scheduling
- [**Crontab README**](./crontab/README.md) - Crontab management overview
- [**Enhanced Crontab System**](./crontab/enhanced-crontab-system.md) - Advanced crontab management
- [**Multi-System Crontab Management**](./crontab/multi-system-crontab-management.md) - Multi-server crontab coordination
### System Configuration
- [**Dotfiles README**](./dotfiles/README.md) - Dotfiles setup and configuration guide
- [**Completions README**](./completions/README.md) - Tab completion system documentation
### Telegram Integration
- [**Telegram Bot README**](./telegram/github-issues/README.md) - Telegram backup monitoring bot
## 🧪 Development & Testing
- [**Docker Bootstrap Testing Framework**](./docs/docker-bootstrap-testing-framework.md) - Comprehensive testing framework documentation
- [**Docker Stack Deployment Strategy**](./docs/docker-stack-deployment-strategy.md) - Deployment strategy guide
- [**Production Deployment Guide**](./docs/production-deployment-guide.md) - Production deployment documentation
## 🔧 Technical Guides
### System Utilities
- [**Folder Metrics**](./docs/folder-metrics.md) - Directory analysis and metrics documentation
- [**Cleanup Alias Tracking**](./docs/cleanup-alias-tracking.md) - Alias management and cleanup guide
### Development Features
- [**Tab Completion Implementation Summary**](./docs/tab-completion-implementation-summary.md) - Tab completion system implementation
- [**Package Detection Fix Summary**](./docs/package-detection-fix-summary.md) - Package detection improvements
## 📊 Project Summaries & Reviews
- [**Documentation Review Summary**](./docs/documentation-review-summary.md) - Comprehensive documentation review
- [**Project Completion Summary**](./docs/project-completion-summary.md) - Overall project completion status
- [**Issue 11 Completion Summary**](./docs/issue-11-completion-summary.md) - Specific issue resolution summary
## ⚙️ Developer Configuration
- [**GitHub Copilot Instructions**](./.github/copilot-instructions.md) - AI assistant configuration and guidelines
---
## 📂 Directory Structure
```
📁 docs/ - Technical documentation and guides
📁 setup/ - Setup and configuration files
📁 plex/ - Plex Media Server scripts and docs
📁 immich/ - Immich photo management scripts
📁 crontab/ - Crontab management system
📁 dotfiles/ - System configuration files
📁 completions/ - Tab completion scripts
📁 telegram/ - Telegram bot integration
📁 .github/ - GitHub and development configuration
```
## 🏷️ Documentation Categories
- **🚀 Setup & Getting Started**: Essential setup guides and initial configuration
- **🛠️ System Administration**: Server management, backups, and maintenance
- **📱 Media Management**: Plex and Immich configuration and backup systems
- **🤖 Automation**: Crontab scheduling and Telegram bot integration
- **🧪 Development**: Testing frameworks and deployment strategies
- **📚 Reference**: Technical guides and troubleshooting documentation
---
*Last updated: May 29, 2025*

190
docker-deployment/README.md Normal file
View File

@@ -0,0 +1,190 @@
# 🐳 Docker Deployment Management
This directory contains scripts and tools for managing Docker stack deployments across multiple servers in a coordinated, automated fashion.
## 📋 Overview
The Docker deployment system provides centralized management for deploying specific Docker stacks to designated servers while maintaining a monorepo structure. It integrates with the existing environment backup system to ensure proper configuration management.
## 🚀 Available Scripts
### Core Deployment Scripts
- **`docker-deployment-manager.sh`**: Main orchestrator for managing Docker stack deployments across multiple servers
- **`deployment-env-integration.sh`**: Integration bridge between the deployment manager and existing environment backup system
- **`stack-assignment-helper.sh`**: Intelligent stack analysis and server assignment recommendations
## 🎯 Key Features
### Multi-Server Deployment
- **Centralized Control**: Manage deployments across multiple servers from a single interface
- **Stack-to-Server Mapping**: Intelligent assignment of Docker stacks to appropriate servers
- **Configuration Synchronization**: Automated sync of environment files and configurations
### Environment Integration
- **Backup Integration**: Seamless integration with existing backup-env-files system
- **Configuration Management**: Automated management of environment variables and secrets
- **Deployment Verification**: Post-deployment validation and health checks
### Intelligent Stack Management
- **Pattern Recognition**: Automatic categorization of stacks based on content analysis
- **Resource Optimization**: Server assignment based on resource requirements and capabilities
- **Conflict Detection**: Prevention of conflicting stack deployments
## 📊 System Architecture
```
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Central Repo │───▶│ Deployment Mgr │───▶│ Target Servers │
│ (Monorepo) │ │ │ │ │
└─────────────────┘ └──────────────────┘ └─────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Stack Analysis │ │ Env Integration │ │ Health Checks │
│ Helper │ │ │ │ │
└─────────────────┘ └──────────────────┘ └─────────────────┘
```
## 🛠️ Usage Examples
### Basic Deployment
```bash
# Deploy a specific stack to assigned server
./docker-deployment-manager.sh deploy plex
# Deploy multiple stacks
./docker-deployment-manager.sh deploy plex immich nextcloud
# Deploy to specific server
./docker-deployment-manager.sh deploy --server europa plex
```
### Stack Analysis and Assignment
```bash
# Analyze all stacks and suggest assignments
./stack-assignment-helper.sh analyze
# Get recommendations for a specific stack
./stack-assignment-helper.sh recommend plex
# Show current stack assignments
./stack-assignment-helper.sh list-assignments
```
### Environment Integration
```bash
# Sync environment files before deployment
./deployment-env-integration.sh sync
# Deploy with automatic env backup
./deployment-env-integration.sh deploy-with-backup plex
# Validate environment configuration
./deployment-env-integration.sh validate
```
## ⚙️ Configuration
### Server Configuration
The deployment system uses hostname-based server configuration:
- **europa** - Media server (Plex, Immich, media-related stacks)
- **io** - Download/acquisition server (download tools, VPN services)
- **racknerd** - Backup server (backup services, monitoring)
### Stack Categories
#### Media Server Stacks
- Plex, Jellyfin, Emby
- Immich, PhotoPrism
- Nextcloud, file sharing services
#### Download/Acquisition Stacks
- qBittorrent, Transmission
- Sonarr, Radarr, Lidarr
- VPN services, proxy tools
#### Backup & Monitoring Stacks
- Backup services
- Monitoring tools (Prometheus, Grafana)
- Notification services
#### Multi-Server Stacks
- Traefik (reverse proxy)
- Watchtower (auto-updates)
- Portainer (management)
## 🔧 Integration Points
### Environment Backup System
- Automatic backup of environment files before deployment
- Restoration capabilities for rollback scenarios
- Configuration validation and consistency checks
### Monitoring Integration
- Deployment status tracking
- Health check integration
- Alert system for failed deployments
### Documentation Integration
- Links to [Docker Stack Deployment Strategy](../docs/docker-stack-deployment-strategy.md)
- Integration with [Production Deployment Guide](../docs/production-deployment-guide.md)
## 🧪 Validation and Testing
### Pre-Deployment Checks
- Environment file validation
- Docker stack syntax verification
- Server resource availability
### Post-Deployment Validation
- Container health status
- Service accessibility checks
- Integration point verification
### Rollback Capabilities
- Automatic rollback on deployment failure
- Manual rollback commands
- Configuration restoration
## 📝 Logging and Monitoring
- **Deployment Logs**: Detailed logs for all deployment activities
- **Error Tracking**: Comprehensive error logging and reporting
- **Performance Metrics**: Deployment time and resource usage tracking
## 🔗 Related Documentation
- [Docker Stack Deployment Strategy](../docs/docker-stack-deployment-strategy.md) - Comprehensive deployment strategy
- [Production Deployment Guide](../docs/production-deployment-guide.md) - Production deployment procedures
- [Environment Backup System](../docs/env-backup-system.md) - Environment backup integration
- [Crontab Management](../crontab/README.md) - Automated scheduling integration
## 🎯 Future Enhancements
- **Blue-Green Deployments**: Zero-downtime deployment strategies
- **Canary Releases**: Gradual rollout capabilities
- **Auto-Scaling**: Dynamic resource allocation
- **Multi-Cloud Support**: Cross-cloud deployment capabilities
---
*For detailed usage instructions and advanced configuration options, see the individual script documentation and related guides.*

0
deployment-env-integration.sh → docker-deployment/deployment-env-integration.sh
View File

0
docker-deployment-manager.sh → docker-deployment/docker-deployment-manager.sh
View File

0
stack-assignment-helper.sh → docker-deployment/stack-assignment-helper.sh
View File

115
setup/SETUP_COMPLETE.md
View File

@@ -1,115 +0,0 @@
# 🎉 Setup Complete: Ollama + Fabric Integration
## ✅ What's Been Accomplished
### 1. **Ollama Docker Setup**
- ✅ Ollama running in Docker container (`3d8eb0b5caef`)
- ✅ Accessible on port 11434
- ✅ phi3:mini model installed (2.2 GB)
- ✅ API responding correctly
### 2. **Fabric Installation & Configuration**
- ✅ Fabric v1.4.195 installed
- ✅ 216 patterns available
- ✅ Configured to use Google Gemini 2.5 Pro as default provider
- ✅ Environment variables set correctly in `~/.config/fabric/.env`
- ✅ Ollama available as secondary provider
### 3. **Shell Configuration**
- ✅ Zsh configured as default shell
- ✅ Oh My Zsh installed with plugins
- ✅ Custom aliases configured for the system
- ✅ Ollama Docker management aliases created
### 4. **Docker Integration**
- ✅ Docker permission handling configured
- ✅ Ollama container management aliases working
- ✅ Automatic restart policy set for container
### 5. **Development Tools**
- ✅ All packages from packages.list installed
- ✅ VS Code repository configured
- ✅ Node.js (via nvm), zoxide, and other tools ready
- ✅ Bash completion for scripts configured
## 🚀 How to Use
### Basic Fabric Commands
```bash
# List all available patterns
fabric -l
# Use a pattern (uses Gemini 2.5 Pro by default)
echo "Your text here" | fabric -p summarize
# Use with specific model
echo "Your text here" | fabric -p summarize -m gemini-2.0-flash-exp
# Use Ollama models when needed
echo "Your text here" | fabric -p summarize -m ollama:phi3:mini
# List available models
fabric -L
# Update patterns
fabric -U
```
### Ollama Management
```bash
# List installed models
ollama list
# Install a new model
ollama pull llama2
# Container management
ollama-start # Start container
ollama-stop # Stop container
ollama-restart # Restart container
ollama-logs # View logs
ollama-status # Check status
```
### Popular Fabric Patterns
- `summarize` - Summarize text
- `explain_code` - Explain code snippets
- `improve_writing` - Improve writing quality
- `extract_wisdom` - Extract key insights
- `create_quiz` - Generate quiz questions
- `analyze_claims` - Analyze claims in text
## 🔧 System Details
- **OS**: Fedora 42
- **Package Manager**: DNF
- **Shell**: Zsh with Oh My Zsh
- **Primary AI Provider**: Google Gemini 2.5 Pro
- **Secondary Provider**: Ollama running in Docker on port 11434
- **Fabric**: v1.4.195 with 216 patterns
- **Local Model**: phi3:mini (3.8B parameters) available via Ollama
## 🎯 Next Steps
1. **Explore Patterns**: Try different Fabric patterns with Gemini 2.5 Pro
2. **Compare Models**: Test patterns with both Gemini and local Ollama models
3. **Customize**: Add your own patterns to `~/.config/fabric/patterns`
4. **Integrate**: Use Fabric in your development workflow
5. **Update**: Run `fabric -U` periodically to get new patterns
## 📝 Configuration Files
- Fabric config: `~/.config/fabric/.env`
- Ollama aliases: `~/.oh-my-zsh/custom/ollama-aliases.zsh`
- Shell config: `~/.zshrc`
## 🧪 Test the Setup
Run this command to test Gemini integration:
```bash
echo "This is a test of the Gemini and Fabric integration" | fabric -p summarize
```
Test Ollama integration:
```bash
echo "This is a test of the Ollama and Fabric integration" | fabric -p summarize -m ollama:phi3:mini
```
**Status**: ✅ **FULLY FUNCTIONAL** - Ready for AI-assisted development with Google Gemini 2.5 Pro!