mirror of
https://github.com/acedanger/shell.git
synced 2025-12-06 06:40:13 -08:00
c9b69ea789
- Introduced `restore-plex.sh` for restoring Plex backups with logging and validation. - Created `test-plex-backup.sh` for automated testing of backup functionalities. - Developed `validate-plex-backups.sh` for validating backup integrity and monitoring. - Updated `update.sh` to reference the correct path for Plex service management.
227 lines
5.6 KiB
Markdown
227 lines
5.6 KiB
Markdown
# Enhanced Crontab and Backup Monitoring System
|
|
|
|
## Overview
|
|
|
|
This enhanced system provides comprehensive crontab management with automatic backups, system logging, and advanced monitoring capabilities for your backup operations.
|
|
|
|
## Components
|
|
|
|
### 1. Crontab Entries (`crontab-*.txt`)
|
|
|
|
- **Move backups** (01:00): Transfers Docker backups with logging
|
|
- **Plex backup** (04:15): Daily Plex database backup with auto-repair
|
|
- **Plex validation** (07:00): Validates and fixes backup integrity
|
|
- **Immich backup** (05:00 Mon): Weekly Immich database backup move
|
|
- **Weekly report** (08:00 Sun): Comprehensive backup status report
|
|
|
|
### 2. Crontab Management (`manage-enhanced-crontab.sh`)
|
|
|
|
- Install enhanced crontab entries with validation
|
|
- Verify all backup scripts exist and are executable
|
|
- Setup automated backups and log rotation
|
|
- Integration with backup and monitoring systems
|
|
|
|
### 3. Crontab Backup System (`crontab-backup-system.sh`)
|
|
|
|
- Automated timestamped crontab backups
|
|
- Backup comparison and restoration
|
|
- Syntax validation
|
|
- Automated cleanup of old backups
|
|
- Daily automated backup scheduling
|
|
|
|
### 4. Backup Log Monitor (`backup-log-monitor.sh`)
|
|
|
|
- Real-time log monitoring with color coding
|
|
- Error analysis and reporting
|
|
- System health checks
|
|
- Comprehensive backup reports
|
|
- Service status overview
|
|
|
|
## Quick Start
|
|
|
|
### 1. Install the Enhanced System
|
|
|
|
```bash
|
|
# Make scripts executable
|
|
chmod +x manage-enhanced-crontab.sh crontab-backup-system.sh backup-log-monitor.sh
|
|
|
|
# Install enhanced crontab with all features
|
|
sudo ./manage-enhanced-crontab.sh install
|
|
```
|
|
|
|
### 2. Monitor Your Backups
|
|
|
|
```bash
|
|
# Real-time monitoring
|
|
./backup-log-monitor.sh monitor
|
|
|
|
# Check system health
|
|
./backup-log-monitor.sh health
|
|
|
|
# View recent activity
|
|
./backup-log-monitor.sh recent 24
|
|
|
|
# Generate weekly report
|
|
./backup-log-monitor.sh report 7
|
|
```
|
|
|
|
### 3. Manage Crontab Backups
|
|
|
|
```bash
|
|
# Create manual backup
|
|
./crontab-backup-system.sh backup manual
|
|
|
|
# List all backups
|
|
./crontab-backup-system.sh list
|
|
|
|
# Compare current with backup
|
|
./crontab-backup-system.sh compare current
|
|
|
|
# System status
|
|
./crontab-backup-system.sh status
|
|
```
|
|
|
|
## Features
|
|
|
|
### Enhanced Logging
|
|
|
|
All backup operations now log to syslog with specific tags:
|
|
|
|
- `plex-backup`: Plex database backup operations
|
|
- `backup-move`: Docker backup file transfers
|
|
- `plex-validation`: Backup integrity checks
|
|
- `immich-backup`: Immich database operations
|
|
- `plex-report`: Weekly reporting
|
|
- `crontab-backup`: Crontab backup operations
|
|
|
|
### Automatic Backups
|
|
|
|
- **Crontab backups**: Daily automated backups at midnight
|
|
- **Cleanup**: Automatic removal of backups older than 30 days
|
|
- **Validation**: Syntax checking before applying changes
|
|
- **Recovery**: Easy restoration from any backup point
|
|
|
|
### Health Monitoring
|
|
|
|
- Script existence and permissions
|
|
- Backup directory availability
|
|
- Recent activity tracking
|
|
- Error rate monitoring
|
|
- Overall system health scoring
|
|
|
|
### Error Handling
|
|
|
|
- Graceful failure handling in cron jobs
|
|
- Detailed error logging and reporting
|
|
- Exit code tracking for debugging
|
|
- Comprehensive error summaries
|
|
|
|
## Log Analysis
|
|
|
|
### View Real-time Logs
|
|
|
|
```bash
|
|
# All backup services
|
|
sudo journalctl -f -t plex-backup -t backup-move -t plex-validation -t immich-backup -t plex-report
|
|
|
|
# Specific service
|
|
sudo journalctl -f -t plex-backup
|
|
|
|
# With our monitor (recommended)
|
|
./backup-log-monitor.sh monitor
|
|
```
|
|
|
|
### Historical Analysis
|
|
|
|
```bash
|
|
# Last 24 hours
|
|
sudo journalctl --since '24 hours ago' -t plex-backup
|
|
|
|
# Last week with errors only
|
|
sudo journalctl --since '1 week ago' --priority=err -t plex-backup
|
|
|
|
# Using our tools
|
|
./backup-log-monitor.sh recent 24 plex-backup
|
|
./backup-log-monitor.sh errors 7
|
|
```
|
|
|
|
## Backup Recovery
|
|
|
|
### Restore Crontab from Backup
|
|
|
|
```bash
|
|
# List available backups
|
|
./crontab-backup-system.sh list
|
|
|
|
# Restore specific backup
|
|
./crontab-backup-system.sh restore crontab-manual-20250526_120000.backup
|
|
|
|
# Compare before restoring
|
|
./crontab-backup-system.sh compare current crontab-manual-20250526_120000.backup
|
|
```
|
|
|
|
### Emergency Recovery
|
|
|
|
If you need to quickly restore the original crontab:
|
|
|
|
```bash
|
|
# The system automatically creates pre-install backups
|
|
./crontab-backup-system.sh list | grep pre-install
|
|
./crontab-backup-system.sh restore [backup-filename]
|
|
```
|
|
|
|
## Maintenance
|
|
|
|
### Regular Tasks
|
|
|
|
```bash
|
|
# Weekly health check
|
|
./backup-log-monitor.sh health
|
|
|
|
# Monthly backup cleanup
|
|
./crontab-backup-system.sh cleanup 30
|
|
|
|
# Quarterly comprehensive report
|
|
./backup-log-monitor.sh report 90
|
|
```
|
|
|
|
### Troubleshooting
|
|
|
|
```bash
|
|
# Verify all components
|
|
./manage-enhanced-crontab.sh verify
|
|
|
|
# Check system status
|
|
./manage-enhanced-crontab.sh status
|
|
|
|
# View configuration
|
|
./manage-enhanced-crontab.sh show
|
|
|
|
# Monitor for issues
|
|
./backup-log-monitor.sh monitor
|
|
```
|
|
|
|
## Integration Notes
|
|
|
|
- All scripts follow the established shell repository coding standards
|
|
- Logging uses consistent tags and priorities
|
|
- Error handling preserves backup integrity
|
|
- Color-coded output for better readability
|
|
- Comprehensive documentation and help systems
|
|
|
|
## Security Considerations
|
|
|
|
- Scripts validate input and handle errors gracefully
|
|
- Backup files include metadata for tracking
|
|
- Permissions are properly managed
|
|
- Sensitive operations require sudo confirmation
|
|
- All operations are logged for audit trails
|
|
|
|
## Next Steps
|
|
|
|
1. **Install the system**: Run `sudo ./manage-enhanced-crontab.sh install`
|
|
2. **Test monitoring**: Use `./backup-log-monitor.sh monitor` during next backup
|
|
3. **Review reports**: Generate weekly reports to establish baseline
|
|
4. **Set up alerts**: Consider integrating with your notification system
|
|
5. **Document customizations**: Add any local modifications to this guide
|