shell/plex/docs/database-corruption-auto-repair-enhancement.md

# Enhanced Plex Backup Script - Database Corruption Auto-Repair

## Overview

The Plex backup script has been enhanced with comprehensive database corruption detection and automatic repair capabilities. These enhancements address critical corruption issues identified in the log analysis, including "database disk image is malformed," rowid ordering issues, and index corruption.

## Completed Enhancements

### 1. Enhanced Backup Verification (`verify_backup` function)

**Improvements:**

- Multiple retry strategies (3 attempts with progressive delays)
- Robust checksum calculation with error handling
- Enhanced database integrity checking for backup files
- Intelligent handling of checksum mismatches during file modifications

**Benefits:**

- Reduces false verification failures
- Better handling of timing issues during backup
- Database-specific validation for corrupt files

### 2. Enhanced Service Management (`manage_plex_service` function)

**New Features:**

- Force stop capabilities for stubborn Plex processes
- Progressive shutdown: systemctl stop → TERM signal → KILL signal
- Better process monitoring and status reporting
- Enhanced error handling with detailed service diagnostics

**Benefits:**

- Prevents database lock issues during repairs
- Ensures clean service state for critical operations
- Better recovery from service management failures

### 3. Enhanced WAL File Management (`handle_wal_files_for_repair` function)

**New Function Features:**

- Dedicated WAL handling for repair operations
- Three operation modes: prepare, cleanup, restore
- WAL checkpoint with TRUNCATE for complete consolidation
- Backup and restore of WAL/SHM files during repair

**Benefits:**

- Ensures database consistency during repairs
- Prevents WAL-related corruption during repair operations
- Proper state management for repair rollbacks

### 4. Enhanced Database Repair Strategy

**Modifications to `repair_database` function:**

- Integration with enhanced WAL handling
- Better error recovery and state management
- Improved cleanup and restoration on repair failure
- Multiple backup creation before repair attempts

**Repair Strategies (Progressive):**

1. **Dump and Restore**: SQL export/import for data preservation
2. **Schema Recreation**: Rebuild database structure with data recovery
3. **Backup Recovery**: Restore from previous backup as last resort

### 5. Preventive Corruption Detection (`detect_early_corruption` function)

**Early Warning System:**

- WAL file size anomaly detection (alerts if >10% of DB size)
- Quick integrity checks for performance optimization
- Foreign key violation detection
- Database statistics health monitoring

**Benefits:**

- Catches corruption before it becomes severe
- Enables proactive maintenance
- Reduces catastrophic database failures

### 6. Critical Database Operations Enhancement

**Improvements:**

- Force stop capability for integrity checking operations
- Better handling of corrupt databases during backup
- Enhanced error recovery and restoration
- Improved service state management during critical operations

## Corruption Issues Addressed

Based on log analysis from `plex-backup-2025-06-08.log`, the enhanced script addresses:

### Critical Issues Detected

```
- "Rowid 5837 out of order" → Handled by dump/restore strategy
- "Offset 38675 out of range 245..4092" → Fixed via schema recreation
- "row 1049 missing from index index_directories_on_path" → Index rebuilding
- "database disk image is malformed" → Multiple recovery strategies
```

### Previous Repair Limitations

- Old approach only tried VACUUM and REINDEX
- No fallback strategies when REINDEX failed
- Inadequate WAL file handling
- Poor service management during repairs

## Key Benefits

### 1. Automatic Corruption Detection

- Early warning system prevents severe corruption
- Proactive monitoring reduces backup failures
- Intelligent detection of corruption patterns

### 2. Multiple Repair Strategies

- Progressive approach from least to most destructive
- Data preservation prioritized over backup speed
- Fallback options when primary repair fails

### 3. Better Service Management

- Force stop prevents database lock issues
- Clean state enforcement for repairs
- Proper process monitoring and cleanup

### 4. Enhanced WAL Handling

- Proper WAL file management prevents corruption
- Consistent database state during operations
- Better recovery from WAL-related issues

### 5. Improved Verification

- Multiple retry strategies reduce false failures
- Database-specific validation for corrupted files
- Better handling of timing-related issues

### 6. Preventive Monitoring

- Early corruption indicators detected
- Proactive maintenance recommendations
- Health monitoring for database statistics

## Usage

The enhanced script maintains full backward compatibility while adding robust auto-repair:

```bash
# Standard backup with auto-repair (default)
./backup-plex.sh

# Backup without auto-repair (legacy mode)
./backup-plex.sh --disable-auto-repair

# Integrity check only with repair
./backup-plex.sh --check-integrity

# Non-interactive mode for automation
./backup-plex.sh --non-interactive
```

## Technical Implementation

### Auto-Repair Flow

1. **Detection**: Early corruption indicators or integrity check failure
2. **Preparation**: WAL handling, backup creation, service management
3. **Strategy 1**: Dump and restore approach (preserves most data)
4. **Strategy 2**: Schema recreation with table-by-table recovery
5. **Strategy 3**: Recovery from previous backup (last resort)
6. **Cleanup**: WAL restoration, service restart, file cleanup

### Error Handling

- Multiple backup creation before repair attempts
- State restoration on repair failure
- Comprehensive logging of all repair activities
- Graceful degradation when repairs fail

## Monitoring and Logging

Enhanced logging includes:

- Detailed repair attempt tracking
- Performance metrics for repair operations
- Early corruption warning indicators
- WAL file management activities
- Service management status and timing

## Future Enhancements

Potential areas for further improvement:

1. Machine learning-based corruption prediction
2. Automated backup rotation based on corruption patterns
3. Integration with external monitoring systems
4. Real-time corruption monitoring during operation

## Conclusion

The enhanced Plex backup script now provides comprehensive protection against database corruption while maintaining user data integrity. The multi-strategy repair approach ensures maximum data preservation, and the preventive monitoring helps catch issues before they become critical.