# Data Persistence Configuration

## Overview
All data services are configured to use bind mounts to local directories for data persistence. This ensures data survives container restarts and rebuilds.

## Directory Structure
```
data/
├── mongodb/          # MongoDB database files
├── redis/            # Redis persistence files  
├── kafka/            # Kafka log data
├── zookeeper/        # Zookeeper data and logs
│   ├── data/
│   └── logs/
├── minio/            # MinIO object storage
├── solr/             # Solr search index
├── files-temp/       # Temporary file storage
└── images-cache/     # Image processing cache
```

## Volume Mappings

### MongoDB
- `./data/mongodb:/data/db` - Database files
- `./data/mongodb/configdb:/data/configdb` - Configuration database

### Redis
- `./data/redis:/data` - RDB snapshots and AOF logs

### Kafka
- `./data/kafka:/var/lib/kafka/data` - Message logs

### Zookeeper
- `./data/zookeeper/data:/var/lib/zookeeper/data` - Coordination data
- `./data/zookeeper/logs:/var/lib/zookeeper/log` - Transaction logs

### MinIO
- `./data/minio:/data` - Object storage buckets

### Solr
- `./data/solr:/var/solr` - Search index and configuration

### Application Caches
- `./data/files-temp:/tmp` - Temporary file processing
- `./data/images-cache:/app/cache` - Processed image cache

## Backup and Restore

### Backup All Data
```bash
# Stop services
docker-compose down

# Create backup
tar -czf backup-$(date +%Y%m%d).tar.gz data/

# Restart services
docker-compose up -d
```

### Restore Data
```bash
# Stop services
docker-compose down

# Extract backup
tar -xzf backup-YYYYMMDD.tar.gz

# Restart services
docker-compose up -d
```

### Individual Service Backups

#### MongoDB Backup
```bash
docker exec site11_mongodb mongodump --out /data/db/backup
tar -czf mongodb-backup.tar.gz data/mongodb/backup/
```

#### Redis Backup
```bash
docker exec site11_redis redis-cli BGSAVE
# Wait for completion
cp data/redis/dump.rdb redis-backup-$(date +%Y%m%d).rdb
```

## Permissions
Ensure proper permissions for data directories:
```bash
# Set appropriate permissions
chmod -R 755 data/
```

## Disk Space Monitoring
Monitor disk usage regularly:
```bash
# Check data directory size
du -sh data/*

# Check individual services
du -sh data/mongodb
du -sh data/minio
du -sh data/kafka
```

## Clean Up Old Data

### Clear Kafka Logs (older than 7 days)
```bash
docker exec site11_kafka kafka-log-dirs.sh --describe --bootstrap-server localhost:9092
```

### Clear Image Cache
```bash
rm -rf data/images-cache/*
```

### Clear Temporary Files
```bash
rm -rf data/files-temp/*
```

## Migration from Docker Volumes
If migrating from named Docker volumes to bind mounts:

1. Export data from Docker volumes:
```bash
docker run --rm -v site11_mongodb_data:/source -v $(pwd)/data/mongodb:/dest alpine cp -av /source/. /dest/
```

2. Update docker-compose.yml (already done)

3. Restart services with new configuration

## Notes
- The `data/` directory is excluded from git via .gitignore
- Ensure sufficient disk space for data growth
- Consider setting up automated backups for production
- Monitor disk I/O performance for database services