site11-backups/README.md

# Site11 Backups

MongoDB database backups for the Site11 project.

## Overview

This repository contains automated backups of all MongoDB databases used in the Site11 platform, including:
- News API database
- AI Writer database
- Service-specific databases (AI, Apple, Crypto, Google, Samsung, etc.)
- Artist databases (NCT, TWICE, ENHYPEN, IVE, Stray Kids, BLACKPINK)
- OAuth and authentication databases

## Backup Schedule

Backups are created regularly and stored in compressed tar.gz format:
- **Format**: `mongodb-YYYYMMDD-HHMMSS.tar.gz`
- **Compression**: gzip
- **Structure**: Full dump using `mongodump`

## Backup Contents

Each backup includes:
- All databases and collections
- Indexes
- User data
- Configuration data

### Latest Backup

- **Date**: 2025-10-22 09:36:51
- **File**: `mongodb-20251022-093651.tar.gz`
- **Size**: 233 MB (compressed), 768 MB (uncompressed)
- **Databases**: 24 databases

## Restoring from Backup

### Extract the Backup

```bash
# Extract the tar.gz file
tar -xzf mongodb-YYYYMMDD-HHMMSS.tar.gz

# This will create a 'backup' directory with all database dumps
cd mongodb-YYYYMMDD-HHMMSS/backup
```

### Restore All Databases

```bash
# Restore all databases
mongorestore --host localhost --port 27017 backup/

# Or restore to Docker container
docker exec -i site11_mongodb mongorestore --archive < backup.archive
```

### Restore Specific Database

```bash
# Restore only a specific database
mongorestore --host localhost --port 27017 --db ai_writer_db backup/ai_writer_db

# For Docker
docker cp backup/ai_writer_db site11_mongodb:/tmp/
docker exec site11_mongodb mongorestore --db ai_writer_db /tmp/ai_writer_db
```

### Restore with Drop (Replace existing data)

```bash
# WARNING: This will drop existing collections before restoring
mongorestore --host localhost --port 27017 --drop backup/
```

## Backup Details

### Included Databases

| Database | Size | Description |
|----------|------|-------------|
| ai_writer_db | 461 MB | Main AI content generation database |
| news_api_db | ~7 MB | News API data and outlets |
| ai_service | ~0.18 MB | AI service configuration |
| apple_service | ~0.18 MB | Apple news service |
| crypto_service | ~0.18 MB | Cryptocurrency news |
| google_service | ~0.18 MB | Google news service |
| samsung_service | ~0.18 MB | Samsung news service |
| lg_service | ~0.18 MB | LG news service |
| wsj_service | ~0.18 MB | Wall Street Journal service |
| musk_service | ~0.18 MB | Elon Musk news service |
| korea_service | ~0.18 MB | Korea news service |
| artist_*_service | ~0.18 MB each | K-pop artist news services |
| oauth_db | ~0.05 MB | Authentication and OAuth data |
| pipeline_db | ~0.32 MB | Data pipeline configuration |
| k_culture_db | ~0.05 MB | Korean culture content |
| files_db | ~0.03 MB | File storage metadata |

## Creating a New Backup

### Using Docker

```bash
# Set backup directory
BACKUP_DIR="backups/mongodb-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Create backup
docker exec site11_mongodb mongodump --out /tmp/backup --quiet
docker cp site11_mongodb:/tmp/backup "$BACKUP_DIR/"
docker exec site11_mongodb rm -rf /tmp/backup

# Compress
cd backups
tar -czf "$(basename "$BACKUP_DIR").tar.gz" "$(basename "$BACKUP_DIR")"

# Optional: Remove uncompressed backup
rm -rf "$(basename "$BACKUP_DIR")"
```

### Using MongoDB Tools

```bash
# Direct backup
mongodump --host localhost --port 27017 --out backup/$(date +%Y%m%d-%H%M%S)

# Compress
tar -czf backup-$(date +%Y%m%d-%H%M%S).tar.gz backup/
```

## Backup Verification

After creating a backup, verify its integrity:

```bash
# Check tar.gz file integrity
tar -tzf mongodb-YYYYMMDD-HHMMSS.tar.gz > /dev/null && echo "OK" || echo "CORRUPTED"

# Check file size
ls -lh mongodb-YYYYMMDD-HHMMSS.tar.gz

# List databases in backup
tar -xzf mongodb-YYYYMMDD-HHMMSS.tar.gz -O backup/ | head -20
```

## Storage Recommendations

- Keep at least **3 recent backups** (daily)
- Keep **weekly backups** for the last month
- Keep **monthly backups** for the last year
- Store critical backups in **multiple locations**:
  - Local storage
  - Git repository (this repo)
  - Cloud storage (S3, GCS, etc.)
  - External drive

## Security Notes

- Backups contain **sensitive data** including user information
- Ensure proper **access controls** on this repository
- Consider **encryption** for production backups
- Rotate backups regularly and **delete old backups** securely

## Automation

To automate backups, create a cron job:

```bash
# Edit crontab
crontab -e

# Add daily backup at 2 AM
0 2 * * * cd /path/to/site11 && ./scripts/backup-mongodb.sh
```

## Emergency Recovery

In case of data loss:

1. **Stop the application** to prevent data corruption
2. **Identify the most recent valid backup**
3. **Extract and verify** the backup
4. **Restore the database** using mongorestore
5. **Verify data integrity** after restoration
6. **Restart the application**

## Support

For backup-related issues:
1. Check MongoDB logs
2. Verify disk space
3. Test restore on a separate instance first
4. Contact the development team if needed

## License

Proprietary - All rights reserved