site11-backups/README.md

Site11 Backups

MongoDB database backups for the Site11 project.

Overview

This repository contains automated backups of all MongoDB databases used in the Site11 platform, including:

• News API database
• AI Writer database
• Service-specific databases (AI, Apple, Crypto, Google, Samsung, etc.)
• Artist databases (NCT, TWICE, ENHYPEN, IVE, Stray Kids, BLACKPINK)
• OAuth and authentication databases

Backup Schedule

Backups are created regularly and stored in compressed tar.gz format:

• Format: mongodb-YYYYMMDD-HHMMSS.tar.gz
• Compression: gzip
• Structure: Full dump using mongodump

Backup Contents

Each backup includes:

• All databases and collections
• Indexes
• User data
• Configuration data

Latest Backup

• Date: 2025-10-22 09:36:51
• File: mongodb-20251022-093651.tar.gz
• Size: 233 MB (compressed), 768 MB (uncompressed)
• Databases: 24 databases

Restoring from Backup

Extract the Backup

# Extract the tar.gz file
tar -xzf mongodb-YYYYMMDD-HHMMSS.tar.gz

# This will create a 'backup' directory with all database dumps
cd mongodb-YYYYMMDD-HHMMSS/backup

Restore All Databases

# Restore all databases
mongorestore --host localhost --port 27017 backup/

# Or restore to Docker container
docker exec -i site11_mongodb mongorestore --archive < backup.archive

Restore Specific Database

# Restore only a specific database
mongorestore --host localhost --port 27017 --db ai_writer_db backup/ai_writer_db

# For Docker
docker cp backup/ai_writer_db site11_mongodb:/tmp/
docker exec site11_mongodb mongorestore --db ai_writer_db /tmp/ai_writer_db

Restore with Drop (Replace existing data)

# WARNING: This will drop existing collections before restoring
mongorestore --host localhost --port 27017 --drop backup/

Backup Details

Included Databases

Database	Size	Description
ai_writer_db	461 MB	Main AI content generation database
news_api_db	~7 MB	News API data and outlets
ai_service	~0.18 MB	AI service configuration
apple_service	~0.18 MB	Apple news service
crypto_service	~0.18 MB	Cryptocurrency news
google_service	~0.18 MB	Google news service
samsung_service	~0.18 MB	Samsung news service
lg_service	~0.18 MB	LG news service
wsj_service	~0.18 MB	Wall Street Journal service
musk_service	~0.18 MB	Elon Musk news service
korea_service	~0.18 MB	Korea news service
artist_*_service	~0.18 MB each	K-pop artist news services
oauth_db	~0.05 MB	Authentication and OAuth data
pipeline_db	~0.32 MB	Data pipeline configuration
k_culture_db	~0.05 MB	Korean culture content
files_db	~0.03 MB	File storage metadata

Creating a New Backup

Using Docker

# Set backup directory
BACKUP_DIR="backups/mongodb-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Create backup
docker exec site11_mongodb mongodump --out /tmp/backup --quiet
docker cp site11_mongodb:/tmp/backup "$BACKUP_DIR/"
docker exec site11_mongodb rm -rf /tmp/backup

# Compress
cd backups
tar -czf "$(basename "$BACKUP_DIR").tar.gz" "$(basename "$BACKUP_DIR")"

# Optional: Remove uncompressed backup
rm -rf "$(basename "$BACKUP_DIR")"

Using MongoDB Tools

# Direct backup
mongodump --host localhost --port 27017 --out backup/$(date +%Y%m%d-%H%M%S)

# Compress
tar -czf backup-$(date +%Y%m%d-%H%M%S).tar.gz backup/

Backup Verification

After creating a backup, verify its integrity:

# Check tar.gz file integrity
tar -tzf mongodb-YYYYMMDD-HHMMSS.tar.gz > /dev/null && echo "OK" || echo "CORRUPTED"

# Check file size
ls -lh mongodb-YYYYMMDD-HHMMSS.tar.gz

# List databases in backup
tar -xzf mongodb-YYYYMMDD-HHMMSS.tar.gz -O backup/ | head -20

Storage Recommendations

• Keep at least 3 recent backups (daily)
• Keep weekly backups for the last month
• Keep monthly backups for the last year
• Store critical backups in multiple locations:
  • Local storage
  • Git repository (this repo)
  • Cloud storage (S3, GCS, etc.)
  • External drive

Security Notes

• Backups contain sensitive data including user information
• Ensure proper access controls on this repository
• Consider encryption for production backups
• Rotate backups regularly and delete old backups securely

Automation

To automate backups, create a cron job:

# Edit crontab
crontab -e

# Add daily backup at 2 AM
0 2 * * * cd /path/to/site11 && ./scripts/backup-mongodb.sh

Emergency Recovery

In case of data loss:

1. Stop the application to prevent data corruption
2. Identify the most recent valid backup
3. Extract and verify the backup
4. Restore the database using mongorestore
5. Verify data integrity after restoration
6. Restart the application

Support

For backup-related issues:

1. Check MongoDB logs
2. Verify disk space
3. Test restore on a separate instance first
4. Contact the development team if needed

License

Proprietary - All rights reserved