diff --git a/README.md b/README.md index 4dcaa96..fcc3a66 100644 --- a/README.md +++ b/README.md @@ -1,29 +1,430 @@ -# Site11 - Microservices Architecture +# Site11 - Multi-Language News Platform with Microservices Architecture -## Overview -Microservices platform with Console as API Gateway orchestrating multiple domain services. +## πŸ“‹ λͺ©μ°¨ (Table of Contents) +- [κ°œμš” (Overview)](#κ°œμš”-overview) +- [μ‹œμŠ€ν…œ μš”κ΅¬μ‚¬ν•­ (System Requirements)](#μ‹œμŠ€ν…œ-μš”κ΅¬μ‚¬ν•­-system-requirements) +- [λΉ λ₯Έ μ‹œμž‘ (Quick Start)](#λΉ λ₯Έ-μ‹œμž‘-quick-start) +- [상세 μ„€μΉ˜ κ°€μ΄λ“œ (Detailed Installation)](#상세-μ„€μΉ˜-κ°€μ΄λ“œ-detailed-installation) +- [μ„œλΉ„μŠ€ ꡬ쑰 (Service Architecture)](#μ„œλΉ„μŠ€-ꡬ쑰-service-architecture) +- [μ„œλΉ„μŠ€ 관리 (Service Management)](#μ„œλΉ„μŠ€-관리-service-management) +- [ν™˜κ²½ μ„€μ • (Configuration)](#ν™˜κ²½-μ„€μ •-configuration) +- [개발 κ°€μ΄λ“œ (Development Guide)](#개발-κ°€μ΄λ“œ-development-guide) +- [문제 ν•΄κ²° (Troubleshooting)](#문제-ν•΄κ²°-troubleshooting) +- [Git μ €μž₯μ†Œ 관리 (Repository Management)](#git-μ €μž₯μ†Œ-관리-repository-management) -## Quick Start +## κ°œμš” (Overview) -### Start Services -```bash -# Start console service -docker-compose up -d console-backend +Site11은 λ‹€κ΅­μ–΄ λ‰΄μŠ€ μ½˜ν…μΈ λ₯Ό μžλ™μœΌλ‘œ μˆ˜μ§‘, λ²ˆμ—­, μƒμ„±ν•˜λŠ” λ§ˆμ΄ν¬λ‘œμ„œλΉ„μŠ€ 기반 ν”Œλž«νΌμž…λ‹ˆλ‹€. -# Check status -curl http://localhost:8011/health +### μ£Όμš” κΈ°λŠ₯ +- 🌐 8개 μ–Έμ–΄ μžλ™ λ²ˆμ—­ (ν•œκ΅­μ–΄, μ˜μ–΄, 쀑ꡭ어, 일본어, ν”„λž‘μŠ€μ–΄, 독일어, μŠ€νŽ˜μΈμ–΄, μ΄νƒˆλ¦¬μ•„μ–΄) +- πŸ“° μ‹€μ‹œκ°„ λ‰΄μŠ€ μˆ˜μ§‘ 및 AI 기반 μ½˜ν…μΈ  생성 +- 🎨 μžλ™ 이미지 생성 및 μ΅œμ ν™” +- πŸ”„ 비동기 μž‘μ—… 처리 νŒŒμ΄ν”„λΌμΈ +- πŸ“Š μ‹€μ‹œκ°„ λͺ¨λ‹ˆν„°λ§ 및 톡계 + +## μ‹œμŠ€ν…œ μš”κ΅¬μ‚¬ν•­ (System Requirements) + +### ν•„μˆ˜ μš”κ΅¬μ‚¬ν•­ +- **Docker**: 20.10.0 이상 +- **Docker Compose**: 2.0.0 이상 +- **λ©”λͺ¨λ¦¬**: μ΅œμ†Œ 8GB RAM (16GB ꢌμž₯) +- **λ””μŠ€ν¬**: μ΅œμ†Œ 50GB μ—¬μœ  곡간 +- **OS**: Linux, macOS, Windows with WSL2 + +### 포트 μ‚¬μš© +``` +- 80: Nginx (Reverse Proxy) +- 8000-8100: λ§ˆμ΄ν¬λ‘œμ„œλΉ„μŠ€ λ°±μ—”λ“œ +- 3000-3100: ν”„λ‘ νŠΈμ—”λ“œ μ„œλΉ„μŠ€ +- 27017: MongoDB +- 6379: Redis +- 9092: Kafka +- 2181: Zookeeper +- 9000: MinIO +- 8983: Solr ``` -### Available Endpoints -- `http://localhost:8011/` - Root endpoint -- `http://localhost:8011/health` - Health check -- `http://localhost:8011/api/status` - System status +## λΉ λ₯Έ μ‹œμž‘ (Quick Start) -## Architecture -- **Console**: API Gateway and orchestrator -- **Services**: Domain-specific microservices (users, oauth, images, etc.) -- **Database**: MongoDB for persistence -- **Cache**: Redis for caching and pub/sub +### 1. μ €μž₯μ†Œ 클둠 +```bash +# HTTPS +git clone http://gitea.yakenator.io/aimond/site11.git -## Development -See `docs/PLAN.md` for implementation roadmap and `docs/PROGRESS.md` for current status. \ No newline at end of file +# 디렉토리 이동 +cd site11 +``` + +### 2. ν™˜κ²½ λ³€μˆ˜ μ„€μ • +```bash +# .env 파일 생성 +cp .env.example .env + +# ν•„μˆ˜ ν™˜κ²½ λ³€μˆ˜ μ„€μ • +echo "COMPOSE_PROJECT_NAME=site11" >> .env +echo "DEEPL_API_KEY=your-deepl-api-key" >> .env +echo "OPENAI_API_KEY=your-openai-api-key" >> .env +``` + +### 3. 전체 μ‹œμŠ€ν…œ μ‹œμž‘ +```bash +# λͺ¨λ“  μ„œλΉ„μŠ€ λΉŒλ“œ 및 μ‹œμž‘ (λ°±κ·ΈλΌμš΄λ“œ) +docker-compose up -d --build + +# μƒνƒœ 확인 +docker-compose ps + +# 둜그 확인 +docker-compose logs -f +``` + +### 4. μ„œλΉ„μŠ€ 확인 +```bash +# ν—¬μŠ€ 체크 +curl http://localhost:8000/health + +# MongoDB μ—°κ²° 확인 +docker exec -it site11_mongodb mongosh --eval "db.serverStatus()" +``` + +## 상세 μ„€μΉ˜ κ°€μ΄λ“œ (Detailed Installation) + +### 단계별 μ„œλΉ„μŠ€ μ‹œμž‘ + +#### 1단계: 핡심 인프라 +```bash +# MongoDB, Redis, Kafka μ‹œμž‘ +docker-compose up -d mongodb redis zookeeper kafka + +# 인프라 μ€€λΉ„ λŒ€κΈ° (μ•½ 30초) +sleep 30 + +# μƒνƒœ 확인 +docker-compose ps mongodb redis kafka +``` + +#### 2단계: Pipeline μ„œλΉ„μŠ€ +```bash +# Pipeline μ„œλΉ„μŠ€ μ‹œμž‘ +docker-compose up -d pipeline-scheduler pipeline-monitor \ + pipeline-rss-collector pipeline-google-search \ + pipeline-ai-generator pipeline-translator \ + pipeline-image-generator + +# Pipeline λͺ¨λ‹ˆν„° 확인 +curl http://localhost:8100/status +``` + +#### 3단계: λ‰΄μŠ€ μ„œλΉ„μŠ€ +```bash +# ꡭ가별 λ‰΄μŠ€ μ„œλΉ„μŠ€ +docker-compose up -d korea-backend korea-frontend \ + usa-backend usa-frontend \ + japan-backend japan-frontend \ + china-backend china-frontend + +# κΈ°μ—… λ‰΄μŠ€ μ„œλΉ„μŠ€ +docker-compose up -d samsung-backend samsung-frontend \ + apple-backend apple-frontend \ + google-backend google-frontend +``` + +#### 4단계: μ•„ν‹°μŠ€νŠΈ μ„œλΉ„μŠ€ +```bash +# K-POP μ•„ν‹°μŠ€νŠΈ μ„œλΉ„μŠ€ +docker-compose up -d k-culture-backend \ + blackpink-backend blackpink-frontend \ + ive-backend ive-frontend \ + twice-backend twice-frontend +``` + +## μ„œλΉ„μŠ€ ꡬ쑰 (Service Architecture) + +### 핡심 인프라 +- **MongoDB**: 데이터 μ €μž₯μ†Œ +- **Redis**: μΊμ‹œ 및 μž‘μ—… 큐 +- **Kafka**: 이벀트 슀트리밍 +- **MinIO**: 이미지/파일 μ €μž₯μ†Œ +- **Solr**: 검색 엔진 + +### Pipeline μ„œλΉ„μŠ€ +``` +pipeline-scheduler (8099) + β”œβ”€β”€ pipeline-rss-collector # RSS ν”Όλ“œ μˆ˜μ§‘ + β”œβ”€β”€ pipeline-google-search # Google 검색 μˆ˜μ§‘ + β”œβ”€β”€ pipeline-ai-generator # AI μ½˜ν…μΈ  생성 + β”œβ”€β”€ pipeline-translator # λ‹€κ΅­μ–΄ λ²ˆμ—­ + └── pipeline-image-generator # 이미지 생성 + +pipeline-monitor (8100) # μ‹€μ‹œκ°„ λͺ¨λ‹ˆν„°λ§ +``` + +### μ½˜ν…μΈ  μ„œλΉ„μŠ€ +- **Media Empire** (8200/3200): 톡합 λ‹€κ΅­μ–΄ λ‰΄μŠ€ 포털 +- **ꡭ가별 μ„œλΉ„μŠ€**: Korea, USA, Japan, China +- **κΈ°μ—… μ„œλΉ„μŠ€**: Samsung, Apple, Google, LG, Musk, WSJ +- **μ•„ν‹°μŠ€νŠΈ μ„œλΉ„μŠ€**: K-Culture, Blackpink, IVE, Twice λ“± + +## μ„œλΉ„μŠ€ 관리 (Service Management) + +### κ°œλ³„ μ„œλΉ„μŠ€ 관리 +```bash +# νŠΉμ • μ„œλΉ„μŠ€λ§Œ μ‹œμž‘ +docker-compose up -d [service-name] + +# νŠΉμ • μ„œλΉ„μŠ€ μž¬μ‹œμž‘ +docker-compose restart [service-name] + +# νŠΉμ • μ„œλΉ„μŠ€ 쀑지 +docker-compose stop [service-name] + +# μ„œλΉ„μŠ€ 둜그 확인 +docker-compose logs -f [service-name] + +# μ„œλΉ„μŠ€ μž¬λΉŒλ“œ +docker-compose build [service-name] +docker-compose up -d [service-name] +``` + +### λ°μ΄ν„°λ² μ΄μŠ€ 관리 +```bash +# MongoDB 접속 +docker exec -it site11_mongodb mongosh + +# λ°μ΄ν„°λ² μ΄μŠ€ λͺ©λ‘ +show dbs + +# AI Writer DB μ‚¬μš© +use ai_writer_db + +# μ»¬λ ‰μ…˜ 확인 +show collections + +# 기사 수 확인 +db.articles_ko.countDocuments() +``` + +### Redis 관리 +```bash +# Redis CLI 접속 +docker exec -it site11_redis redis-cli + +# λͺ¨λ“  ν‚€ 확인 +KEYS * + +# 큐 μƒνƒœ 확인 +LLEN queue:google_search +LLEN queue:ai_generation +LLEN queue:translation +``` + +## ν™˜κ²½ μ„€μ • (Configuration) + +### ν•„μˆ˜ API ν‚€ μ„€μ • +```bash +# .env 파일 νŽΈμ§‘ +vim .env + +# ν•„μˆ˜ μ„€μ • +DEEPL_API_KEY=your-deepl-api-key # λ²ˆμ—­ API +OPENAI_API_KEY=your-openai-api-key # 이미지 생성 +CLAUDE_API_KEY=your-claude-api-key # AI μ½˜ν…μΈ  생성 +SERP_API_KEY=your-serp-api-key # Google 검색 +``` + +### Pipeline μ„€μ • +```bash +# services/pipeline/.env +WORKER_COUNT=3 # μ›Œμ»€ ν”„λ‘œμ„ΈμŠ€ 수 +BATCH_SIZE=10 # 배치 처리 크기 +TRANSLATION_DELAY=1.0 # λ²ˆμ—­ 지연 μ‹œκ°„ +``` + +## 개발 κ°€μ΄λ“œ (Development Guide) + +### μƒˆ μ„œλΉ„μŠ€ μΆ”κ°€ +```bash +# μ„œλΉ„μŠ€ 디렉토리 ꡬ쑰 +services/ + └── new-service/ + β”œβ”€β”€ backend/ + β”‚ β”œβ”€β”€ Dockerfile + β”‚ β”œβ”€β”€ requirements.txt + β”‚ └── main.py + └── frontend/ + β”œβ”€β”€ Dockerfile + β”œβ”€β”€ package.json + └── src/ +``` + +### 둜컬 개발 ν™˜κ²½ +```bash +# λ°±μ—”λ“œ 개발 (Python) +cd services/[service-name]/backend +python -m venv venv +source venv/bin/activate +pip install -r requirements.txt +python main.py + +# ν”„λ‘ νŠΈμ—”λ“œ 개발 (React) +cd services/[service-name]/frontend +npm install +npm run dev +``` + +### ν…ŒμŠ€νŠΈ μ‹€ν–‰ +```bash +# 톡합 ν…ŒμŠ€νŠΈ +python test_integration.py + +# Pipeline ν…ŒμŠ€νŠΈ +python services/pipeline/test_pipeline.py + +# νŠΉμ • μ„œλΉ„μŠ€ ν…ŒμŠ€νŠΈ +docker exec -it site11_[service-name] pytest +``` + +## 문제 ν•΄κ²° (Troubleshooting) + +### μ„œλΉ„μŠ€κ°€ μ‹œμž‘λ˜μ§€ μ•Šμ„ λ•Œ +```bash +# 포트 좩돌 확인 +lsof -i :8000 + +# μ»¨ν…Œμ΄λ„ˆ μƒνƒœ 확인 +docker ps -a | grep site11 + +# 였λ₯˜ 둜그 확인 +docker-compose logs [service-name] | tail -100 +``` + +### MongoDB μ—°κ²° 문제 +```bash +# MongoDB μƒνƒœ 확인 +docker exec -it site11_mongodb mongosh --eval "db.serverStatus()" + +# μ—°κ²° ν…ŒμŠ€νŠΈ +docker run --rm -it --network site11_network mongo:7.0 mongosh mongodb://mongodb:27017 +``` + +### λ©”λͺ¨λ¦¬ λΆ€μ‘± 문제 +```bash +# Docker λ¦¬μ†ŒμŠ€ 정리 +docker system prune -a + +# λΆˆν•„μš”ν•œ 이미지 μ‚­μ œ +docker image prune -a + +# μ»¨ν…Œμ΄λ„ˆ 둜그 정리 +docker-compose logs --tail=0 --follow +``` + +### Pipeline μž‘μ—… λ§‰νž˜ +```bash +# Redis 큐 확인 +docker exec -it site11_redis redis-cli +> LLEN queue:translation +> LPOP queue:translation # λ§‰νžŒ μž‘μ—… 제거 + +# Pipeline μž¬μ‹œμž‘ +docker-compose restart pipeline-scheduler pipeline-translator +``` + +## Git μ €μž₯μ†Œ 관리 (Repository Management) + +### .gitignore μ„€μ • +``` +# λŒ€μš©λŸ‰ 파일 μ œμ™Έ +backups/ +*.archive +*.log +data/ + +# 이미지 파일 μ œμ™Έ +*.png +*.jpg +*.jpeg + +# Node modules μ œμ™Έ +node_modules/ +**/node_modules/ + +# ν™˜κ²½ 파일 +.env +*.env.local +``` + +### μ €μž₯μ†Œ 정리 +```bash +# Git νžˆμŠ€ν† λ¦¬ 정리 +git gc --aggressive --prune=now + +# λŒ€μš©λŸ‰ 파일 μ°ΎκΈ° +find . -size +100M -type f + +# μΊμ‹œ 정리 +git rm -r --cached [file/folder] +``` + +## λͺ¨λ‹ˆν„°λ§ (Monitoring) + +### Pipeline λͺ¨λ‹ˆν„°λ§ +- URL: http://localhost:8100 +- μ‹€μ‹œκ°„ 큐 μƒνƒœ 확인 +- 처리 톡계 확인 + +### 둜그 λͺ¨λ‹ˆν„°λ§ +```bash +# 전체 둜그 슀트림 +docker-compose logs -f + +# νŠΉμ • μ„œλΉ„μŠ€ 둜그 +docker-compose logs -f pipeline-scheduler + +# μ—λŸ¬λ§Œ 필터링 +docker-compose logs -f 2>&1 | grep ERROR +``` + +### μ„±λŠ₯ λͺ¨λ‹ˆν„°λ§ +```bash +# μ»¨ν…Œμ΄λ„ˆ λ¦¬μ†ŒμŠ€ μ‚¬μš©λŸ‰ +docker stats + +# MongoDB μ„±λŠ₯ +docker exec -it site11_mongodb mongosh --eval "db.serverStatus().opcounters" +``` + +## λ°±μ—… 및 볡ꡬ (Backup & Recovery) + +### MongoDB λ°±μ—… +```bash +# λ°±μ—… 슀크립트 μ‹€ν–‰ +./scripts/backup-mongodb.sh + +# μˆ˜λ™ λ°±μ—… +docker exec -it site11_mongodb mongodump --archive=/data/backup.archive +docker cp site11_mongodb:/data/backup.archive ./backups/ +``` + +### 전체 μ‹œμŠ€ν…œ λ°±μ—… +```bash +# 데이터 디렉토리 λ°±μ—… +tar -czf backup-$(date +%Y%m%d).tar.gz data/ + +# μ„€μ • λ°±μ—… +tar -czf config-backup.tar.gz .env docker-compose.yml services/ +``` + +## 문의 및 지원 (Support) + +- **이슈 트래컀**: http://gitea.yakenator.io/aimond/site11/issues +- **λ¬Έμ„œ**: `/docs` 디렉토리 μ°Έμ‘° + - `PLAN.md`: κ΅¬ν˜„ κ³„νš + - `PROGRESS.md`: 진행 상황 + - `ARCHITECTURE.md`: μ‹œμŠ€ν…œ ꡬ쑰 + - `PIPELINE_SCHEDULER_GUIDE.md`: Pipeline κ°€μ΄λ“œ + +## λΌμ΄μ„ μŠ€ (License) + +Copyright (c) 2024 Site11 Project. All rights reserved. \ No newline at end of file