aimond/site11
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Add hybrid deployment with Docker and Kubernetes

Browse Source 
- Docker Compose for infrastructure (MongoDB, Redis, Kafka, Zookeeper)
- Docker for central control (Scheduler, Monitor, Language Sync)
- K8s for scalable workers (RSS, Google Search, Translator, AI Generator, Image Generator)
- Automatic scaling with HPA (Horizontal Pod Autoscaler)
- Comprehensive deployment scripts and documentation
- Updated README with hybrid deployment guide
This commit is contained in:
jungwoo choi
2025-09-28 21:03:14 +09:00
parent 1ff9afe6f4
commit 46b5135f45
10 changed files with 872 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

147
README.md
View File

@ -458,6 +458,153 @@ tar -czf config-backup.tar.gz .env docker-compose.yml services/
- `ARCHITECTURE.md`: 시스템 구조
- `PIPELINE_SCHEDULER_GUIDE.md`: Pipeline 가이드
## 하이브리드 배포 (Hybrid Deployment with K8s)
Site11은 Docker Compose와 Kubernetes를 함께 사용하는 하이브리드 배포를 지원합니다.
### 배포 아키텍처
#### Docker Compose (인프라 및 중앙 제어)
- **인프라**: MongoDB, Redis, Kafka, Zookeeper
- **중앙 제어**: Pipeline Scheduler, Pipeline Monitor, Language Sync
- **관리 콘솔**: Console Backend/Frontend
#### Kubernetes (무상태 워커)
- **데이터 수집**: RSS Collector, Google Search
- **처리 워커**: Translator, AI Article Generator, Image Generator
- **자동 스케일링**: HPA(Horizontal Pod Autoscaler) 적용
### 하이브리드 배포 시작
#### 1. Docker 인프라 시작
```bash
# 하이브리드 용 docker-compose 사용
docker-compose -f docker-compose-hybrid.yml up -d
# 상태 확인
docker-compose -f docker-compose-hybrid.yml ps
# 로그 확인
docker-compose -f docker-compose-hybrid.yml logs -f pipeline-scheduler
```
#### 2. K8s 워커 배포
```bash
# K8s 매니페스트 디렉토리로 이동
cd k8s/pipeline
# API 키 설정 (configmap.yaml 편집)
vim configmap.yaml
# 배포 실행
./deploy.sh
# 배포 상태 확인
kubectl -n site11-pipeline get pods
kubectl -n site11-pipeline get hpa
```
### K8s 자동 스케일링 설정
#### HPA (Horizontal Pod Autoscaler) 구성
| 서비스 | 최소 Pod | 최대 Pod | CPU 임계값 | 메모리 임계값 |
|--------|---------|---------|------------|-------------|
| RSS Collector | 1 | 5 | 70% | 80% |
| Google Search | 1 | 5 | 70% | 80% |
| Translator | 2 | 10 | 70% | 80% |
| AI Generator | 1 | 8 | 70% | 80% |
| Image Generator | 1 | 6 | 70% | 80% |
#### 스케일링 모니터링
```bash
# HPA 상태 실시간 모니터링
kubectl -n site11-pipeline get hpa -w
# Pod 수 확인
kubectl -n site11-pipeline get pods -o wide
# 메트릭 상세 정보
kubectl -n site11-pipeline describe hpa pipeline-translator-hpa
# 수동 스케일링 (필요시)
kubectl -n site11-pipeline scale deployment pipeline-translator --replicas=5
```
### K8s 워커 관리
#### 로그 확인
```bash
# 특정 디플로이먼트 로그
kubectl -n site11-pipeline logs -f deployment/pipeline-translator
# 모든 Pod 로그
kubectl -n site11-pipeline logs -f -l component=processor
# 에러만 필터링
kubectl -n site11-pipeline logs -f deployment/pipeline-translator | grep ERROR
```
#### 업데이트 및 롤백
```bash
# 이미지 업데이트
kubectl -n site11-pipeline set image deployment/pipeline-translator \
translator=site11/pipeline-translator:v2
# 롤아웃 상태 확인
kubectl -n site11-pipeline rollout status deployment/pipeline-translator
# 롤백 (필요시)
kubectl -n site11-pipeline rollout undo deployment/pipeline-translator
```
#### 리소스 모니터링
```bash
# Pod 리소스 사용량
kubectl -n site11-pipeline top pods
# Node 리소스 사용량
kubectl top nodes
# 상세 리소스 정보
kubectl -n site11-pipeline describe pod <pod-name>
```
### 문제 해결 (K8s)
#### Pod가 시작되지 않을 때
```bash
# Pod 상태 확인
kubectl -n site11-pipeline describe pod <pod-name>
# 이벤트 확인
kubectl -n site11-pipeline get events --sort-by='.lastTimestamp'
# 이미지 풀 문제 확인
kubectl -n site11-pipeline get pods -o jsonpath='{.items[*].spec.containers[*].image}'
```
#### Redis/MongoDB 연결 실패
```bash
# Docker 호스트 네트워크 확인
# K8s Pod는 host.docker.internal 사용
kubectl -n site11-pipeline exec -it <pod-name> -- ping host.docker.internal
# 연결 테스트
kubectl -n site11-pipeline exec -it <pod-name> -- redis-cli -h host.docker.internal ping
```
#### 전체 정리
```bash
# K8s 리소스 삭제
kubectl delete namespace site11-pipeline
# Docker 정리
docker-compose -f docker-compose-hybrid.yml down
# 볼륨 정리 (주의!)
docker-compose -f docker-compose-hybrid.yml down -v
```
## 라이선스 (License)
Copyright (c) 2024 Site11 Project. All rights reserved.

194
docker-compose-hybrid.yml Normal file
View File

@ -0,0 +1,194 @@
# Docker Compose for Hybrid Deployment (with K8s)
# ================================================
# 이 파일은 K8s와 함께 사용하는 하이브리드 배포용입니다.
# Pipeline 워커들은 K8s로 이동하고, 인프라와 중앙 제어만 Docker에 유지합니다.
version: '3.8'
services:
# ============ Infrastructure Services ============
mongodb:
image: mongo:7.0
container_name: ${COMPOSE_PROJECT_NAME}_mongodb
ports:
- "27017:27017"
volumes:
- ./data/mongodb:/data/db
- ./data/mongodb-config:/data/configdb
environment:
- MONGO_INITDB_DATABASE=site11_db
networks:
- site11_network
restart: unless-stopped
redis:
image: redis:7-alpine
container_name: ${COMPOSE_PROJECT_NAME}_redis
ports:
- "6379:6379"
volumes:
- ./data/redis:/data
command: redis-server --appendonly yes
networks:
- site11_network
restart: unless-stopped
zookeeper:
image: confluentinc/cp-zookeeper:7.5.0
container_name: ${COMPOSE_PROJECT_NAME}_zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
volumes:
- ./data/zookeeper/data:/var/lib/zookeeper/data
- ./data/zookeeper/logs:/var/lib/zookeeper/logs
networks:
- site11_network
restart: unless-stopped
kafka:
image: confluentinc/cp-kafka:7.5.0
container_name: ${COMPOSE_PROJECT_NAME}_kafka
depends_on:
- zookeeper
ports:
- "9092:9092"
- "9101:9101"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_JMX_PORT: 9101
KAFKA_JMX_HOSTNAME: localhost
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
volumes:
- ./data/kafka:/var/lib/kafka/data
networks:
- site11_network
restart: unless-stopped
# ============ Central Control & Monitoring ============
# Pipeline Scheduler - 중앙 제어 (Docker에 유지)
pipeline-scheduler:
build:
context: ./services/pipeline
dockerfile: scheduler/Dockerfile
container_name: ${COMPOSE_PROJECT_NAME}_pipeline_scheduler
restart: unless-stopped
depends_on:
- redis
- mongodb
environment:
- REDIS_URL=redis://redis:6379
- MONGODB_URL=mongodb://mongodb:27017
- DB_NAME=ai_writer_db
- LOG_LEVEL=INFO
- SCHEDULER_INTERVAL=60
volumes:
- ./services/pipeline/shared:/app/shared:ro
- ./services/pipeline/config:/app/config:ro
networks:
- site11_network
# Pipeline Monitor - 모니터링 대시보드 (Docker에 유지)
pipeline-monitor:
build:
context: ./services/pipeline
dockerfile: monitor/Dockerfile
container_name: ${COMPOSE_PROJECT_NAME}_pipeline_monitor
restart: unless-stopped
depends_on:
- redis
- mongodb
ports:
- "8100:8000"
environment:
- REDIS_URL=redis://redis:6379
- MONGODB_URL=mongodb://mongodb:27017
- DB_NAME=ai_writer_db
- LOG_LEVEL=INFO
volumes:
- ./services/pipeline/shared:/app/shared:ro
networks:
- site11_network
# Pipeline Language Sync - 언어 동기화 (Docker에 유지)
pipeline-language-sync:
build:
context: ./services/pipeline
dockerfile: translator/Dockerfile
container_name: ${COMPOSE_PROJECT_NAME}_pipeline_language_sync
restart: unless-stopped
depends_on:
- redis
- mongodb
env_file:
- ./services/pipeline/.env
environment:
- REDIS_URL=redis://redis:6379
- MONGODB_URL=mongodb://mongodb:27017
- DB_NAME=ai_writer_db
- LOG_LEVEL=INFO
- MODE=sync # sync 모드로 실행
volumes:
- ./services/pipeline/shared:/app/shared:ro
networks:
- site11_network
command: ["python", "language_sync.py"]
# ============ Other Essential Services ============
console-backend:
build:
context: ./console/backend
dockerfile: Dockerfile
container_name: ${COMPOSE_PROJECT_NAME}_console_backend
ports:
- "8011:8000"
environment:
- ENV=${ENV}
- MONGODB_URL=mongodb://mongodb:27017
- REDIS_URL=redis://redis:6379
- USERS_SERVICE_URL=http://users-backend:8000
depends_on:
- mongodb
- redis
networks:
- site11_network
restart: unless-stopped
console-frontend:
build:
context: ./console/frontend
dockerfile: Dockerfile
container_name: ${COMPOSE_PROJECT_NAME}_console_frontend
ports:
- "3000:80"
environment:
- VITE_API_URL=http://localhost:8011
networks:
- site11_network
restart: unless-stopped
networks:
site11_network:
driver: bridge
name: site11_network
# ============ K8s로 이동한 서비스들 ============
# 다음 서비스들은 K8s에서 실행됩니다:
# - pipeline-rss-collector
# - pipeline-google-search
# - pipeline-translator
# - pipeline-ai-article-generator
# - pipeline-image-generator
#
# K8s 배포 방법:
# cd k8s/pipeline
# ./deploy.sh

78
k8s/pipeline/ai-article-generator.yaml Normal file
View File

@ -0,0 +1,78 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: pipeline-ai-article-generator
namespace: site11-pipeline
labels:
app: pipeline-ai-article-generator
component: processor
spec:
replicas: 2
selector:
matchLabels:
app: pipeline-ai-article-generator
template:
metadata:
labels:
app: pipeline-ai-article-generator
component: processor
spec:
containers:
- name: ai-article-generator
image: site11/pipeline-ai-article-generator:latest
imagePullPolicy: Always
envFrom:
- configMapRef:
name: pipeline-config
- secretRef:
name: pipeline-secrets
resources:
requests:
memory: "512Mi"
cpu: "200m"
limits:
memory: "1Gi"
cpu: "1000m"
livenessProbe:
exec:
command:
- python
- -c
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
initialDelaySeconds: 30
periodSeconds: 30
readinessProbe:
exec:
command:
- python
- -c
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
initialDelaySeconds: 10
periodSeconds: 10
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: pipeline-ai-article-generator-hpa
namespace: site11-pipeline
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: pipeline-ai-article-generator
minReplicas: 1
maxReplicas: 8
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80

38
k8s/pipeline/configmap.yaml Normal file
View File

@ -0,0 +1,38 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: pipeline-config
namespace: site11-pipeline
data:
# Redis 연결 (Docker 호스트)
REDIS_URL: "redis://host.docker.internal:6379"
# MongoDB 연결 (Docker 호스트)
MONGODB_URL: "mongodb://host.docker.internal:27017"
DB_NAME: "ai_writer_db"
# 로깅
LOG_LEVEL: "INFO"
# 워커 설정
WORKER_COUNT: "2"
BATCH_SIZE: "10"
# 큐 설정
RSS_ENQUEUE_DELAY: "1.0"
GOOGLE_SEARCH_DELAY: "2.0"
TRANSLATION_DELAY: "1.0"
---
apiVersion: v1
kind: Secret
metadata:
name: pipeline-secrets
namespace: site11-pipeline
type: Opaque
stringData:
# API Keys (실제 값)
DEEPL_API_KEY: "3abbc796-2515-44a8-972d-22dcf27ab54a"
OPENAI_API_KEY: "sk-openai-api-key-here" # OpenAI API 키 필요
CLAUDE_API_KEY: "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"
SERP_API_KEY: "serp-api-key-here" # SERP API 키 필요

96
k8s/pipeline/deploy.sh Executable file
View File

@ -0,0 +1,96 @@
#!/bin/bash
# Site11 Pipeline K8s Deployment Script
# ======================================
set -e
echo "🚀 Site11 Pipeline K8s Deployment"
echo "================================="
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
# Check if kubectl is available
if ! command -v kubectl &> /dev/null; then
echo -e "${RED}❌ kubectl is not installed${NC}"
exit 1
fi
# Check K8s cluster connection
echo -n "Checking K8s cluster connection... "
if kubectl cluster-info &> /dev/null; then
echo -e "${GREEN}✓${NC}"
else
echo -e "${RED}✗ Cannot connect to K8s cluster${NC}"
exit 1
fi
# Step 1: Create namespace
echo ""
echo "1. Creating namespace..."
kubectl apply -f namespace.yaml
# Step 2: Create ConfigMap and Secrets
echo ""
echo "2. Creating ConfigMap and Secrets..."
echo -e "${YELLOW}⚠️ Remember to update API keys in configmap.yaml${NC}"
kubectl apply -f configmap.yaml
# Step 3: Build and push Docker images
echo ""
echo "3. Building Docker images..."
echo -e "${YELLOW}Note: This script assumes images are already built${NC}"
echo "To build images, run from project root:"
echo " docker-compose build pipeline-rss-collector"
echo " docker-compose build pipeline-google-search"
echo " docker-compose build pipeline-translator"
echo " docker-compose build pipeline-ai-article-generator"
echo " docker-compose build pipeline-image-generator"
# Step 4: Tag and push images (if using local registry)
echo ""
echo "4. Tagging images for K8s..."
services=("rss-collector" "google-search" "translator" "ai-article-generator" "image-generator")
for service in "${services[@]}"; do
echo "Tagging pipeline-$service..."
docker tag site11_pipeline-$service:latest site11/pipeline-$service:latest
done
# Step 5: Deploy services
echo ""
echo "5. Deploying services to K8s..."
for service in "${services[@]}"; do
echo "Deploying $service..."
kubectl apply -f $service.yaml
done
# Step 6: Check deployment status
echo ""
echo "6. Checking deployment status..."
kubectl -n site11-pipeline get deployments
echo ""
echo "7. Waiting for pods to be ready..."
kubectl -n site11-pipeline wait --for=condition=Ready pods --all --timeout=300s || true
# Step 7: Show final status
echo ""
echo "✅ Deployment Complete!"
echo ""
echo "Current status:"
kubectl -n site11-pipeline get pods
echo ""
echo "To view logs:"
echo " kubectl -n site11-pipeline logs -f deployment/pipeline-translator"
echo ""
echo "To scale deployments:"
echo " kubectl -n site11-pipeline scale deployment pipeline-translator --replicas=5"
echo ""
echo "To delete all resources:"
echo " kubectl delete namespace site11-pipeline"

78
k8s/pipeline/google-search.yaml Normal file
View File

@ -0,0 +1,78 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: pipeline-google-search
namespace: site11-pipeline
labels:
app: pipeline-google-search
component: data-collector
spec:
replicas: 2
selector:
matchLabels:
app: pipeline-google-search
template:
metadata:
labels:
app: pipeline-google-search
component: data-collector
spec:
containers:
- name: google-search
image: site11/pipeline-google-search:latest
imagePullPolicy: Always
envFrom:
- configMapRef:
name: pipeline-config
- secretRef:
name: pipeline-secrets
resources:
requests:
memory: "256Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "500m"
livenessProbe:
exec:
command:
- python
- -c
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
initialDelaySeconds: 30
periodSeconds: 30
readinessProbe:
exec:
command:
- python
- -c
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
initialDelaySeconds: 10
periodSeconds: 10
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: pipeline-google-search-hpa
namespace: site11-pipeline
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: pipeline-google-search
minReplicas: 1
maxReplicas: 5
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80

78
k8s/pipeline/image-generator.yaml Normal file
View File

@ -0,0 +1,78 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: pipeline-image-generator
namespace: site11-pipeline
labels:
app: pipeline-image-generator
component: processor
spec:
replicas: 2
selector:
matchLabels:
app: pipeline-image-generator
template:
metadata:
labels:
app: pipeline-image-generator
component: processor
spec:
containers:
- name: image-generator
image: site11/pipeline-image-generator:latest
imagePullPolicy: Always
envFrom:
- configMapRef:
name: pipeline-config
- secretRef:
name: pipeline-secrets
resources:
requests:
memory: "512Mi"
cpu: "200m"
limits:
memory: "1Gi"
cpu: "1000m"
livenessProbe:
exec:
command:
- python
- -c
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
initialDelaySeconds: 30
periodSeconds: 30
readinessProbe:
exec:
command:
- python
- -c
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
initialDelaySeconds: 10
periodSeconds: 10
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: pipeline-image-generator-hpa
namespace: site11-pipeline
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: pipeline-image-generator
minReplicas: 1
maxReplicas: 6
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80

7
k8s/pipeline/namespace.yaml Normal file
View File

@ -0,0 +1,7 @@
apiVersion: v1
kind: Namespace
metadata:
name: site11-pipeline
labels:
name: site11-pipeline
environment: production

78
k8s/pipeline/rss-collector.yaml Normal file
View File

@ -0,0 +1,78 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: pipeline-rss-collector
namespace: site11-pipeline
labels:
app: pipeline-rss-collector
component: data-collector
spec:
replicas: 2
selector:
matchLabels:
app: pipeline-rss-collector
template:
metadata:
labels:
app: pipeline-rss-collector
component: data-collector
spec:
containers:
- name: rss-collector
image: site11/pipeline-rss-collector:latest
imagePullPolicy: Always
envFrom:
- configMapRef:
name: pipeline-config
- secretRef:
name: pipeline-secrets
resources:
requests:
memory: "256Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "500m"
livenessProbe:
exec:
command:
- python
- -c
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
initialDelaySeconds: 30
periodSeconds: 30
readinessProbe:
exec:
command:
- python
- -c
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
initialDelaySeconds: 10
periodSeconds: 10
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: pipeline-rss-collector-hpa
namespace: site11-pipeline
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: pipeline-rss-collector
minReplicas: 1
maxReplicas: 5
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80

78
k8s/pipeline/translator.yaml Normal file
View File

@ -0,0 +1,78 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: pipeline-translator
namespace: site11-pipeline
labels:
app: pipeline-translator
component: processor
spec:
replicas: 3
selector:
matchLabels:
app: pipeline-translator
template:
metadata:
labels:
app: pipeline-translator
component: processor
spec:
containers:
- name: translator
image: site11/pipeline-translator:latest
imagePullPolicy: Always
envFrom:
- configMapRef:
name: pipeline-config
- secretRef:
name: pipeline-secrets
resources:
requests:
memory: "512Mi"
cpu: "200m"
limits:
memory: "1Gi"
cpu: "1000m"
livenessProbe:
exec:
command:
- python
- -c
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
initialDelaySeconds: 30
periodSeconds: 30
readinessProbe:
exec:
command:
- python
- -c
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
initialDelaySeconds: 10
periodSeconds: 10
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: pipeline-translator-hpa
namespace: site11-pipeline
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: pipeline-translator
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80