feat: Add hybrid deployment with Docker and Kubernetes

- Docker Compose for infrastructure (MongoDB, Redis, Kafka, Zookeeper)
- Docker for central control (Scheduler, Monitor, Language Sync)
- K8s for scalable workers (RSS, Google Search, Translator, AI Generator, Image Generator)
- Automatic scaling with HPA (Horizontal Pod Autoscaler)
- Comprehensive deployment scripts and documentation
- Updated README with hybrid deployment guide

README.md

@@ -458,6 +458,153 @@ tar -czf config-backup.tar.gz .env docker-compose.yml services/
   - `ARCHITECTURE.md`: 시스템 구조
   - `PIPELINE_SCHEDULER_GUIDE.md`: Pipeline 가이드
 
+## 하이브리드 배포 (Hybrid Deployment with K8s)
+
+Site11은 Docker Compose와 Kubernetes를 함께 사용하는 하이브리드 배포를 지원합니다.
+
+### 배포 아키텍처
+
+#### Docker Compose (인프라 및 중앙 제어)
+- **인프라**: MongoDB, Redis, Kafka, Zookeeper
+- **중앙 제어**: Pipeline Scheduler, Pipeline Monitor, Language Sync
+- **관리 콘솔**: Console Backend/Frontend
+
+#### Kubernetes (무상태 워커)
+- **데이터 수집**: RSS Collector, Google Search
+- **처리 워커**: Translator, AI Article Generator, Image Generator
+- **자동 스케일링**: HPA(Horizontal Pod Autoscaler) 적용
+
+### 하이브리드 배포 시작
+
+#### 1. Docker 인프라 시작
+```bash
+# 하이브리드 용 docker-compose 사용
+docker-compose -f docker-compose-hybrid.yml up -d
+
+# 상태 확인
+docker-compose -f docker-compose-hybrid.yml ps
+
+# 로그 확인
+docker-compose -f docker-compose-hybrid.yml logs -f pipeline-scheduler
+```
+
+#### 2. K8s 워커 배포
+```bash
+# K8s 매니페스트 디렉토리로 이동
+cd k8s/pipeline
+
+# API 키 설정 (configmap.yaml 편집)
+vim configmap.yaml
+
+# 배포 실행
+./deploy.sh
+
+# 배포 상태 확인
+kubectl -n site11-pipeline get pods
+kubectl -n site11-pipeline get hpa
+```
+
+### K8s 자동 스케일링 설정
+
+#### HPA (Horizontal Pod Autoscaler) 구성
+| 서비스 | 최소 Pod | 최대 Pod | CPU 임계값 | 메모리 임계값 |
+|--------|---------|---------|------------|-------------|
+| RSS Collector | 1 | 5 | 70% | 80% |
+| Google Search | 1 | 5 | 70% | 80% |
+| Translator | 2 | 10 | 70% | 80% |
+| AI Generator | 1 | 8 | 70% | 80% |
+| Image Generator | 1 | 6 | 70% | 80% |
+
+#### 스케일링 모니터링
+```bash
+# HPA 상태 실시간 모니터링
+kubectl -n site11-pipeline get hpa -w
+
+# Pod 수 확인
+kubectl -n site11-pipeline get pods -o wide
+
+# 메트릭 상세 정보
+kubectl -n site11-pipeline describe hpa pipeline-translator-hpa
+
+# 수동 스케일링 (필요시)
+kubectl -n site11-pipeline scale deployment pipeline-translator --replicas=5
+```
+
+### K8s 워커 관리
+
+#### 로그 확인
+```bash
+# 특정 디플로이먼트 로그
+kubectl -n site11-pipeline logs -f deployment/pipeline-translator
+
+# 모든 Pod 로그
+kubectl -n site11-pipeline logs -f -l component=processor
+
+# 에러만 필터링
+kubectl -n site11-pipeline logs -f deployment/pipeline-translator | grep ERROR
+```
+
+#### 업데이트 및 롤백
+```bash
+# 이미지 업데이트
+kubectl -n site11-pipeline set image deployment/pipeline-translator \
+  translator=site11/pipeline-translator:v2
+
+# 롤아웃 상태 확인
+kubectl -n site11-pipeline rollout status deployment/pipeline-translator
+
+# 롤백 (필요시)
+kubectl -n site11-pipeline rollout undo deployment/pipeline-translator
+```
+
+#### 리소스 모니터링
+```bash
+# Pod 리소스 사용량
+kubectl -n site11-pipeline top pods
+
+# Node 리소스 사용량
+kubectl top nodes
+
+# 상세 리소스 정보
+kubectl -n site11-pipeline describe pod <pod-name>
+```
+
+### 문제 해결 (K8s)
+
+#### Pod가 시작되지 않을 때
+```bash
+# Pod 상태 확인
+kubectl -n site11-pipeline describe pod <pod-name>
+
+# 이벤트 확인
+kubectl -n site11-pipeline get events --sort-by='.lastTimestamp'
+
+# 이미지 풀 문제 확인
+kubectl -n site11-pipeline get pods -o jsonpath='{.items[*].spec.containers[*].image}'
+```
+
+#### Redis/MongoDB 연결 실패
+```bash
+# Docker 호스트 네트워크 확인
+# K8s Pod는 host.docker.internal 사용
+kubectl -n site11-pipeline exec -it <pod-name> -- ping host.docker.internal
+
+# 연결 테스트
+kubectl -n site11-pipeline exec -it <pod-name> -- redis-cli -h host.docker.internal ping
+```
+
+#### 전체 정리
+```bash
+# K8s 리소스 삭제
+kubectl delete namespace site11-pipeline
+
+# Docker 정리
+docker-compose -f docker-compose-hybrid.yml down
+
+# 볼륨 정리 (주의!)
+docker-compose -f docker-compose-hybrid.yml down -v
+```
+
 ## 라이선스 (License)
 
 Copyright (c) 2024 Site11 Project. All rights reserved.

docker-compose-hybrid.yml

@@ -0,0 +1,194 @@
+# Docker Compose for Hybrid Deployment (with K8s)
+# ================================================
+# 이 파일은 K8s와 함께 사용하는 하이브리드 배포용입니다.
+# Pipeline 워커들은 K8s로 이동하고, 인프라와 중앙 제어만 Docker에 유지합니다.
+
+version: '3.8'
+
+services:
+  # ============ Infrastructure Services ============
+  mongodb:
+    image: mongo:7.0
+    container_name: ${COMPOSE_PROJECT_NAME}_mongodb
+    ports:
+      - "27017:27017"
+    volumes:
+      - ./data/mongodb:/data/db
+      - ./data/mongodb-config:/data/configdb
+    environment:
+      - MONGO_INITDB_DATABASE=site11_db
+    networks:
+      - site11_network
+    restart: unless-stopped
+
+  redis:
+    image: redis:7-alpine
+    container_name: ${COMPOSE_PROJECT_NAME}_redis
+    ports:
+      - "6379:6379"
+    volumes:
+      - ./data/redis:/data
+    command: redis-server --appendonly yes
+    networks:
+      - site11_network
+    restart: unless-stopped
+
+  zookeeper:
+    image: confluentinc/cp-zookeeper:7.5.0
+    container_name: ${COMPOSE_PROJECT_NAME}_zookeeper
+    environment:
+      ZOOKEEPER_CLIENT_PORT: 2181
+      ZOOKEEPER_TICK_TIME: 2000
+    volumes:
+      - ./data/zookeeper/data:/var/lib/zookeeper/data
+      - ./data/zookeeper/logs:/var/lib/zookeeper/logs
+    networks:
+      - site11_network
+    restart: unless-stopped
+
+  kafka:
+    image: confluentinc/cp-kafka:7.5.0
+    container_name: ${COMPOSE_PROJECT_NAME}_kafka
+    depends_on:
+      - zookeeper
+    ports:
+      - "9092:9092"
+      - "9101:9101"
+    environment:
+      KAFKA_BROKER_ID: 1
+      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
+      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
+      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
+      KAFKA_JMX_PORT: 9101
+      KAFKA_JMX_HOSTNAME: localhost
+      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
+    volumes:
+      - ./data/kafka:/var/lib/kafka/data
+    networks:
+      - site11_network
+    restart: unless-stopped
+
+  # ============ Central Control & Monitoring ============
+
+  # Pipeline Scheduler - 중앙 제어 (Docker에 유지)
+  pipeline-scheduler:
+    build:
+      context: ./services/pipeline
+      dockerfile: scheduler/Dockerfile
+    container_name: ${COMPOSE_PROJECT_NAME}_pipeline_scheduler
+    restart: unless-stopped
+    depends_on:
+      - redis
+      - mongodb
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - MONGODB_URL=mongodb://mongodb:27017
+      - DB_NAME=ai_writer_db
+      - LOG_LEVEL=INFO
+      - SCHEDULER_INTERVAL=60
+    volumes:
+      - ./services/pipeline/shared:/app/shared:ro
+      - ./services/pipeline/config:/app/config:ro
+    networks:
+      - site11_network
+
+  # Pipeline Monitor - 모니터링 대시보드 (Docker에 유지)
+  pipeline-monitor:
+    build:
+      context: ./services/pipeline
+      dockerfile: monitor/Dockerfile
+    container_name: ${COMPOSE_PROJECT_NAME}_pipeline_monitor
+    restart: unless-stopped
+    depends_on:
+      - redis
+      - mongodb
+    ports:
+      - "8100:8000"
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - MONGODB_URL=mongodb://mongodb:27017
+      - DB_NAME=ai_writer_db
+      - LOG_LEVEL=INFO
+    volumes:
+      - ./services/pipeline/shared:/app/shared:ro
+    networks:
+      - site11_network
+
+  # Pipeline Language Sync - 언어 동기화 (Docker에 유지)
+  pipeline-language-sync:
+    build:
+      context: ./services/pipeline
+      dockerfile: translator/Dockerfile
+    container_name: ${COMPOSE_PROJECT_NAME}_pipeline_language_sync
+    restart: unless-stopped
+    depends_on:
+      - redis
+      - mongodb
+    env_file:
+      - ./services/pipeline/.env
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - MONGODB_URL=mongodb://mongodb:27017
+      - DB_NAME=ai_writer_db
+      - LOG_LEVEL=INFO
+      - MODE=sync  # sync 모드로 실행
+    volumes:
+      - ./services/pipeline/shared:/app/shared:ro
+    networks:
+      - site11_network
+    command: ["python", "language_sync.py"]
+
+  # ============ Other Essential Services ============
+
+  console-backend:
+    build:
+      context: ./console/backend
+      dockerfile: Dockerfile
+    container_name: ${COMPOSE_PROJECT_NAME}_console_backend
+    ports:
+      - "8011:8000"
+    environment:
+      - ENV=${ENV}
+      - MONGODB_URL=mongodb://mongodb:27017
+      - REDIS_URL=redis://redis:6379
+      - USERS_SERVICE_URL=http://users-backend:8000
+    depends_on:
+      - mongodb
+      - redis
+    networks:
+      - site11_network
+    restart: unless-stopped
+
+  console-frontend:
+    build:
+      context: ./console/frontend
+      dockerfile: Dockerfile
+    container_name: ${COMPOSE_PROJECT_NAME}_console_frontend
+    ports:
+      - "3000:80"
+    environment:
+      - VITE_API_URL=http://localhost:8011
+    networks:
+      - site11_network
+    restart: unless-stopped
+
+networks:
+  site11_network:
+    driver: bridge
+    name: site11_network
+
+# ============ K8s로 이동한 서비스들 ============
+# 다음 서비스들은 K8s에서 실행됩니다:
+# - pipeline-rss-collector
+# - pipeline-google-search
+# - pipeline-translator
+# - pipeline-ai-article-generator
+# - pipeline-image-generator
+#
+# K8s 배포 방법:
+# cd k8s/pipeline
+# ./deploy.sh

k8s/pipeline/ai-article-generator.yaml

@@ -0,0 +1,78 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: pipeline-ai-article-generator
+  namespace: site11-pipeline
+  labels:
+    app: pipeline-ai-article-generator
+    component: processor
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: pipeline-ai-article-generator
+  template:
+    metadata:
+      labels:
+        app: pipeline-ai-article-generator
+        component: processor
+    spec:
+      containers:
+      - name: ai-article-generator
+        image: site11/pipeline-ai-article-generator:latest
+        imagePullPolicy: Always
+        envFrom:
+        - configMapRef:
+            name: pipeline-config
+        - secretRef:
+            name: pipeline-secrets
+        resources:
+          requests:
+            memory: "512Mi"
+            cpu: "200m"
+          limits:
+            memory: "1Gi"
+            cpu: "1000m"
+        livenessProbe:
+          exec:
+            command:
+            - python
+            - -c
+            - "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
+          initialDelaySeconds: 30
+          periodSeconds: 30
+        readinessProbe:
+          exec:
+            command:
+            - python
+            - -c
+            - "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
+          initialDelaySeconds: 10
+          periodSeconds: 10
+
+---
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: pipeline-ai-article-generator-hpa
+  namespace: site11-pipeline
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: pipeline-ai-article-generator
+  minReplicas: 1
+  maxReplicas: 8
+  metrics:
+  - type: Resource
+    resource:
+      name: cpu
+      target:
+        type: Utilization
+        averageUtilization: 70
+  - type: Resource
+    resource:
+      name: memory
+      target:
+        type: Utilization
+        averageUtilization: 80

k8s/pipeline/configmap.yaml

@@ -0,0 +1,38 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: pipeline-config
+  namespace: site11-pipeline
+data:
+  # Redis 연결 (Docker 호스트)
+  REDIS_URL: "redis://host.docker.internal:6379"
+
+  # MongoDB 연결 (Docker 호스트)
+  MONGODB_URL: "mongodb://host.docker.internal:27017"
+  DB_NAME: "ai_writer_db"
+
+  # 로깅
+  LOG_LEVEL: "INFO"
+
+  # 워커 설정
+  WORKER_COUNT: "2"
+  BATCH_SIZE: "10"
+
+  # 큐 설정
+  RSS_ENQUEUE_DELAY: "1.0"
+  GOOGLE_SEARCH_DELAY: "2.0"
+  TRANSLATION_DELAY: "1.0"
+
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: pipeline-secrets
+  namespace: site11-pipeline
+type: Opaque
+stringData:
+  # API Keys (실제 값)
+  DEEPL_API_KEY: "3abbc796-2515-44a8-972d-22dcf27ab54a"
+  OPENAI_API_KEY: "sk-openai-api-key-here"  # OpenAI API 키 필요
+  CLAUDE_API_KEY: "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"
+  SERP_API_KEY: "serp-api-key-here"  # SERP API 키 필요

k8s/pipeline/deploy.sh

@@ -0,0 +1,96 @@
+#!/bin/bash
+
+# Site11 Pipeline K8s Deployment Script
+# ======================================
+
+set -e
+
+echo "🚀 Site11 Pipeline K8s Deployment"
+echo "================================="
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m' # No Color
+
+# Check if kubectl is available
+if ! command -v kubectl &> /dev/null; then
+    echo -e "${RED}❌ kubectl is not installed${NC}"
+    exit 1
+fi
+
+# Check K8s cluster connection
+echo -n "Checking K8s cluster connection... "
+if kubectl cluster-info &> /dev/null; then
+    echo -e "${GREEN}✓${NC}"
+else
+    echo -e "${RED}✗ Cannot connect to K8s cluster${NC}"
+    exit 1
+fi
+
+# Step 1: Create namespace
+echo ""
+echo "1. Creating namespace..."
+kubectl apply -f namespace.yaml
+
+# Step 2: Create ConfigMap and Secrets
+echo ""
+echo "2. Creating ConfigMap and Secrets..."
+echo -e "${YELLOW}⚠️  Remember to update API keys in configmap.yaml${NC}"
+kubectl apply -f configmap.yaml
+
+# Step 3: Build and push Docker images
+echo ""
+echo "3. Building Docker images..."
+echo -e "${YELLOW}Note: This script assumes images are already built${NC}"
+echo "To build images, run from project root:"
+echo "  docker-compose build pipeline-rss-collector"
+echo "  docker-compose build pipeline-google-search"
+echo "  docker-compose build pipeline-translator"
+echo "  docker-compose build pipeline-ai-article-generator"
+echo "  docker-compose build pipeline-image-generator"
+
+# Step 4: Tag and push images (if using local registry)
+echo ""
+echo "4. Tagging images for K8s..."
+services=("rss-collector" "google-search" "translator" "ai-article-generator" "image-generator")
+
+for service in "${services[@]}"; do
+    echo "Tagging pipeline-$service..."
+    docker tag site11_pipeline-$service:latest site11/pipeline-$service:latest
+done
+
+# Step 5: Deploy services
+echo ""
+echo "5. Deploying services to K8s..."
+
+for service in "${services[@]}"; do
+    echo "Deploying $service..."
+    kubectl apply -f $service.yaml
+done
+
+# Step 6: Check deployment status
+echo ""
+echo "6. Checking deployment status..."
+kubectl -n site11-pipeline get deployments
+
+echo ""
+echo "7. Waiting for pods to be ready..."
+kubectl -n site11-pipeline wait --for=condition=Ready pods --all --timeout=300s || true
+
+# Step 7: Show final status
+echo ""
+echo "✅ Deployment Complete!"
+echo ""
+echo "Current status:"
+kubectl -n site11-pipeline get pods
+echo ""
+echo "To view logs:"
+echo "  kubectl -n site11-pipeline logs -f deployment/pipeline-translator"
+echo ""
+echo "To scale deployments:"
+echo "  kubectl -n site11-pipeline scale deployment pipeline-translator --replicas=5"
+echo ""
+echo "To delete all resources:"
+echo "  kubectl delete namespace site11-pipeline"

k8s/pipeline/google-search.yaml

@@ -0,0 +1,78 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: pipeline-google-search
+  namespace: site11-pipeline
+  labels:
+    app: pipeline-google-search
+    component: data-collector
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: pipeline-google-search
+  template:
+    metadata:
+      labels:
+        app: pipeline-google-search
+        component: data-collector
+    spec:
+      containers:
+      - name: google-search
+        image: site11/pipeline-google-search:latest
+        imagePullPolicy: Always
+        envFrom:
+        - configMapRef:
+            name: pipeline-config
+        - secretRef:
+            name: pipeline-secrets
+        resources:
+          requests:
+            memory: "256Mi"
+            cpu: "100m"
+          limits:
+            memory: "512Mi"
+            cpu: "500m"
+        livenessProbe:
+          exec:
+            command:
+            - python
+            - -c
+            - "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
+          initialDelaySeconds: 30
+          periodSeconds: 30
+        readinessProbe:
+          exec:
+            command:
+            - python
+            - -c
+            - "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
+          initialDelaySeconds: 10
+          periodSeconds: 10
+
+---
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: pipeline-google-search-hpa
+  namespace: site11-pipeline
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: pipeline-google-search
+  minReplicas: 1
+  maxReplicas: 5
+  metrics:
+  - type: Resource
+    resource:
+      name: cpu
+      target:
+        type: Utilization
+        averageUtilization: 70
+  - type: Resource
+    resource:
+      name: memory
+      target:
+        type: Utilization
+        averageUtilization: 80

k8s/pipeline/image-generator.yaml

@@ -0,0 +1,78 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: pipeline-image-generator
+  namespace: site11-pipeline
+  labels:
+    app: pipeline-image-generator
+    component: processor
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: pipeline-image-generator
+  template:
+    metadata:
+      labels:
+        app: pipeline-image-generator
+        component: processor
+    spec:
+      containers:
+      - name: image-generator
+        image: site11/pipeline-image-generator:latest
+        imagePullPolicy: Always
+        envFrom:
+        - configMapRef:
+            name: pipeline-config
+        - secretRef:
+            name: pipeline-secrets
+        resources:
+          requests:
+            memory: "512Mi"
+            cpu: "200m"
+          limits:
+            memory: "1Gi"
+            cpu: "1000m"
+        livenessProbe:
+          exec:
+            command:
+            - python
+            - -c
+            - "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
+          initialDelaySeconds: 30
+          periodSeconds: 30
+        readinessProbe:
+          exec:
+            command:
+            - python
+            - -c
+            - "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
+          initialDelaySeconds: 10
+          periodSeconds: 10
+
+---
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: pipeline-image-generator-hpa
+  namespace: site11-pipeline
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: pipeline-image-generator
+  minReplicas: 1
+  maxReplicas: 6
+  metrics:
+  - type: Resource
+    resource:
+      name: cpu
+      target:
+        type: Utilization
+        averageUtilization: 70
+  - type: Resource
+    resource:
+      name: memory
+      target:
+        type: Utilization
+        averageUtilization: 80

k8s/pipeline/namespace.yaml

@@ -0,0 +1,7 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: site11-pipeline
+  labels:
+    name: site11-pipeline
+    environment: production

k8s/pipeline/rss-collector.yaml

@@ -0,0 +1,78 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: pipeline-rss-collector
+  namespace: site11-pipeline
+  labels:
+    app: pipeline-rss-collector
+    component: data-collector
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: pipeline-rss-collector
+  template:
+    metadata:
+      labels:
+        app: pipeline-rss-collector
+        component: data-collector
+    spec:
+      containers:
+      - name: rss-collector
+        image: site11/pipeline-rss-collector:latest
+        imagePullPolicy: Always
+        envFrom:
+        - configMapRef:
+            name: pipeline-config
+        - secretRef:
+            name: pipeline-secrets
+        resources:
+          requests:
+            memory: "256Mi"
+            cpu: "100m"
+          limits:
+            memory: "512Mi"
+            cpu: "500m"
+        livenessProbe:
+          exec:
+            command:
+            - python
+            - -c
+            - "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
+          initialDelaySeconds: 30
+          periodSeconds: 30
+        readinessProbe:
+          exec:
+            command:
+            - python
+            - -c
+            - "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
+          initialDelaySeconds: 10
+          periodSeconds: 10
+
+---
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: pipeline-rss-collector-hpa
+  namespace: site11-pipeline
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: pipeline-rss-collector
+  minReplicas: 1
+  maxReplicas: 5
+  metrics:
+  - type: Resource
+    resource:
+      name: cpu
+      target:
+        type: Utilization
+        averageUtilization: 70
+  - type: Resource
+    resource:
+      name: memory
+      target:
+        type: Utilization
+        averageUtilization: 80

k8s/pipeline/translator.yaml

@@ -0,0 +1,78 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: pipeline-translator
+  namespace: site11-pipeline
+  labels:
+    app: pipeline-translator
+    component: processor
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: pipeline-translator
+  template:
+    metadata:
+      labels:
+        app: pipeline-translator
+        component: processor
+    spec:
+      containers:
+      - name: translator
+        image: site11/pipeline-translator:latest
+        imagePullPolicy: Always
+        envFrom:
+        - configMapRef:
+            name: pipeline-config
+        - secretRef:
+            name: pipeline-secrets
+        resources:
+          requests:
+            memory: "512Mi"
+            cpu: "200m"
+          limits:
+            memory: "1Gi"
+            cpu: "1000m"
+        livenessProbe:
+          exec:
+            command:
+            - python
+            - -c
+            - "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
+          initialDelaySeconds: 30
+          periodSeconds: 30
+        readinessProbe:
+          exec:
+            command:
+            - python
+            - -c
+            - "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
+          initialDelaySeconds: 10
+          periodSeconds: 10
+
+---
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: pipeline-translator-hpa
+  namespace: site11-pipeline
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: pipeline-translator
+  minReplicas: 2
+  maxReplicas: 10
+  metrics:
+  - type: Resource
+    resource:
+      name: cpu
+      target:
+        type: Utilization
+        averageUtilization: 70
+  - type: Resource
+    resource:
+      name: memory
+      target:
+        type: Utilization
+        averageUtilization: 80