Compare commits
32 Commits
46b5135f45
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 7d29b7ca85 | |||
| d6ae03f42b | |||
| a9024ef9a1 | |||
| 30fe4d0368 | |||
| 55fcce9a38 | |||
| 94bcf9fe9f | |||
| a09ea72c00 | |||
| f4c708c6b4 | |||
| 1d461a7ded | |||
| 52c857fced | |||
| 07088e60e9 | |||
| 7649844023 | |||
| e40f50005d | |||
| de0d548b7a | |||
| 0da9922bc6 | |||
| fb7cf01e6e | |||
| fde852b797 | |||
| e008f17457 | |||
| e60e531cdc | |||
| f4b75b96a5 | |||
| 161f206ae2 | |||
| 0b5a97fd0e | |||
| 07579ea9f5 | |||
| 86ca214dd8 | |||
| e467e76d02 | |||
| deb52e51f2 | |||
| 3ce504e0b1 | |||
| 68cc70118f | |||
| dca130d300 | |||
| d7898f2c98 | |||
| 9c171fb5ef | |||
| aa89057bec |
519
ARCHITECTURE.md
Normal file
519
ARCHITECTURE.md
Normal file
@ -0,0 +1,519 @@
|
|||||||
|
# Site11 Platform Architecture
|
||||||
|
|
||||||
|
## Executive Summary
|
||||||
|
|
||||||
|
Site11 is a **large-scale, AI-powered content generation and aggregation platform** built on a microservices architecture. The platform automatically collects, processes, generates, and distributes multi-language content across various domains including news, entertainment, technology, and regional content for multiple countries.
|
||||||
|
|
||||||
|
### Key Capabilities
|
||||||
|
- **Automated Content Pipeline**: 24/7 content generation without human intervention
|
||||||
|
- **Multi-language Support**: Content in 8+ languages (Korean, English, Chinese, Japanese, French, German, Spanish, Italian)
|
||||||
|
- **Domain-Specific Services**: 30+ specialized microservices for different content domains
|
||||||
|
- **Real-time Processing**: Event-driven architecture with Kafka for real-time data flow
|
||||||
|
- **Scalable Infrastructure**: Containerized services with Kubernetes deployment support
|
||||||
|
|
||||||
|
## System Overview
|
||||||
|
|
||||||
|
### Architecture Pattern
|
||||||
|
**Hybrid Microservices Architecture** combining:
|
||||||
|
- **API Gateway Pattern**: Console service acts as the central orchestrator
|
||||||
|
- **Event-Driven Architecture**: Asynchronous communication via Kafka
|
||||||
|
- **Pipeline Architecture**: Multi-stage content processing workflow
|
||||||
|
- **Service Mesh Ready**: Prepared for Istio/Linkerd integration
|
||||||
|
|
||||||
|
### Technology Stack
|
||||||
|
|
||||||
|
| Layer | Technology | Purpose |
|
||||||
|
|-------|------------|---------|
|
||||||
|
| **Backend** | FastAPI (Python 3.11) | High-performance async API services |
|
||||||
|
| **Frontend** | React 18 + TypeScript + Vite | Modern responsive web interfaces |
|
||||||
|
| **Primary Database** | MongoDB 7.0 | Document storage for flexible content |
|
||||||
|
| **Cache Layer** | Redis 7 | High-speed caching and queue management |
|
||||||
|
| **Message Broker** | Apache Kafka | Event streaming and service communication |
|
||||||
|
| **Search Engine** | Apache Solr 9.4 | Full-text search capabilities |
|
||||||
|
| **Object Storage** | MinIO | Media and file storage |
|
||||||
|
| **Containerization** | Docker & Docker Compose | Service isolation and deployment |
|
||||||
|
| **Orchestration** | Kubernetes (Kind/Docker Desktop) | Production deployment and scaling |
|
||||||
|
|
||||||
|
## Core Services Architecture
|
||||||
|
|
||||||
|
### 1. Infrastructure Services
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────┐
|
||||||
|
│ Infrastructure Layer │
|
||||||
|
├───────────────┬───────────────┬──────────────┬──────────────┤
|
||||||
|
│ MongoDB │ Redis │ Kafka │ MinIO │
|
||||||
|
│ (Primary DB) │ (Cache) │ (Events) │ (Storage) │
|
||||||
|
├───────────────┼───────────────┼──────────────┼──────────────┤
|
||||||
|
│ Port: 27017 │ Port: 6379 │ Port: 9092 │ Port: 9000 │
|
||||||
|
└───────────────┴───────────────┴──────────────┴──────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Core Application Services
|
||||||
|
|
||||||
|
#### Console Service (API Gateway)
|
||||||
|
- **Port**: 8000 (Backend), 3000 (Frontend via Envoy)
|
||||||
|
- **Role**: Central orchestrator and monitoring dashboard
|
||||||
|
- **Responsibilities**:
|
||||||
|
- Service discovery and health monitoring
|
||||||
|
- Unified authentication portal
|
||||||
|
- Request routing to microservices
|
||||||
|
- Real-time metrics aggregation
|
||||||
|
|
||||||
|
#### Content Services
|
||||||
|
- **AI Writer** (8019): AI-powered article generation using Claude API
|
||||||
|
- **News Aggregator** (8018): Aggregates content from multiple sources
|
||||||
|
- **RSS Feed** (8017): RSS feed collection and management
|
||||||
|
- **Google Search** (8016): Search integration for content discovery
|
||||||
|
- **Search Service** (8015): Full-text search via Solr
|
||||||
|
|
||||||
|
#### Support Services
|
||||||
|
- **Users** (8007-8008): User management and authentication
|
||||||
|
- **OAuth** (8003-8004): OAuth2 authentication provider
|
||||||
|
- **Images** (8001-8002): Image processing and caching
|
||||||
|
- **Files** (8014): File management with MinIO integration
|
||||||
|
- **Notifications** (8013): Email, SMS, and push notifications
|
||||||
|
- **Statistics** (8012): Analytics and metrics collection
|
||||||
|
|
||||||
|
### 3. Pipeline Architecture
|
||||||
|
|
||||||
|
The pipeline represents the **heart of the content generation system**, processing content through multiple stages:
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────────────────────────────────────────────────────┐
|
||||||
|
│ Content Pipeline Flow │
|
||||||
|
├──────────────────────────────────────────────────────────────┤
|
||||||
|
│ │
|
||||||
|
│ [Scheduler] ─────> [RSS Collector] ────> [Google Search] │
|
||||||
|
│ │ │ │
|
||||||
|
│ │ ▼ │
|
||||||
|
│ │ [AI Generator] │
|
||||||
|
│ │ │ │
|
||||||
|
│ ▼ ▼ │
|
||||||
|
│ [Keywords] [Translator] │
|
||||||
|
│ Manager │ │
|
||||||
|
│ ▼ │
|
||||||
|
│ [Image Generator] │
|
||||||
|
│ │ │
|
||||||
|
│ ▼ │
|
||||||
|
│ [Language Sync] │
|
||||||
|
│ │
|
||||||
|
└──────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Pipeline Components
|
||||||
|
|
||||||
|
1. **Multi-threaded Scheduler**: Orchestrates the entire pipeline workflow
|
||||||
|
2. **Keyword Manager** (API Port 8100): Manages search keywords and topics
|
||||||
|
3. **RSS Collector**: Collects content from RSS feeds
|
||||||
|
4. **Google Search Worker**: Searches for trending content
|
||||||
|
5. **AI Article Generator**: Generates articles using Claude AI
|
||||||
|
6. **Translator**: Translates content using DeepL API
|
||||||
|
7. **Image Generator**: Creates images for articles
|
||||||
|
8. **Language Sync**: Ensures content consistency across languages
|
||||||
|
9. **Pipeline Monitor** (Port 8100): Real-time pipeline monitoring dashboard
|
||||||
|
|
||||||
|
### 4. Domain-Specific Services
|
||||||
|
|
||||||
|
The platform includes **30+ specialized services** for different content domains:
|
||||||
|
|
||||||
|
#### Entertainment Services
|
||||||
|
- **Artist Services**: blackpink, enhypen, ive, nct, straykids, twice
|
||||||
|
- **K-Culture**: Korean cultural content
|
||||||
|
- **Media Empire**: Entertainment industry coverage
|
||||||
|
|
||||||
|
#### Regional Services
|
||||||
|
- **Korea** (8020-8021): Korean market content
|
||||||
|
- **Japan** (8022-8023): Japanese market content
|
||||||
|
- **China** (8024-8025): Chinese market content
|
||||||
|
- **USA** (8026-8027): US market content
|
||||||
|
|
||||||
|
#### Technology Services
|
||||||
|
- **AI Service** (8028-8029): AI technology news
|
||||||
|
- **Crypto** (8030-8031): Cryptocurrency coverage
|
||||||
|
- **Apple** (8032-8033): Apple ecosystem news
|
||||||
|
- **Google** (8034-8035): Google technology updates
|
||||||
|
- **Samsung** (8036-8037): Samsung product news
|
||||||
|
- **LG** (8038-8039): LG technology coverage
|
||||||
|
|
||||||
|
#### Business Services
|
||||||
|
- **WSJ** (8040-8041): Wall Street Journal integration
|
||||||
|
- **Musk** (8042-8043): Elon Musk related content
|
||||||
|
|
||||||
|
## Data Flow Architecture
|
||||||
|
|
||||||
|
### 1. Content Generation Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
User Request / Scheduled Task
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[Console API Gateway]
|
||||||
|
│
|
||||||
|
├──> [Keyword Manager] ──> Topics/Keywords
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[Pipeline Scheduler]
|
||||||
|
│
|
||||||
|
├──> [RSS Collector] ──> Feed Content
|
||||||
|
├──> [Google Search] ──> Search Results
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[AI Article Generator]
|
||||||
|
│
|
||||||
|
├──> [MongoDB] (Store Korean Original)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[Translator Service]
|
||||||
|
│
|
||||||
|
├──> [MongoDB] (Store Translations)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[Image Generator]
|
||||||
|
│
|
||||||
|
├──> [MinIO] (Store Images)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
[Language Sync]
|
||||||
|
│
|
||||||
|
└──> [Content Ready for Distribution]
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Event-Driven Communication
|
||||||
|
|
||||||
|
```
|
||||||
|
Service A ──[Publish]──> Kafka Topic ──[Subscribe]──> Service B
|
||||||
|
│
|
||||||
|
├──> Service C
|
||||||
|
└──> Service D
|
||||||
|
|
||||||
|
Topics:
|
||||||
|
- content.created
|
||||||
|
- content.updated
|
||||||
|
- translation.completed
|
||||||
|
- image.generated
|
||||||
|
- user.activity
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Caching Strategy
|
||||||
|
|
||||||
|
```
|
||||||
|
Client Request ──> [Console] ──> [Redis Cache]
|
||||||
|
│
|
||||||
|
├─ HIT ──> Return Cached
|
||||||
|
│
|
||||||
|
└─ MISS ──> [Service] ──> [MongoDB]
|
||||||
|
│
|
||||||
|
└──> Update Cache
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deployment Architecture
|
||||||
|
|
||||||
|
### 1. Development Environment (Docker Compose)
|
||||||
|
|
||||||
|
All services run in Docker containers with:
|
||||||
|
- **Single docker-compose.yml**: Defines all services
|
||||||
|
- **Shared network**: `site11_network` for inter-service communication
|
||||||
|
- **Persistent volumes**: Data stored in `./data/` directory
|
||||||
|
- **Hot-reload**: Code mounted for development
|
||||||
|
|
||||||
|
### 2. Production Environment (Kubernetes)
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────┐
|
||||||
|
│ Kubernetes Cluster │
|
||||||
|
├─────────────────────────────────────────────────────────────┤
|
||||||
|
│ │
|
||||||
|
│ ┌──────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Ingress (Nginx) │ │
|
||||||
|
│ └──────────────────────────────────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ┌──────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Service Mesh (Optional) │ │
|
||||||
|
│ └──────────────────────────────────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ┌───────────────────────┼───────────────────────────┐ │
|
||||||
|
│ │ Namespace: site11-core │ │
|
||||||
|
│ ├──────────────┬────────────────┬──────────────────┤ │
|
||||||
|
│ │ Console │ MongoDB │ Redis │ │
|
||||||
|
│ │ Deployment │ StatefulSet │ StatefulSet │ │
|
||||||
|
│ └──────────────┴────────────────┴──────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ ┌───────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Namespace: site11-pipeline │ │
|
||||||
|
│ ├──────────────┬────────────────┬──────────────────┤ │
|
||||||
|
│ │ Scheduler │ RSS Collector │ AI Generator │ │
|
||||||
|
│ │ Deployment │ Deployment │ Deployment │ │
|
||||||
|
│ └──────────────┴────────────────┴──────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ ┌───────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Namespace: site11-services │ │
|
||||||
|
│ ├──────────────┬────────────────┬──────────────────┤ │
|
||||||
|
│ │ Artist Svcs │ Regional Svcs │ Tech Svcs │ │
|
||||||
|
│ │ Deployments │ Deployments │ Deployments │ │
|
||||||
|
│ └──────────────┴────────────────┴──────────────────┘ │
|
||||||
|
│ │
|
||||||
|
└─────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Hybrid Deployment
|
||||||
|
|
||||||
|
The platform supports **hybrid deployment** combining:
|
||||||
|
- **Docker Compose**: For development and small deployments
|
||||||
|
- **Kubernetes**: For production scaling
|
||||||
|
- **Docker Desktop Kubernetes**: For local K8s testing
|
||||||
|
- **Kind**: For lightweight K8s development
|
||||||
|
|
||||||
|
## Security Architecture
|
||||||
|
|
||||||
|
### Authentication & Authorization
|
||||||
|
```
|
||||||
|
┌──────────────────────────────────────────────────────────────┐
|
||||||
|
│ Security Flow │
|
||||||
|
├──────────────────────────────────────────────────────────────┤
|
||||||
|
│ │
|
||||||
|
│ Client ──> [Console Gateway] ──> [OAuth Service] │
|
||||||
|
│ │ │ │
|
||||||
|
│ │ ▼ │
|
||||||
|
│ │ [JWT Generation] │
|
||||||
|
│ │ │ │
|
||||||
|
│ ▼ ▼ │
|
||||||
|
│ [Token Validation] <────── [Token] │
|
||||||
|
│ │ │
|
||||||
|
│ ▼ │
|
||||||
|
│ [Service Access] │
|
||||||
|
│ │
|
||||||
|
└──────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Security Measures
|
||||||
|
- **JWT-based authentication**: Stateless token authentication
|
||||||
|
- **Service-to-service auth**: Internal service tokens
|
||||||
|
- **Rate limiting**: API Gateway level throttling
|
||||||
|
- **CORS configuration**: Controlled cross-origin access
|
||||||
|
- **Environment variables**: Sensitive data in `.env` files
|
||||||
|
- **Network isolation**: Services communicate within Docker/K8s network
|
||||||
|
|
||||||
|
## Monitoring & Observability
|
||||||
|
|
||||||
|
### 1. Health Checks
|
||||||
|
Every service implements health endpoints:
|
||||||
|
```python
|
||||||
|
GET /health
|
||||||
|
Response: {"status": "healthy", "service": "service-name"}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Monitoring Stack
|
||||||
|
- **Pipeline Monitor**: Real-time pipeline status (Port 8100)
|
||||||
|
- **Console Dashboard**: Service health overview
|
||||||
|
- **Redis Queue Monitoring**: Queue depth and processing rates
|
||||||
|
- **MongoDB Metrics**: Database performance metrics
|
||||||
|
|
||||||
|
### 3. Logging Strategy
|
||||||
|
- Centralized logging with structured JSON format
|
||||||
|
- Log levels: DEBUG, INFO, WARNING, ERROR
|
||||||
|
- Correlation IDs for distributed tracing
|
||||||
|
|
||||||
|
## Scalability & Performance
|
||||||
|
|
||||||
|
### Horizontal Scaling
|
||||||
|
- **Stateless services**: Easy horizontal scaling
|
||||||
|
- **Load balancing**: Kubernetes service mesh
|
||||||
|
- **Auto-scaling**: Based on CPU/memory metrics
|
||||||
|
|
||||||
|
### Performance Optimizations
|
||||||
|
- **Redis caching**: Reduces database load
|
||||||
|
- **Async processing**: FastAPI async endpoints
|
||||||
|
- **Batch processing**: Pipeline processes in batches
|
||||||
|
- **Connection pooling**: Database connection reuse
|
||||||
|
- **CDN ready**: Static content delivery
|
||||||
|
|
||||||
|
### Resource Management
|
||||||
|
```yaml
|
||||||
|
Resources per Service:
|
||||||
|
- CPU: 100m - 500m (request), 1000m (limit)
|
||||||
|
- Memory: 128Mi - 512Mi (request), 1Gi (limit)
|
||||||
|
- Storage: 1Gi - 10Gi PVC for data services
|
||||||
|
```
|
||||||
|
|
||||||
|
## Development Workflow
|
||||||
|
|
||||||
|
### 1. Local Development
|
||||||
|
```bash
|
||||||
|
# Start all services
|
||||||
|
docker-compose up -d
|
||||||
|
|
||||||
|
# Start specific services
|
||||||
|
docker-compose up -d console mongodb redis
|
||||||
|
|
||||||
|
# View logs
|
||||||
|
docker-compose logs -f [service-name]
|
||||||
|
|
||||||
|
# Rebuild after changes
|
||||||
|
docker-compose build [service-name]
|
||||||
|
docker-compose up -d [service-name]
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Testing
|
||||||
|
```bash
|
||||||
|
# Run unit tests
|
||||||
|
docker-compose exec [service-name] pytest
|
||||||
|
|
||||||
|
# Integration tests
|
||||||
|
docker-compose exec [service-name] pytest tests/integration
|
||||||
|
|
||||||
|
# Load testing
|
||||||
|
docker-compose exec [service-name] locust
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Deployment
|
||||||
|
```bash
|
||||||
|
# Development
|
||||||
|
./deploy-local.sh
|
||||||
|
|
||||||
|
# Staging (Kind)
|
||||||
|
./deploy-kind.sh
|
||||||
|
|
||||||
|
# Production (Kubernetes)
|
||||||
|
./deploy-k8s.sh
|
||||||
|
|
||||||
|
# Docker Hub
|
||||||
|
./deploy-dockerhub.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
## Key Design Decisions
|
||||||
|
|
||||||
|
### 1. Microservices over Monolith
|
||||||
|
- **Reasoning**: Independent scaling, technology diversity, fault isolation
|
||||||
|
- **Trade-off**: Increased complexity, network overhead
|
||||||
|
|
||||||
|
### 2. MongoDB as Primary Database
|
||||||
|
- **Reasoning**: Flexible schema for diverse content types
|
||||||
|
- **Trade-off**: Eventual consistency, complex queries
|
||||||
|
|
||||||
|
### 3. Event-Driven with Kafka
|
||||||
|
- **Reasoning**: Decoupling, scalability, real-time processing
|
||||||
|
- **Trade-off**: Operational complexity, debugging challenges
|
||||||
|
|
||||||
|
### 4. Python/FastAPI for Backend
|
||||||
|
- **Reasoning**: Async support, fast development, AI library ecosystem
|
||||||
|
- **Trade-off**: GIL limitations, performance vs compiled languages
|
||||||
|
|
||||||
|
### 5. Container-First Approach
|
||||||
|
- **Reasoning**: Consistent environments, easy deployment, cloud-native
|
||||||
|
- **Trade-off**: Resource overhead, container management
|
||||||
|
|
||||||
|
## Performance Metrics
|
||||||
|
|
||||||
|
### Current Capacity (Single Instance)
|
||||||
|
- **Content Generation**: 1000+ articles/day
|
||||||
|
- **Translation Throughput**: 8 languages simultaneously
|
||||||
|
- **API Response Time**: <100ms p50, <500ms p99
|
||||||
|
- **Queue Processing**: 100+ jobs/minute
|
||||||
|
- **Storage**: Scalable to TBs with MinIO
|
||||||
|
|
||||||
|
### Scaling Potential
|
||||||
|
- **Horizontal**: Each service can scale to 10+ replicas
|
||||||
|
- **Vertical**: Services can use up to 4GB RAM, 4 CPUs
|
||||||
|
- **Geographic**: Multi-region deployment ready
|
||||||
|
|
||||||
|
## Future Roadmap
|
||||||
|
|
||||||
|
### Phase 1: Current State ✅
|
||||||
|
- Core microservices architecture
|
||||||
|
- Automated content pipeline
|
||||||
|
- Multi-language support
|
||||||
|
- Basic monitoring
|
||||||
|
|
||||||
|
### Phase 2: Enhanced Observability (Q1 2025)
|
||||||
|
- Prometheus + Grafana integration
|
||||||
|
- Distributed tracing with Jaeger
|
||||||
|
- ELK stack for logging
|
||||||
|
- Advanced alerting
|
||||||
|
|
||||||
|
### Phase 3: Advanced Features (Q2 2025)
|
||||||
|
- Machine Learning pipeline
|
||||||
|
- Real-time analytics
|
||||||
|
- GraphQL API layer
|
||||||
|
- WebSocket support
|
||||||
|
|
||||||
|
### Phase 4: Enterprise Features (Q3 2025)
|
||||||
|
- Multi-tenancy support
|
||||||
|
- Advanced RBAC
|
||||||
|
- Audit logging
|
||||||
|
- Compliance features
|
||||||
|
|
||||||
|
## Conclusion
|
||||||
|
|
||||||
|
Site11 represents a **modern, scalable, AI-driven content platform** that leverages:
|
||||||
|
- **Microservices architecture** for modularity and scalability
|
||||||
|
- **Event-driven design** for real-time processing
|
||||||
|
- **Container orchestration** for deployment flexibility
|
||||||
|
- **AI integration** for automated content generation
|
||||||
|
- **Multi-language support** for global reach
|
||||||
|
|
||||||
|
The architecture is designed to handle **massive scale**, support **rapid development**, and provide **high availability** while maintaining **operational simplicity** through automation and monitoring.
|
||||||
|
|
||||||
|
## Appendix
|
||||||
|
|
||||||
|
### A. Service Port Mapping
|
||||||
|
|
||||||
|
| Service | Backend Port | Frontend Port | Description |
|
||||||
|
|---------|-------------|---------------|-------------|
|
||||||
|
| Console | 8000 | 3000 | API Gateway & Dashboard |
|
||||||
|
| Users | 8007 | 8008 | User Management |
|
||||||
|
| OAuth | 8003 | 8004 | Authentication |
|
||||||
|
| Images | 8001 | 8002 | Image Processing |
|
||||||
|
| Statistics | 8012 | - | Analytics |
|
||||||
|
| Notifications | 8013 | - | Alerts & Messages |
|
||||||
|
| Files | 8014 | - | File Storage |
|
||||||
|
| Search | 8015 | - | Full-text Search |
|
||||||
|
| Google Search | 8016 | - | Search Integration |
|
||||||
|
| RSS Feed | 8017 | - | RSS Management |
|
||||||
|
| News Aggregator | 8018 | - | Content Aggregation |
|
||||||
|
| AI Writer | 8019 | - | AI Content Generation |
|
||||||
|
| Pipeline Monitor | 8100 | - | Pipeline Dashboard |
|
||||||
|
| Keyword Manager | 8100 | - | Keyword API |
|
||||||
|
|
||||||
|
### B. Environment Variables
|
||||||
|
|
||||||
|
Key configuration managed through `.env`:
|
||||||
|
- Database connections (MongoDB, Redis)
|
||||||
|
- API keys (Claude, DeepL, Google)
|
||||||
|
- Service URLs and ports
|
||||||
|
- JWT secrets
|
||||||
|
- Cache TTLs
|
||||||
|
|
||||||
|
### C. Database Schema
|
||||||
|
|
||||||
|
MongoDB Collections:
|
||||||
|
- `users`: User profiles and authentication
|
||||||
|
- `articles_[lang]`: Articles by language
|
||||||
|
- `keywords`: Search keywords and topics
|
||||||
|
- `rss_feeds`: RSS feed configurations
|
||||||
|
- `statistics`: Analytics data
|
||||||
|
- `files`: File metadata
|
||||||
|
|
||||||
|
### D. API Documentation
|
||||||
|
|
||||||
|
All services provide OpenAPI/Swagger documentation at:
|
||||||
|
```
|
||||||
|
http://[service-url]/docs
|
||||||
|
```
|
||||||
|
|
||||||
|
### E. Deployment Scripts
|
||||||
|
|
||||||
|
| Script | Purpose |
|
||||||
|
|--------|---------|
|
||||||
|
| `deploy-local.sh` | Local Docker Compose deployment |
|
||||||
|
| `deploy-kind.sh` | Kind Kubernetes deployment |
|
||||||
|
| `deploy-docker-desktop.sh` | Docker Desktop K8s deployment |
|
||||||
|
| `deploy-dockerhub.sh` | Push images to Docker Hub |
|
||||||
|
| `backup-mongodb.sh` | MongoDB backup utility |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Document Version**: 1.0.0
|
||||||
|
**Last Updated**: September 2025
|
||||||
|
**Platform Version**: Site11 v1.0
|
||||||
|
**Architecture Review**: Approved for Production
|
||||||
38
CLAUDE.md
38
CLAUDE.md
@ -273,3 +273,41 @@ Services register themselves with Console on startup and send periodic heartbeat
|
|||||||
- Console validates tokens and forwards to services
|
- Console validates tokens and forwards to services
|
||||||
- Internal service communication uses service tokens
|
- Internal service communication uses service tokens
|
||||||
- Rate limiting at API Gateway level
|
- Rate limiting at API Gateway level
|
||||||
|
|
||||||
|
## Deployment Guide
|
||||||
|
|
||||||
|
### News API Deployment
|
||||||
|
**IMPORTANT**: News API는 Kubernetes에 배포되며 Docker 이미지 버전 관리가 필요합니다.
|
||||||
|
|
||||||
|
상세 가이드: `services/news-api/DEPLOYMENT.md` 참조
|
||||||
|
|
||||||
|
#### Quick Deploy
|
||||||
|
```bash
|
||||||
|
# 1. 버전 설정
|
||||||
|
export VERSION=v1.1.0
|
||||||
|
|
||||||
|
# 2. 빌드 및 푸시 (버전 태그 + latest)
|
||||||
|
cd services/news-api
|
||||||
|
docker build -t yakenator/news-api:${VERSION} -t yakenator/news-api:latest -f backend/Dockerfile backend
|
||||||
|
docker push yakenator/news-api:${VERSION}
|
||||||
|
docker push yakenator/news-api:latest
|
||||||
|
|
||||||
|
# 3. Kubernetes 재시작
|
||||||
|
kubectl -n site11-news rollout restart deployment news-api-deployment
|
||||||
|
kubectl -n site11-news rollout status deployment news-api-deployment
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Version Management
|
||||||
|
- **Major (v2.0.0)**: Breaking changes, API 스펙 변경
|
||||||
|
- **Minor (v1.1.0)**: 새 기능 추가, 하위 호환성 유지
|
||||||
|
- **Patch (v1.0.1)**: 버그 수정, 작은 개선
|
||||||
|
|
||||||
|
#### Rollback
|
||||||
|
```bash
|
||||||
|
# 이전 버전으로 롤백
|
||||||
|
kubectl -n site11-news rollout undo deployment news-api-deployment
|
||||||
|
|
||||||
|
# 특정 버전으로 롤백
|
||||||
|
kubectl -n site11-news set image deployment/news-api-deployment \
|
||||||
|
news-api=yakenator/news-api:v1.0.0
|
||||||
|
```
|
||||||
|
|||||||
152
KIND_README.md
Normal file
152
KIND_README.md
Normal file
@ -0,0 +1,152 @@
|
|||||||
|
# Site11 KIND Kubernetes 개발 환경
|
||||||
|
|
||||||
|
Docker Compose를 통한 간편한 로컬 Kubernetes 개발 환경
|
||||||
|
|
||||||
|
## 빠른 시작
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. 관리 컨테이너 시작 (한 번만 실행)
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml up -d
|
||||||
|
|
||||||
|
# 2. KIND 클러스터 생성 및 Console 배포
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh setup
|
||||||
|
|
||||||
|
# 3. 상태 확인
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh status
|
||||||
|
```
|
||||||
|
|
||||||
|
완료! 이제 브라우저에서 접속 가능합니다:
|
||||||
|
- **Frontend**: http://localhost:3000
|
||||||
|
- **Backend**: http://localhost:8000
|
||||||
|
|
||||||
|
## 실시간 모니터링
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml logs -f monitor
|
||||||
|
```
|
||||||
|
|
||||||
|
## 일상적인 작업
|
||||||
|
|
||||||
|
### 클러스터 상태 확인
|
||||||
|
```bash
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh status
|
||||||
|
```
|
||||||
|
|
||||||
|
### kubectl 사용
|
||||||
|
```bash
|
||||||
|
# Pod 목록
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli kubectl get pods -n site11-console
|
||||||
|
|
||||||
|
# 로그 확인
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli kubectl logs <pod-name> -n site11-console
|
||||||
|
|
||||||
|
# Shell 접속
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli kubectl exec -it <pod-name> -n site11-console -- /bin/bash
|
||||||
|
```
|
||||||
|
|
||||||
|
### 서비스 재배포
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 이미지 빌드 (로컬)
|
||||||
|
docker build -t yakenator/site11-console-backend:latest \
|
||||||
|
-f services/console/backend/Dockerfile services/console/backend
|
||||||
|
|
||||||
|
# 이미지 KIND에 로드
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli \
|
||||||
|
kind load docker-image yakenator/site11-console-backend:latest --name site11-dev
|
||||||
|
|
||||||
|
# Pod 재시작
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli \
|
||||||
|
kubectl rollout restart deployment/console-backend -n site11-console
|
||||||
|
```
|
||||||
|
|
||||||
|
## Shell 접속
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli bash
|
||||||
|
```
|
||||||
|
|
||||||
|
Shell 내에서는 `kind`, `kubectl`, `docker` 명령을 모두 사용할 수 있습니다.
|
||||||
|
|
||||||
|
## 클러스터 삭제 및 재생성
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 클러스터 삭제
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh delete
|
||||||
|
|
||||||
|
# 클러스터 재생성
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh setup
|
||||||
|
```
|
||||||
|
|
||||||
|
## 관리 컨테이너 중지
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml down
|
||||||
|
```
|
||||||
|
|
||||||
|
**참고**: 이것은 관리 헬퍼 컨테이너만 중지합니다. KIND 클러스터 자체는 계속 실행됩니다.
|
||||||
|
|
||||||
|
## 별칭(Alias) 설정 (선택사항)
|
||||||
|
|
||||||
|
`.bashrc` 또는 `.zshrc`에 추가:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
alias k8s='docker-compose -f docker-compose.kubernetes.yml'
|
||||||
|
alias k8s-exec='docker-compose -f docker-compose.kubernetes.yml exec kind-cli'
|
||||||
|
alias k8s-setup='docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh'
|
||||||
|
alias k8s-kubectl='docker-compose -f docker-compose.kubernetes.yml exec kind-cli kubectl'
|
||||||
|
```
|
||||||
|
|
||||||
|
사용 예:
|
||||||
|
```bash
|
||||||
|
k8s up -d
|
||||||
|
k8s-setup setup
|
||||||
|
k8s-setup status
|
||||||
|
k8s-kubectl get pods -A
|
||||||
|
k8s logs -f monitor
|
||||||
|
```
|
||||||
|
|
||||||
|
## 상세 문서
|
||||||
|
|
||||||
|
더 자세한 정보는 다음 문서를 참고하세요:
|
||||||
|
- [KUBERNETES.md](./KUBERNETES.md) - 전체 가이드
|
||||||
|
- [docs/KIND_SETUP.md](./docs/KIND_SETUP.md) - KIND 상세 설정
|
||||||
|
|
||||||
|
## 문제 해결
|
||||||
|
|
||||||
|
### 클러스터가 시작되지 않는 경우
|
||||||
|
```bash
|
||||||
|
# Docker Desktop이 실행 중인지 확인
|
||||||
|
docker ps
|
||||||
|
|
||||||
|
# KIND 클러스터 상태 확인
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli kind get clusters
|
||||||
|
|
||||||
|
# 클러스터 재생성
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh delete
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh setup
|
||||||
|
```
|
||||||
|
|
||||||
|
### 이미지가 로드되지 않는 경우
|
||||||
|
```bash
|
||||||
|
# 로컬에 이미지가 있는지 확인
|
||||||
|
docker images | grep site11
|
||||||
|
|
||||||
|
# 이미지 빌드 후 다시 로드
|
||||||
|
docker build -t yakenator/site11-console-backend:latest \
|
||||||
|
-f services/console/backend/Dockerfile services/console/backend
|
||||||
|
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli \
|
||||||
|
kind load docker-image yakenator/site11-console-backend:latest --name site11-dev
|
||||||
|
```
|
||||||
|
|
||||||
|
### NodePort 접속이 안되는 경우
|
||||||
|
```bash
|
||||||
|
# 서비스 확인
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli \
|
||||||
|
kubectl get svc -n site11-console
|
||||||
|
|
||||||
|
# NodePort 확인 (30080, 30081이어야 함)
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli \
|
||||||
|
kubectl describe svc console-frontend -n site11-console
|
||||||
|
```
|
||||||
371
KUBERNETES.md
Normal file
371
KUBERNETES.md
Normal file
@ -0,0 +1,371 @@
|
|||||||
|
# Kubernetes Development Environment (KIND)
|
||||||
|
|
||||||
|
Site11 프로젝트는 KIND (Kubernetes IN Docker)를 사용하여 로컬 Kubernetes 개발 환경을 구성합니다.
|
||||||
|
|
||||||
|
## 목차
|
||||||
|
- [사전 요구사항](#사전-요구사항)
|
||||||
|
- [빠른 시작](#빠른-시작)
|
||||||
|
- [관리 방법](#관리-방법)
|
||||||
|
- [접속 정보](#접속-정보)
|
||||||
|
- [문제 해결](#문제-해결)
|
||||||
|
|
||||||
|
## 사전 요구사항
|
||||||
|
|
||||||
|
다음 도구들이 설치되어 있어야 합니다:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Docker Desktop
|
||||||
|
brew install --cask docker
|
||||||
|
|
||||||
|
# KIND
|
||||||
|
brew install kind
|
||||||
|
|
||||||
|
# kubectl
|
||||||
|
brew install kubectl
|
||||||
|
```
|
||||||
|
|
||||||
|
## 빠른 시작
|
||||||
|
|
||||||
|
### 방법 1: docker-compose 사용 (권장) ⭐
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. 관리 컨테이너 시작
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml up -d
|
||||||
|
|
||||||
|
# 2. KIND 클러스터 생성 및 배포
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh setup
|
||||||
|
|
||||||
|
# 3. 상태 확인
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh status
|
||||||
|
|
||||||
|
# 4. 실시간 모니터링
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml logs -f monitor
|
||||||
|
```
|
||||||
|
|
||||||
|
### 방법 2: 로컬 스크립트 사용
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 전체 환경 한번에 설정 (클러스터 생성 + 서비스 배포)
|
||||||
|
./scripts/kind-setup.sh setup
|
||||||
|
|
||||||
|
# 상태 확인
|
||||||
|
./scripts/kind-setup.sh status
|
||||||
|
|
||||||
|
# 접속 정보 확인
|
||||||
|
./scripts/kind-setup.sh access
|
||||||
|
```
|
||||||
|
|
||||||
|
### 방법 3: 수동 설정
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. 클러스터 생성
|
||||||
|
kind create cluster --config k8s/kind-dev-cluster.yaml
|
||||||
|
|
||||||
|
# 2. 네임스페이스 생성
|
||||||
|
kubectl create namespace site11-console
|
||||||
|
kubectl create namespace site11-pipeline
|
||||||
|
|
||||||
|
# 3. Docker 이미지 로드
|
||||||
|
kind load docker-image yakenator/site11-console-backend:latest --name site11-dev
|
||||||
|
kind load docker-image yakenator/site11-console-frontend:latest --name site11-dev
|
||||||
|
|
||||||
|
# 4. 서비스 배포
|
||||||
|
kubectl apply -f k8s/kind/console-mongodb-redis.yaml
|
||||||
|
kubectl apply -f k8s/kind/console-backend.yaml
|
||||||
|
kubectl apply -f k8s/kind/console-frontend.yaml
|
||||||
|
|
||||||
|
# 5. 상태 확인
|
||||||
|
kubectl get pods -n site11-console
|
||||||
|
```
|
||||||
|
|
||||||
|
## 관리 방법
|
||||||
|
|
||||||
|
### docker-compose 명령어 (권장)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 관리 컨테이너 시작
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml up -d
|
||||||
|
|
||||||
|
# 클러스터 생성
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh create
|
||||||
|
|
||||||
|
# 클러스터 삭제
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh delete
|
||||||
|
|
||||||
|
# 전체 설정 (생성 + 배포)
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh setup
|
||||||
|
|
||||||
|
# 상태 확인
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh status
|
||||||
|
|
||||||
|
# 실시간 모니터링
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml logs -f monitor
|
||||||
|
|
||||||
|
# kubectl 직접 사용
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli kubectl get pods -A
|
||||||
|
|
||||||
|
# Shell 접속
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml exec kind-cli bash
|
||||||
|
|
||||||
|
# 관리 컨테이너 중지
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml down
|
||||||
|
```
|
||||||
|
|
||||||
|
### 로컬 스크립트 명령어
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 클러스터 생성
|
||||||
|
./scripts/kind-setup.sh create
|
||||||
|
|
||||||
|
# 클러스터 삭제
|
||||||
|
./scripts/kind-setup.sh delete
|
||||||
|
|
||||||
|
# 네임스페이스 생성
|
||||||
|
./scripts/kind-setup.sh deploy-namespaces
|
||||||
|
|
||||||
|
# Docker 이미지 로드
|
||||||
|
./scripts/kind-setup.sh load-images
|
||||||
|
|
||||||
|
# Console 서비스 배포
|
||||||
|
./scripts/kind-setup.sh deploy-console
|
||||||
|
|
||||||
|
# 상태 확인
|
||||||
|
./scripts/kind-setup.sh status
|
||||||
|
|
||||||
|
# Pod 로그 확인
|
||||||
|
./scripts/kind-setup.sh logs site11-console [pod-name]
|
||||||
|
|
||||||
|
# 접속 정보 표시
|
||||||
|
./scripts/kind-setup.sh access
|
||||||
|
```
|
||||||
|
|
||||||
|
### kubectl 명령어
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 전체 리소스 확인
|
||||||
|
kubectl get all -n site11-console
|
||||||
|
|
||||||
|
# Pod 상세 정보
|
||||||
|
kubectl describe pod <pod-name> -n site11-console
|
||||||
|
|
||||||
|
# Pod 로그 확인
|
||||||
|
kubectl logs <pod-name> -n site11-console -f
|
||||||
|
|
||||||
|
# Pod 내부 접속
|
||||||
|
kubectl exec -it <pod-name> -n site11-console -- /bin/bash
|
||||||
|
|
||||||
|
# 서비스 확인
|
||||||
|
kubectl get svc -n site11-console
|
||||||
|
|
||||||
|
# 노드 확인
|
||||||
|
kubectl get nodes
|
||||||
|
```
|
||||||
|
|
||||||
|
## 클러스터 구성
|
||||||
|
|
||||||
|
### 노드 구성 (5 노드)
|
||||||
|
|
||||||
|
- **Control Plane (1개)**: 클러스터 마스터 노드
|
||||||
|
- NodePort 매핑: 30080 → 3000 (Frontend), 30081 → 8000 (Backend)
|
||||||
|
|
||||||
|
- **Worker Nodes (4개)**:
|
||||||
|
- `workload=console`: Console 서비스 전용
|
||||||
|
- `workload=pipeline-collector`: 데이터 수집 서비스
|
||||||
|
- `workload=pipeline-processor`: 데이터 처리 서비스
|
||||||
|
- `workload=pipeline-generator`: 콘텐츠 생성 서비스
|
||||||
|
|
||||||
|
### 네임스페이스
|
||||||
|
|
||||||
|
- `site11-console`: Console 프론트엔드/백엔드, MongoDB, Redis
|
||||||
|
- `site11-pipeline`: Pipeline 관련 서비스들
|
||||||
|
|
||||||
|
## 접속 정보
|
||||||
|
|
||||||
|
### Console Services
|
||||||
|
|
||||||
|
- **Frontend**: http://localhost:3000
|
||||||
|
- NodePort: 30080
|
||||||
|
- 컨테이너 포트: 80
|
||||||
|
|
||||||
|
- **Backend**: http://localhost:8000
|
||||||
|
- NodePort: 30081
|
||||||
|
- 컨테이너 포트: 8000
|
||||||
|
|
||||||
|
### 내부 서비스 (Pod 내부에서만 접근 가능)
|
||||||
|
|
||||||
|
- **MongoDB**: `mongodb://mongodb:27017`
|
||||||
|
- **Redis**: `redis://redis:6379`
|
||||||
|
|
||||||
|
## 개발 워크플로우
|
||||||
|
|
||||||
|
### 1. 코드 변경 후 배포
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Docker 이미지 빌드
|
||||||
|
docker build -t yakenator/site11-console-backend:latest \
|
||||||
|
-f services/console/backend/Dockerfile \
|
||||||
|
services/console/backend
|
||||||
|
|
||||||
|
# 2. KIND 클러스터에 이미지 로드
|
||||||
|
kind load docker-image yakenator/site11-console-backend:latest --name site11-dev
|
||||||
|
|
||||||
|
# 3. Pod 재시작
|
||||||
|
kubectl rollout restart deployment/console-backend -n site11-console
|
||||||
|
|
||||||
|
# 4. 배포 상태 확인
|
||||||
|
kubectl rollout status deployment/console-backend -n site11-console
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. 스크립트로 간편하게
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 이미지 빌드 후 로드
|
||||||
|
./scripts/kind-setup.sh load-images
|
||||||
|
|
||||||
|
# 배포 재시작
|
||||||
|
kubectl rollout restart deployment/console-backend -n site11-console
|
||||||
|
kubectl rollout restart deployment/console-frontend -n site11-console
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. 전체 재배포
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 클러스터 삭제 후 재생성
|
||||||
|
./scripts/kind-setup.sh delete
|
||||||
|
./scripts/kind-setup.sh setup
|
||||||
|
```
|
||||||
|
|
||||||
|
## 모니터링
|
||||||
|
|
||||||
|
### docker-compose 모니터링 사용
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 모니터링 시작
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml up -d
|
||||||
|
|
||||||
|
# 실시간 로그 확인 (30초마다 업데이트)
|
||||||
|
docker-compose -f docker-compose.kubernetes.yml logs -f kind-monitor
|
||||||
|
```
|
||||||
|
|
||||||
|
모니터링 컨테이너는 다음 정보를 30초마다 출력합니다:
|
||||||
|
- 노드 상태
|
||||||
|
- Console 네임스페이스 Pod 상태
|
||||||
|
- Pipeline 네임스페이스 Pod 상태
|
||||||
|
|
||||||
|
## 문제 해결
|
||||||
|
|
||||||
|
### Pod이 시작되지 않는 경우
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Pod 상태 확인
|
||||||
|
kubectl get pods -n site11-console
|
||||||
|
|
||||||
|
# Pod 상세 정보 확인
|
||||||
|
kubectl describe pod <pod-name> -n site11-console
|
||||||
|
|
||||||
|
# Pod 로그 확인
|
||||||
|
kubectl logs <pod-name> -n site11-console
|
||||||
|
```
|
||||||
|
|
||||||
|
### 이미지 Pull 에러
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 로컬 이미지 확인
|
||||||
|
docker images | grep site11
|
||||||
|
|
||||||
|
# 이미지가 없으면 빌드
|
||||||
|
docker build -t yakenator/site11-console-backend:latest \
|
||||||
|
-f services/console/backend/Dockerfile \
|
||||||
|
services/console/backend
|
||||||
|
|
||||||
|
# KIND에 이미지 로드
|
||||||
|
kind load docker-image yakenator/site11-console-backend:latest --name site11-dev
|
||||||
|
```
|
||||||
|
|
||||||
|
### NodePort 접속 불가
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 서비스 확인
|
||||||
|
kubectl get svc -n site11-console
|
||||||
|
|
||||||
|
# NodePort 확인 (30080, 30081이어야 함)
|
||||||
|
kubectl describe svc console-frontend -n site11-console
|
||||||
|
kubectl describe svc console-backend -n site11-console
|
||||||
|
|
||||||
|
# 포트 포워딩 대안 (문제가 계속되면)
|
||||||
|
kubectl port-forward svc/console-frontend 3000:3000 -n site11-console
|
||||||
|
kubectl port-forward svc/console-backend 8000:8000 -n site11-console
|
||||||
|
```
|
||||||
|
|
||||||
|
### 클러스터 완전 초기화
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# KIND 클러스터 삭제
|
||||||
|
kind delete cluster --name site11-dev
|
||||||
|
|
||||||
|
# Docker 네트워크 정리 (필요시)
|
||||||
|
docker network prune -f
|
||||||
|
|
||||||
|
# 클러스터 재생성
|
||||||
|
./scripts/kind-setup.sh setup
|
||||||
|
```
|
||||||
|
|
||||||
|
### MongoDB 연결 실패
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# MongoDB Pod 확인
|
||||||
|
kubectl get pod -n site11-console -l app=mongodb
|
||||||
|
|
||||||
|
# MongoDB 로그 확인
|
||||||
|
kubectl logs -n site11-console -l app=mongodb
|
||||||
|
|
||||||
|
# MongoDB 서비스 확인
|
||||||
|
kubectl get svc mongodb -n site11-console
|
||||||
|
|
||||||
|
# Pod 내에서 연결 테스트
|
||||||
|
kubectl exec -it <console-backend-pod> -n site11-console -- \
|
||||||
|
curl mongodb:27017
|
||||||
|
```
|
||||||
|
|
||||||
|
## 참고 문서
|
||||||
|
|
||||||
|
- [KIND 공식 문서](https://kind.sigs.k8s.io/)
|
||||||
|
- [Kubernetes 공식 문서](https://kubernetes.io/docs/)
|
||||||
|
- [KIND 설정 가이드](./docs/KIND_SETUP.md)
|
||||||
|
|
||||||
|
## 유용한 팁
|
||||||
|
|
||||||
|
### kubectl 자동완성 설정
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Bash
|
||||||
|
echo 'source <(kubectl completion bash)' >>~/.bashrc
|
||||||
|
|
||||||
|
# Zsh
|
||||||
|
echo 'source <(kubectl completion zsh)' >>~/.zshrc
|
||||||
|
```
|
||||||
|
|
||||||
|
### kubectl 단축어 설정
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# ~/.bashrc 또는 ~/.zshrc에 추가
|
||||||
|
alias k='kubectl'
|
||||||
|
alias kgp='kubectl get pods'
|
||||||
|
alias kgs='kubectl get svc'
|
||||||
|
alias kgn='kubectl get nodes'
|
||||||
|
alias kl='kubectl logs'
|
||||||
|
alias kd='kubectl describe'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Context 빠른 전환
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 현재 context 확인
|
||||||
|
kubectl config current-context
|
||||||
|
|
||||||
|
# KIND context로 전환
|
||||||
|
kubectl config use-context kind-site11-dev
|
||||||
|
|
||||||
|
# 기본 namespace 설정
|
||||||
|
kubectl config set-context --current --namespace=site11-console
|
||||||
|
```
|
||||||
530
PRESENTATION.md
Normal file
530
PRESENTATION.md
Normal file
@ -0,0 +1,530 @@
|
|||||||
|
# Site11 Platform - Architecture Presentation
|
||||||
|
|
||||||
|
## Slide 1: Title
|
||||||
|
```
|
||||||
|
╔═══════════════════════════════════════════════════════════════╗
|
||||||
|
║ ║
|
||||||
|
║ SITE11 PLATFORM ║
|
||||||
|
║ ║
|
||||||
|
║ AI-Powered Multi-Language Content System ║
|
||||||
|
║ ║
|
||||||
|
║ Microservices Architecture Overview ║
|
||||||
|
║ ║
|
||||||
|
║ September 2025 ║
|
||||||
|
║ ║
|
||||||
|
╚═══════════════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Slide 2: Executive Summary
|
||||||
|
```
|
||||||
|
╔═══════════════════════════════════════════════════════════════╗
|
||||||
|
║ WHAT IS SITE11? ║
|
||||||
|
╠═══════════════════════════════════════════════════════════════╣
|
||||||
|
║ ║
|
||||||
|
║ 🚀 Automated Content Generation Platform ║
|
||||||
|
║ ║
|
||||||
|
║ 🌍 8+ Languages Support ║
|
||||||
|
║ (Korean, English, Chinese, Japanese, French, ║
|
||||||
|
║ German, Spanish, Italian) ║
|
||||||
|
║ ║
|
||||||
|
║ 🤖 AI-Powered with Claude API ║
|
||||||
|
║ ║
|
||||||
|
║ 📊 30+ Specialized Microservices ║
|
||||||
|
║ ║
|
||||||
|
║ ⚡ Real-time Event-Driven Architecture ║
|
||||||
|
║ ║
|
||||||
|
║ 📈 1000+ Articles/Day Capacity ║
|
||||||
|
║ ║
|
||||||
|
╚═══════════════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Slide 3: System Architecture Overview
|
||||||
|
```
|
||||||
|
╔═══════════════════════════════════════════════════════════════╗
|
||||||
|
║ ARCHITECTURE OVERVIEW ║
|
||||||
|
╠═══════════════════════════════════════════════════════════════╣
|
||||||
|
║ ║
|
||||||
|
║ ┌─────────────────────────────────────────────┐ ║
|
||||||
|
║ │ Client Layer │ ║
|
||||||
|
║ └────────────────┬──────────────────────────────┘ ║
|
||||||
|
║ │ ║
|
||||||
|
║ ┌────────────────▼──────────────────────────────┐ ║
|
||||||
|
║ │ API Gateway (Console) │ ║
|
||||||
|
║ │ - Authentication │ ║
|
||||||
|
║ │ - Routing │ ║
|
||||||
|
║ │ - Monitoring │ ║
|
||||||
|
║ └────────────────┬──────────────────────────────┘ ║
|
||||||
|
║ │ ║
|
||||||
|
║ ┌────────────────▼──────────────────────────────┐ ║
|
||||||
|
║ │ Microservices Layer │ ║
|
||||||
|
║ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ ║
|
||||||
|
║ │ │ Core │ │ Pipeline │ │ Domain │ │ ║
|
||||||
|
║ │ │ Services │ │ Services │ │ Services │ │ ║
|
||||||
|
║ │ └──────────┘ └──────────┘ └──────────┘ │ ║
|
||||||
|
║ └────────────────┬──────────────────────────────┘ ║
|
||||||
|
║ │ ║
|
||||||
|
║ ┌────────────────▼──────────────────────────────┐ ║
|
||||||
|
║ │ Infrastructure Layer │ ║
|
||||||
|
║ │ MongoDB │ Redis │ Kafka │ MinIO │ Solr │ ║
|
||||||
|
║ └───────────────────────────────────────────────┘ ║
|
||||||
|
║ ║
|
||||||
|
╚═══════════════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Slide 4: Technology Stack
|
||||||
|
```
|
||||||
|
╔═══════════════════════════════════════════════════════════════╗
|
||||||
|
║ TECHNOLOGY STACK ║
|
||||||
|
╠═══════════════════════════════════════════════════════════════╣
|
||||||
|
║ ║
|
||||||
|
║ Backend Framework: ║
|
||||||
|
║ ├─ FastAPI (Python 3.11) ║
|
||||||
|
║ └─ Async/await for high performance ║
|
||||||
|
║ ║
|
||||||
|
║ Frontend: ║
|
||||||
|
║ ├─ React 18 + TypeScript ║
|
||||||
|
║ └─ Vite + Material-UI ║
|
||||||
|
║ ║
|
||||||
|
║ Data Layer: ║
|
||||||
|
║ ├─ MongoDB 7.0 (Primary Database) ║
|
||||||
|
║ ├─ Redis 7 (Cache & Queue) ║
|
||||||
|
║ └─ MinIO (Object Storage) ║
|
||||||
|
║ ║
|
||||||
|
║ Messaging: ║
|
||||||
|
║ ├─ Apache Kafka (Event Streaming) ║
|
||||||
|
║ └─ Redis Pub/Sub (Real-time) ║
|
||||||
|
║ ║
|
||||||
|
║ Infrastructure: ║
|
||||||
|
║ ├─ Docker & Docker Compose ║
|
||||||
|
║ └─ Kubernetes (Production) ║
|
||||||
|
║ ║
|
||||||
|
╚═══════════════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Slide 5: Content Pipeline Architecture
|
||||||
|
```
|
||||||
|
╔═══════════════════════════════════════════════════════════════╗
|
||||||
|
║ AUTOMATED CONTENT PIPELINE ║
|
||||||
|
╠═══════════════════════════════════════════════════════════════╣
|
||||||
|
║ ║
|
||||||
|
║ [Scheduler] ║
|
||||||
|
║ ↓ ║
|
||||||
|
║ ┌───────────────────────────────────────────┐ ║
|
||||||
|
║ │ 1. Content Discovery │ ║
|
||||||
|
║ │ [RSS Feeds] + [Google Search API] │ ║
|
||||||
|
║ └───────────────────┬───────────────────────┘ ║
|
||||||
|
║ ↓ ║
|
||||||
|
║ ┌───────────────────────────────────────────┐ ║
|
||||||
|
║ │ 2. AI Content Generation │ ║
|
||||||
|
║ │ [Claude API Integration] │ ║
|
||||||
|
║ └───────────────────┬───────────────────────┘ ║
|
||||||
|
║ ↓ ║
|
||||||
|
║ ┌───────────────────────────────────────────┐ ║
|
||||||
|
║ │ 3. Multi-Language Translation │ ║
|
||||||
|
║ │ [DeepL API - 8 Languages] │ ║
|
||||||
|
║ └───────────────────┬───────────────────────┘ ║
|
||||||
|
║ ↓ ║
|
||||||
|
║ ┌───────────────────────────────────────────┐ ║
|
||||||
|
║ │ 4. Image Generation │ ║
|
||||||
|
║ │ [AI Image Generation Service] │ ║
|
||||||
|
║ └───────────────────┬───────────────────────┘ ║
|
||||||
|
║ ↓ ║
|
||||||
|
║ [MongoDB Storage] → [Distribution] ║
|
||||||
|
║ ║
|
||||||
|
╚═══════════════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Slide 6: Microservices Breakdown
|
||||||
|
```
|
||||||
|
╔═══════════════════════════════════════════════════════════════╗
|
||||||
|
║ MICROSERVICES ECOSYSTEM ║
|
||||||
|
╠═══════════════════════════════════════════════════════════════╣
|
||||||
|
║ ║
|
||||||
|
║ Core Services (10) Pipeline Services (9) ║
|
||||||
|
║ ├─ Console (8000) ├─ Scheduler ║
|
||||||
|
║ ├─ Users (8007) ├─ RSS Collector ║
|
||||||
|
║ ├─ OAuth (8003) ├─ Google Search ║
|
||||||
|
║ ├─ Images (8001) ├─ AI Generator ║
|
||||||
|
║ ├─ Files (8014) ├─ Translator ║
|
||||||
|
║ ├─ Notifications (8013) ├─ Image Generator ║
|
||||||
|
║ ├─ Search (8015) ├─ Language Sync ║
|
||||||
|
║ ├─ Statistics (8012) ├─ Keyword Manager ║
|
||||||
|
║ ├─ News Aggregator (8018) └─ Monitor (8100) ║
|
||||||
|
║ └─ AI Writer (8019) ║
|
||||||
|
║ ║
|
||||||
|
║ Domain Services (15+) ║
|
||||||
|
║ ├─ Entertainment: blackpink, nct, twice, k-culture ║
|
||||||
|
║ ├─ Regional: korea, japan, china, usa ║
|
||||||
|
║ ├─ Technology: ai, crypto, apple, google, samsung ║
|
||||||
|
║ └─ Business: wsj, musk ║
|
||||||
|
║ ║
|
||||||
|
║ Total: 30+ Microservices ║
|
||||||
|
║ ║
|
||||||
|
╚═══════════════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Slide 7: Data Flow
|
||||||
|
```
|
||||||
|
╔═══════════════════════════════════════════════════════════════╗
|
||||||
|
║ DATA FLOW ║
|
||||||
|
╠═══════════════════════════════════════════════════════════════╣
|
||||||
|
║ ║
|
||||||
|
║ Request Flow: ║
|
||||||
|
║ ───────────── ║
|
||||||
|
║ Client → Console Gateway → Service → Database ║
|
||||||
|
║ ↓ ↓ ↓ ║
|
||||||
|
║ Cache Event Response ║
|
||||||
|
║ ↓ ║
|
||||||
|
║ Kafka Topic ║
|
||||||
|
║ ↓ ║
|
||||||
|
║ Other Services ║
|
||||||
|
║ ║
|
||||||
|
║ Event Flow: ║
|
||||||
|
║ ──────────── ║
|
||||||
|
║ Service A ──[Publish]──> Kafka ──[Subscribe]──> Service B ║
|
||||||
|
║ ↓ ║
|
||||||
|
║ Service C, D, E ║
|
||||||
|
║ ║
|
||||||
|
║ Cache Strategy: ║
|
||||||
|
║ ─────────────── ║
|
||||||
|
║ Request → Redis Cache → Hit? → Return ║
|
||||||
|
║ ↓ ║
|
||||||
|
║ Miss → Service → MongoDB → Update Cache ║
|
||||||
|
║ ║
|
||||||
|
╚═══════════════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Slide 8: Deployment Architecture
|
||||||
|
```
|
||||||
|
╔═══════════════════════════════════════════════════════════════╗
|
||||||
|
║ DEPLOYMENT ARCHITECTURE ║
|
||||||
|
╠═══════════════════════════════════════════════════════════════╣
|
||||||
|
║ ║
|
||||||
|
║ Development Environment: ║
|
||||||
|
║ ┌──────────────────────────────────────────┐ ║
|
||||||
|
║ │ Docker Compose │ ║
|
||||||
|
║ │ - Single YAML configuration │ ║
|
||||||
|
║ │ - Hot reload for development │ ║
|
||||||
|
║ │ - Local volumes for persistence │ ║
|
||||||
|
║ └──────────────────────────────────────────┘ ║
|
||||||
|
║ ║
|
||||||
|
║ Production Environment: ║
|
||||||
|
║ ┌──────────────────────────────────────────┐ ║
|
||||||
|
║ │ Kubernetes Cluster │ ║
|
||||||
|
║ │ │ ║
|
||||||
|
║ │ Namespaces: │ ║
|
||||||
|
║ │ ├─ site11-core (infrastructure) │ ║
|
||||||
|
║ │ ├─ site11-pipeline (processing) │ ║
|
||||||
|
║ │ └─ site11-services (applications) │ ║
|
||||||
|
║ │ │ ║
|
||||||
|
║ │ Features: │ ║
|
||||||
|
║ │ ├─ Auto-scaling (HPA) │ ║
|
||||||
|
║ │ ├─ Load balancing │ ║
|
||||||
|
║ │ └─ Rolling updates │ ║
|
||||||
|
║ └──────────────────────────────────────────┘ ║
|
||||||
|
║ ║
|
||||||
|
╚═══════════════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Slide 9: Key Features & Capabilities
|
||||||
|
```
|
||||||
|
╔═══════════════════════════════════════════════════════════════╗
|
||||||
|
║ KEY FEATURES ║
|
||||||
|
╠═══════════════════════════════════════════════════════════════╣
|
||||||
|
║ ║
|
||||||
|
║ 🔄 Automated Operation ║
|
||||||
|
║ • 24/7 content generation ║
|
||||||
|
║ • No human intervention required ║
|
||||||
|
║ • Self-healing with retries ║
|
||||||
|
║ ║
|
||||||
|
║ 🌐 Multi-Language Excellence ║
|
||||||
|
║ • Simultaneous 8-language translation ║
|
||||||
|
║ • Cultural adaptation per market ║
|
||||||
|
║ • Consistent quality across languages ║
|
||||||
|
║ ║
|
||||||
|
║ ⚡ Performance ║
|
||||||
|
║ • 1000+ articles per day ║
|
||||||
|
║ • <100ms API response (p50) ║
|
||||||
|
║ • 100+ queue jobs per minute ║
|
||||||
|
║ ║
|
||||||
|
║ 📈 Scalability ║
|
||||||
|
║ • Horizontal scaling (10+ replicas) ║
|
||||||
|
║ • Vertical scaling (up to 4GB/4CPU) ║
|
||||||
|
║ • Multi-region ready ║
|
||||||
|
║ ║
|
||||||
|
╚═══════════════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Slide 10: Security & Monitoring
|
||||||
|
```
|
||||||
|
╔═══════════════════════════════════════════════════════════════╗
|
||||||
|
║ SECURITY & MONITORING ║
|
||||||
|
╠═══════════════════════════════════════════════════════════════╣
|
||||||
|
║ ║
|
||||||
|
║ Security Measures: ║
|
||||||
|
║ ├─ JWT Authentication ║
|
||||||
|
║ ├─ Service-to-Service Auth ║
|
||||||
|
║ ├─ Rate Limiting ║
|
||||||
|
║ ├─ CORS Configuration ║
|
||||||
|
║ ├─ Network Isolation ║
|
||||||
|
║ └─ Secrets Management (.env) ║
|
||||||
|
║ ║
|
||||||
|
║ Monitoring Stack: ║
|
||||||
|
║ ├─ Health Checks (/health endpoints) ║
|
||||||
|
║ ├─ Pipeline Monitor Dashboard (8100) ║
|
||||||
|
║ ├─ Real-time Queue Monitoring ║
|
||||||
|
║ ├─ Service Status Dashboard ║
|
||||||
|
║ └─ Structured JSON Logging ║
|
||||||
|
║ ║
|
||||||
|
║ Observability: ║
|
||||||
|
║ ├─ Correlation IDs for tracing ║
|
||||||
|
║ ├─ Metrics collection ║
|
||||||
|
║ └─ Error tracking and alerting ║
|
||||||
|
║ ║
|
||||||
|
╚═══════════════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Slide 11: Performance Metrics
|
||||||
|
```
|
||||||
|
╔═══════════════════════════════════════════════════════════════╗
|
||||||
|
║ PERFORMANCE METRICS ║
|
||||||
|
╠═══════════════════════════════════════════════════════════════╣
|
||||||
|
║ ║
|
||||||
|
║ Current Capacity (Single Instance): ║
|
||||||
|
║ ║
|
||||||
|
║ ┌────────────────────────────────────────────────┐ ║
|
||||||
|
║ │ Content Generation: 1000+ articles/day │ ║
|
||||||
|
║ │ Translation Speed: 8 languages parallel │ ║
|
||||||
|
║ │ API Response: <100ms (p50) │ ║
|
||||||
|
║ │ <500ms (p99) │ ║
|
||||||
|
║ │ Queue Processing: 100+ jobs/minute │ ║
|
||||||
|
║ │ Storage Capacity: Scalable to TBs │ ║
|
||||||
|
║ │ Concurrent Users: 10,000+ │ ║
|
||||||
|
║ └────────────────────────────────────────────────┘ ║
|
||||||
|
║ ║
|
||||||
|
║ Resource Utilization: ║
|
||||||
|
║ ┌────────────────────────────────────────────────┐ ║
|
||||||
|
║ │ CPU: 100m-500m request, 1000m limit │ ║
|
||||||
|
║ │ Memory: 128Mi-512Mi request, 1Gi limit │ ║
|
||||||
|
║ │ Storage: 1Gi-10Gi per service │ ║
|
||||||
|
║ └────────────────────────────────────────────────┘ ║
|
||||||
|
║ ║
|
||||||
|
╚═══════════════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Slide 12: Development Workflow
|
||||||
|
```
|
||||||
|
╔═══════════════════════════════════════════════════════════════╗
|
||||||
|
║ DEVELOPMENT WORKFLOW ║
|
||||||
|
╠═══════════════════════════════════════════════════════════════╣
|
||||||
|
║ ║
|
||||||
|
║ Local Development: ║
|
||||||
|
║ ───────────────── ║
|
||||||
|
║ $ docker-compose up -d # Start all services ║
|
||||||
|
║ $ docker-compose logs -f [svc] # View logs ║
|
||||||
|
║ $ docker-compose build [svc] # Rebuild service ║
|
||||||
|
║ ║
|
||||||
|
║ Testing: ║
|
||||||
|
║ ──────── ║
|
||||||
|
║ $ docker-compose exec [svc] pytest # Unit tests ║
|
||||||
|
║ $ docker-compose exec [svc] pytest # Integration ║
|
||||||
|
║ tests/integration ║
|
||||||
|
║ ║
|
||||||
|
║ Deployment: ║
|
||||||
|
║ ──────────── ║
|
||||||
|
║ Development: ./deploy-local.sh ║
|
||||||
|
║ Staging: ./deploy-kind.sh ║
|
||||||
|
║ Production: ./deploy-k8s.sh ║
|
||||||
|
║ Docker Hub: ./deploy-dockerhub.sh ║
|
||||||
|
║ ║
|
||||||
|
║ CI/CD Pipeline: ║
|
||||||
|
║ ─────────────── ║
|
||||||
|
║ Git Push → Build → Test → Deploy → Monitor ║
|
||||||
|
║ ║
|
||||||
|
╚═══════════════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Slide 13: Business Impact
|
||||||
|
```
|
||||||
|
╔═══════════════════════════════════════════════════════════════╗
|
||||||
|
║ BUSINESS IMPACT ║
|
||||||
|
╠═══════════════════════════════════════════════════════════════╣
|
||||||
|
║ ║
|
||||||
|
║ Cost Efficiency: ║
|
||||||
|
║ ├─ 90% reduction in content creation costs ║
|
||||||
|
║ ├─ Automated 24/7 operation ║
|
||||||
|
║ └─ No manual translation needed ║
|
||||||
|
║ ║
|
||||||
|
║ Market Reach: ║
|
||||||
|
║ ├─ 8+ language markets simultaneously ║
|
||||||
|
║ ├─ Real-time trend coverage ║
|
||||||
|
║ └─ Domain-specific content targeting ║
|
||||||
|
║ ║
|
||||||
|
║ Scalability: ║
|
||||||
|
║ ├─ From 100 to 10,000+ articles/day ║
|
||||||
|
║ ├─ Linear cost scaling ║
|
||||||
|
║ └─ Global deployment ready ║
|
||||||
|
║ ║
|
||||||
|
║ Time to Market: ║
|
||||||
|
║ ├─ Minutes from news to article ║
|
||||||
|
║ ├─ Instant multi-language deployment ║
|
||||||
|
║ └─ Real-time content updates ║
|
||||||
|
║ ║
|
||||||
|
╚═══════════════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Slide 14: Future Roadmap
|
||||||
|
```
|
||||||
|
╔═══════════════════════════════════════════════════════════════╗
|
||||||
|
║ FUTURE ROADMAP ║
|
||||||
|
╠═══════════════════════════════════════════════════════════════╣
|
||||||
|
║ ║
|
||||||
|
║ Q1 2025: Enhanced Observability ║
|
||||||
|
║ ├─ Prometheus + Grafana ║
|
||||||
|
║ ├─ Distributed tracing (Jaeger) ║
|
||||||
|
║ └─ ELK Stack integration ║
|
||||||
|
║ ║
|
||||||
|
║ Q2 2025: Advanced Features ║
|
||||||
|
║ ├─ Machine Learning pipeline ║
|
||||||
|
║ ├─ Real-time analytics ║
|
||||||
|
║ ├─ GraphQL API layer ║
|
||||||
|
║ └─ WebSocket support ║
|
||||||
|
║ ║
|
||||||
|
║ Q3 2025: Enterprise Features ║
|
||||||
|
║ ├─ Multi-tenancy support ║
|
||||||
|
║ ├─ Advanced RBAC ║
|
||||||
|
║ ├─ Audit logging ║
|
||||||
|
║ └─ Compliance features ║
|
||||||
|
║ ║
|
||||||
|
║ Q4 2025: Global Expansion ║
|
||||||
|
║ ├─ Multi-region deployment ║
|
||||||
|
║ ├─ CDN integration ║
|
||||||
|
║ └─ Edge computing ║
|
||||||
|
║ ║
|
||||||
|
╚═══════════════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Slide 15: Conclusion
|
||||||
|
```
|
||||||
|
╔═══════════════════════════════════════════════════════════════╗
|
||||||
|
║ CONCLUSION ║
|
||||||
|
╠═══════════════════════════════════════════════════════════════╣
|
||||||
|
║ ║
|
||||||
|
║ Site11: Next-Gen Content Platform ║
|
||||||
|
║ ║
|
||||||
|
║ ✅ Proven Architecture ║
|
||||||
|
║ • 30+ microservices in production ║
|
||||||
|
║ • 1000+ articles/day capacity ║
|
||||||
|
║ • 8 language support ║
|
||||||
|
║ ║
|
||||||
|
║ ✅ Modern Technology Stack ║
|
||||||
|
║ • Cloud-native design ║
|
||||||
|
║ • AI-powered automation ║
|
||||||
|
║ • Event-driven architecture ║
|
||||||
|
║ ║
|
||||||
|
║ ✅ Business Ready ║
|
||||||
|
║ • Cost-effective operation ║
|
||||||
|
║ • Scalable to enterprise needs ║
|
||||||
|
║ • Global market reach ║
|
||||||
|
║ ║
|
||||||
|
║ 🚀 Ready for the Future ║
|
||||||
|
║ • Continuous innovation ║
|
||||||
|
║ • Adaptable architecture ║
|
||||||
|
║ • Growing ecosystem ║
|
||||||
|
║ ║
|
||||||
|
╚═══════════════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Slide 16: Q&A
|
||||||
|
```
|
||||||
|
╔═══════════════════════════════════════════════════════════════╗
|
||||||
|
║ ║
|
||||||
|
║ ║
|
||||||
|
║ QUESTIONS & ANSWERS ║
|
||||||
|
║ ║
|
||||||
|
║ ║
|
||||||
|
║ Thank You! ║
|
||||||
|
║ ║
|
||||||
|
║ ║
|
||||||
|
║ Contact Information: ║
|
||||||
|
║ architecture@site11.com ║
|
||||||
|
║ ║
|
||||||
|
║ ║
|
||||||
|
║ GitHub: github.com/site11 ║
|
||||||
|
║ Docs: docs.site11.com ║
|
||||||
|
║ ║
|
||||||
|
║ ║
|
||||||
|
╚═══════════════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix: Quick Reference
|
||||||
|
|
||||||
|
### Demo Commands
|
||||||
|
```bash
|
||||||
|
# Show live pipeline monitoring
|
||||||
|
open http://localhost:8100
|
||||||
|
|
||||||
|
# Check service health
|
||||||
|
curl http://localhost:8000/health
|
||||||
|
|
||||||
|
# View real-time logs
|
||||||
|
docker-compose logs -f pipeline-scheduler
|
||||||
|
|
||||||
|
# Show article generation
|
||||||
|
docker exec site11_mongodb mongosh ai_writer_db --eval "db.articles_ko.find().limit(1).pretty()"
|
||||||
|
|
||||||
|
# Check translation status
|
||||||
|
docker exec site11_mongodb mongosh ai_writer_db --eval "db.articles_en.countDocuments()"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Key Metrics for Demo
|
||||||
|
- Services Running: 30+
|
||||||
|
- Articles Generated Today: Check MongoDB
|
||||||
|
- Languages Supported: 8
|
||||||
|
- Queue Processing Rate: Check Redis
|
||||||
|
- API Response Time: <100ms
|
||||||
|
|
||||||
|
### Architecture Highlights
|
||||||
|
1. **Microservices**: Independent scaling and deployment
|
||||||
|
2. **Event-Driven**: Real-time processing with Kafka
|
||||||
|
3. **AI-Powered**: Claude API for content generation
|
||||||
|
4. **Multi-Language**: DeepL for translations
|
||||||
|
5. **Cloud-Native**: Docker/Kubernetes ready
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Presentation Version**: 1.0
|
||||||
|
**Platform**: Site11 v1.0
|
||||||
|
**Date**: September 2025
|
||||||
128
README.md
128
README.md
@ -33,21 +33,48 @@ Site11은 다국어 뉴스 콘텐츠를 자동으로 수집, 번역, 생성하
|
|||||||
- **OS**: Linux, macOS, Windows with WSL2
|
- **OS**: Linux, macOS, Windows with WSL2
|
||||||
|
|
||||||
### 포트 사용
|
### 포트 사용
|
||||||
|
|
||||||
|
#### 하이브리드 배포 포트 구성 (현재 구성)
|
||||||
|
```
|
||||||
|
[ Docker Compose - 인프라 서비스 ]
|
||||||
|
- 27017: MongoDB (내부)
|
||||||
|
- 6379: Redis (내부)
|
||||||
|
- 9092: Kafka (내부)
|
||||||
|
- 2181: Zookeeper (내부)
|
||||||
|
- 5555: Docker Registry (내부)
|
||||||
|
- 8099: Pipeline Scheduler
|
||||||
|
- 8100: Pipeline Monitor
|
||||||
|
|
||||||
|
[ Kubernetes - 마이크로서비스 ]
|
||||||
|
- 8080: Console Frontend (kubectl port-forward → Service:3000 → Pod:80)
|
||||||
|
- 8000: Console Backend (kubectl port-forward → Service:8000 → Pod:8000)
|
||||||
|
- 30801-30802: Images Service (→ 8001-8002)
|
||||||
|
- 30803-30804: OAuth Service (→ 8003-8004)
|
||||||
|
- 30805-30806: Applications Service (→ 8005-8006)
|
||||||
|
- 30807-30808: Users Service (→ 8007-8008)
|
||||||
|
- 30809-30810: Data Service (→ 8009-8010)
|
||||||
|
- 30811-30812: Statistics Service (→ 8011-8012)
|
||||||
|
|
||||||
|
[ Pipeline Workers (K8s 내부) ]
|
||||||
|
- RSS Collector
|
||||||
|
- Google Search
|
||||||
|
- Translator
|
||||||
|
- AI Article Generator
|
||||||
|
- Image Generator
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 표준 Docker Compose 포트 구성 (전체 Docker 모드)
|
||||||
```
|
```
|
||||||
- 3000: Console Frontend
|
- 3000: Console Frontend
|
||||||
- 8011: Console Backend (API Gateway)
|
- 8000: Console Backend (API Gateway)
|
||||||
- 8012: Users Backend
|
- 8001-8002: Images Service
|
||||||
- 8013: Notifications Backend
|
- 8003-8004: OAuth Service
|
||||||
- 8014: OAuth Backend
|
- 8005-8006: Applications Service
|
||||||
- 8015: Images Backend
|
- 8007-8008: Users Service
|
||||||
- 8016: Google Search Backend
|
- 8009-8010: Data Service
|
||||||
- 8017: RSS Feed Backend
|
- 8011-8012: Statistics Service
|
||||||
- 8018: News Aggregator Backend
|
- 8099: Pipeline Scheduler
|
||||||
- 8019: AI Writer Backend
|
|
||||||
- 8100: Pipeline Monitor
|
- 8100: Pipeline Monitor
|
||||||
- 8983: Solr Search Engine
|
|
||||||
- 9000: MinIO Object Storage
|
|
||||||
- 9001: MinIO Console
|
|
||||||
- 27017: MongoDB (내부)
|
- 27017: MongoDB (내부)
|
||||||
- 6379: Redis (내부)
|
- 6379: Redis (내부)
|
||||||
- 9092: Kafka (내부)
|
- 9092: Kafka (내부)
|
||||||
@ -92,18 +119,44 @@ docker-compose logs -f
|
|||||||
```
|
```
|
||||||
|
|
||||||
### 4. 서비스 확인
|
### 4. 서비스 확인
|
||||||
|
|
||||||
|
#### 하이브리드 배포 확인 (현재 구성)
|
||||||
```bash
|
```bash
|
||||||
|
# Console Frontend 접속 (kubectl port-forward)
|
||||||
|
open http://localhost:8080
|
||||||
|
|
||||||
|
# Console API 헬스 체크 (kubectl port-forward)
|
||||||
|
curl http://localhost:8000/health
|
||||||
|
curl http://localhost:8000/api/health
|
||||||
|
|
||||||
|
# Port forwarding 시작 (필요시)
|
||||||
|
kubectl -n site11-pipeline port-forward service/console-frontend 8080:3000 &
|
||||||
|
kubectl -n site11-pipeline port-forward service/console-backend 8000:8000 &
|
||||||
|
|
||||||
|
# Pipeline 모니터 확인 (Docker)
|
||||||
|
curl http://localhost:8100/health
|
||||||
|
|
||||||
|
# MongoDB 연결 확인 (Docker)
|
||||||
|
docker exec -it site11_mongodb mongosh --eval "db.serverStatus()"
|
||||||
|
|
||||||
|
# K8s Pod 상태 확인
|
||||||
|
kubectl -n site11-pipeline get pods
|
||||||
|
kubectl -n site11-pipeline get services
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 표준 Docker 확인
|
||||||
|
```bash
|
||||||
|
# Console Frontend 접속
|
||||||
|
open http://localhost:3000
|
||||||
|
|
||||||
# Console API 헬스 체크
|
# Console API 헬스 체크
|
||||||
curl http://localhost:8011/health
|
curl http://localhost:8000/health
|
||||||
|
|
||||||
# MongoDB 연결 확인
|
# MongoDB 연결 확인
|
||||||
docker exec -it site11_mongodb mongosh --eval "db.serverStatus()"
|
docker exec -it site11_mongodb mongosh --eval "db.serverStatus()"
|
||||||
|
|
||||||
# Pipeline 모니터 확인
|
# Pipeline 모니터 확인
|
||||||
curl http://localhost:8100/health
|
curl http://localhost:8100/health
|
||||||
|
|
||||||
# Console UI 접속
|
|
||||||
open http://localhost:3000
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## 상세 설치 가이드 (Detailed Installation)
|
## 상세 설치 가이드 (Detailed Installation)
|
||||||
@ -464,12 +517,14 @@ Site11은 Docker Compose와 Kubernetes를 함께 사용하는 하이브리드
|
|||||||
|
|
||||||
### 배포 아키텍처
|
### 배포 아키텍처
|
||||||
|
|
||||||
#### Docker Compose (인프라 및 중앙 제어)
|
#### Docker Compose (인프라 서비스)
|
||||||
- **인프라**: MongoDB, Redis, Kafka, Zookeeper
|
- **인프라**: MongoDB, Redis, Kafka, Zookeeper
|
||||||
- **중앙 제어**: Pipeline Scheduler, Pipeline Monitor, Language Sync
|
- **중앙 제어**: Pipeline Scheduler, Pipeline Monitor, Language Sync
|
||||||
- **관리 콘솔**: Console Backend/Frontend
|
- **레지스트리**: Docker Registry (port 5555)
|
||||||
|
|
||||||
#### Kubernetes (무상태 워커)
|
#### Kubernetes (애플리케이션 및 파이프라인)
|
||||||
|
- **관리 콘솔**: Console Backend/Frontend
|
||||||
|
- **마이크로서비스**: Images, OAuth, Applications, Users, Data, Statistics
|
||||||
- **데이터 수집**: RSS Collector, Google Search
|
- **데이터 수집**: RSS Collector, Google Search
|
||||||
- **처리 워커**: Translator, AI Article Generator, Image Generator
|
- **처리 워커**: Translator, AI Article Generator, Image Generator
|
||||||
- **자동 스케일링**: HPA(Horizontal Pod Autoscaler) 적용
|
- **자동 스케일링**: HPA(Horizontal Pod Autoscaler) 적용
|
||||||
@ -488,15 +543,40 @@ docker-compose -f docker-compose-hybrid.yml ps
|
|||||||
docker-compose -f docker-compose-hybrid.yml logs -f pipeline-scheduler
|
docker-compose -f docker-compose-hybrid.yml logs -f pipeline-scheduler
|
||||||
```
|
```
|
||||||
|
|
||||||
#### 2. K8s 워커 배포
|
#### 2. Docker Hub를 사용한 K8s 배포 (권장)
|
||||||
```bash
|
```bash
|
||||||
# K8s 매니페스트 디렉토리로 이동
|
# Docker Hub에 이미지 푸시
|
||||||
|
./deploy-dockerhub.sh
|
||||||
|
|
||||||
|
# K8s 네임스페이스 및 설정 생성
|
||||||
|
kubectl create namespace site11-pipeline
|
||||||
|
kubectl -n site11-pipeline apply -f k8s/pipeline/configmap.yaml
|
||||||
|
kubectl -n site11-pipeline apply -f k8s/pipeline/secrets.yaml
|
||||||
|
|
||||||
|
# Docker Hub 이미지로 배포
|
||||||
cd k8s/pipeline
|
cd k8s/pipeline
|
||||||
|
for service in console-backend console-frontend \
|
||||||
|
ai-article-generator translator image-generator \
|
||||||
|
rss-collector google-search; do
|
||||||
|
kubectl apply -f ${service}-dockerhub.yaml
|
||||||
|
done
|
||||||
|
|
||||||
# API 키 설정 (configmap.yaml 편집)
|
# 배포 상태 확인
|
||||||
vim configmap.yaml
|
kubectl -n site11-pipeline get pods
|
||||||
|
kubectl -n site11-pipeline get services
|
||||||
|
kubectl -n site11-pipeline get hpa
|
||||||
|
```
|
||||||
|
|
||||||
# 배포 실행
|
#### 3. 로컬 레지스트리를 사용한 K8s 배포 (대안)
|
||||||
|
```bash
|
||||||
|
# 로컬 레지스트리 시작 (Docker Compose)
|
||||||
|
docker-compose -f docker-compose-hybrid.yml up -d registry
|
||||||
|
|
||||||
|
# 이미지 빌드 및 푸시
|
||||||
|
./deploy-local.sh
|
||||||
|
|
||||||
|
# K8s 배포
|
||||||
|
cd k8s/pipeline
|
||||||
./deploy.sh
|
./deploy.sh
|
||||||
|
|
||||||
# 배포 상태 확인
|
# 배포 상태 확인
|
||||||
|
|||||||
@ -1,19 +0,0 @@
|
|||||||
import { Routes, Route } from 'react-router-dom'
|
|
||||||
import Layout from './components/Layout'
|
|
||||||
import Dashboard from './pages/Dashboard'
|
|
||||||
import Services from './pages/Services'
|
|
||||||
import Users from './pages/Users'
|
|
||||||
|
|
||||||
function App() {
|
|
||||||
return (
|
|
||||||
<Routes>
|
|
||||||
<Route path="/" element={<Layout />}>
|
|
||||||
<Route index element={<Dashboard />} />
|
|
||||||
<Route path="services" element={<Services />} />
|
|
||||||
<Route path="users" element={<Users />} />
|
|
||||||
</Route>
|
|
||||||
</Routes>
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
export default App
|
|
||||||
@ -1,98 +0,0 @@
|
|||||||
import {
|
|
||||||
Box,
|
|
||||||
Typography,
|
|
||||||
Table,
|
|
||||||
TableBody,
|
|
||||||
TableCell,
|
|
||||||
TableContainer,
|
|
||||||
TableHead,
|
|
||||||
TableRow,
|
|
||||||
Paper,
|
|
||||||
Chip,
|
|
||||||
} from '@mui/material'
|
|
||||||
|
|
||||||
const servicesData = [
|
|
||||||
{
|
|
||||||
id: 1,
|
|
||||||
name: 'Console',
|
|
||||||
type: 'API Gateway',
|
|
||||||
port: 8011,
|
|
||||||
status: 'Running',
|
|
||||||
description: 'Central orchestrator and API gateway',
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: 2,
|
|
||||||
name: 'Users',
|
|
||||||
type: 'Microservice',
|
|
||||||
port: 8001,
|
|
||||||
status: 'Running',
|
|
||||||
description: 'User management service',
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: 3,
|
|
||||||
name: 'MongoDB',
|
|
||||||
type: 'Database',
|
|
||||||
port: 27017,
|
|
||||||
status: 'Running',
|
|
||||||
description: 'Document database for persistence',
|
|
||||||
},
|
|
||||||
{
|
|
||||||
id: 4,
|
|
||||||
name: 'Redis',
|
|
||||||
type: 'Cache',
|
|
||||||
port: 6379,
|
|
||||||
status: 'Running',
|
|
||||||
description: 'In-memory cache and pub/sub',
|
|
||||||
},
|
|
||||||
]
|
|
||||||
|
|
||||||
function Services() {
|
|
||||||
return (
|
|
||||||
<Box>
|
|
||||||
<Typography variant="h4" gutterBottom>
|
|
||||||
Services
|
|
||||||
</Typography>
|
|
||||||
|
|
||||||
<TableContainer component={Paper}>
|
|
||||||
<Table>
|
|
||||||
<TableHead>
|
|
||||||
<TableRow>
|
|
||||||
<TableCell>Service Name</TableCell>
|
|
||||||
<TableCell>Type</TableCell>
|
|
||||||
<TableCell>Port</TableCell>
|
|
||||||
<TableCell>Status</TableCell>
|
|
||||||
<TableCell>Description</TableCell>
|
|
||||||
</TableRow>
|
|
||||||
</TableHead>
|
|
||||||
<TableBody>
|
|
||||||
{servicesData.map((service) => (
|
|
||||||
<TableRow key={service.id}>
|
|
||||||
<TableCell>
|
|
||||||
<Typography variant="subtitle2">{service.name}</Typography>
|
|
||||||
</TableCell>
|
|
||||||
<TableCell>
|
|
||||||
<Chip
|
|
||||||
label={service.type}
|
|
||||||
size="small"
|
|
||||||
color={service.type === 'API Gateway' ? 'primary' : 'default'}
|
|
||||||
/>
|
|
||||||
</TableCell>
|
|
||||||
<TableCell>{service.port}</TableCell>
|
|
||||||
<TableCell>
|
|
||||||
<Chip
|
|
||||||
label={service.status}
|
|
||||||
size="small"
|
|
||||||
color="success"
|
|
||||||
/>
|
|
||||||
</TableCell>
|
|
||||||
<TableCell>{service.description}</TableCell>
|
|
||||||
</TableRow>
|
|
||||||
))}
|
|
||||||
</TableBody>
|
|
||||||
</Table>
|
|
||||||
</TableContainer>
|
|
||||||
</Box>
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
export default Services
|
|
||||||
@ -7,6 +7,19 @@ version: '3.8'
|
|||||||
|
|
||||||
services:
|
services:
|
||||||
# ============ Infrastructure Services ============
|
# ============ Infrastructure Services ============
|
||||||
|
|
||||||
|
# Local Docker Registry for K8s
|
||||||
|
registry:
|
||||||
|
image: registry:2
|
||||||
|
container_name: ${COMPOSE_PROJECT_NAME}_registry
|
||||||
|
ports:
|
||||||
|
- "5555:5000"
|
||||||
|
volumes:
|
||||||
|
- ./data/registry:/var/lib/registry
|
||||||
|
networks:
|
||||||
|
- site11_network
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
mongodb:
|
mongodb:
|
||||||
image: mongo:7.0
|
image: mongo:7.0
|
||||||
container_name: ${COMPOSE_PROJECT_NAME}_mongodb
|
container_name: ${COMPOSE_PROJECT_NAME}_mongodb
|
||||||
|
|||||||
117
docker-compose-registry-cache.yml
Normal file
117
docker-compose-registry-cache.yml
Normal file
@ -0,0 +1,117 @@
|
|||||||
|
version: '3.8'
|
||||||
|
|
||||||
|
services:
|
||||||
|
# Docker Registry with Cache Configuration
|
||||||
|
registry-cache:
|
||||||
|
image: registry:2
|
||||||
|
container_name: site11_registry_cache
|
||||||
|
restart: always
|
||||||
|
ports:
|
||||||
|
- "5000:5000"
|
||||||
|
environment:
|
||||||
|
# Registry configuration
|
||||||
|
REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry
|
||||||
|
REGISTRY_HTTP_ADDR: 0.0.0.0:5000
|
||||||
|
|
||||||
|
# Enable proxy cache for Docker Hub
|
||||||
|
REGISTRY_PROXY_REMOTEURL: https://registry-1.docker.io
|
||||||
|
REGISTRY_PROXY_USERNAME: ${DOCKER_HUB_USER:-}
|
||||||
|
REGISTRY_PROXY_PASSWORD: ${DOCKER_HUB_PASSWORD:-}
|
||||||
|
|
||||||
|
# Cache configuration
|
||||||
|
REGISTRY_STORAGE_CACHE_BLOBDESCRIPTOR: inmemory
|
||||||
|
REGISTRY_STORAGE_DELETE_ENABLED: "true"
|
||||||
|
|
||||||
|
# Garbage collection
|
||||||
|
REGISTRY_STORAGE_GC_ENABLED: "true"
|
||||||
|
REGISTRY_STORAGE_GC_INTERVAL: 12h
|
||||||
|
|
||||||
|
# Performance tuning
|
||||||
|
REGISTRY_HTTP_SECRET: ${REGISTRY_SECRET:-registrysecret}
|
||||||
|
REGISTRY_COMPATIBILITY_SCHEMA1_ENABLED: "true"
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
- registry-cache-data:/var/lib/registry
|
||||||
|
- ./registry/config.yml:/etc/docker/registry/config.yml:ro
|
||||||
|
networks:
|
||||||
|
- site11_network
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:5000/v2/"]
|
||||||
|
interval: 30s
|
||||||
|
timeout: 10s
|
||||||
|
retries: 3
|
||||||
|
|
||||||
|
# Harbor - Enterprise-grade Registry with Cache (Alternative)
|
||||||
|
harbor-registry:
|
||||||
|
image: goharbor/harbor-core:v2.9.0
|
||||||
|
container_name: site11_harbor
|
||||||
|
profiles: ["harbor"] # Only start with --profile harbor
|
||||||
|
environment:
|
||||||
|
HARBOR_ADMIN_PASSWORD: ${HARBOR_ADMIN_PASSWORD:-Harbor12345}
|
||||||
|
HARBOR_DB_PASSWORD: ${HARBOR_DB_PASSWORD:-Harbor12345}
|
||||||
|
# Enable proxy cache
|
||||||
|
HARBOR_PROXY_CACHE_ENABLED: "true"
|
||||||
|
HARBOR_PROXY_CACHE_ENDPOINT: https://registry-1.docker.io
|
||||||
|
ports:
|
||||||
|
- "8880:8080"
|
||||||
|
- "8443:8443"
|
||||||
|
volumes:
|
||||||
|
- harbor-data:/data
|
||||||
|
- harbor-config:/etc/harbor
|
||||||
|
networks:
|
||||||
|
- site11_network
|
||||||
|
|
||||||
|
# Sonatype Nexus - Repository Manager with Docker Registry (Alternative)
|
||||||
|
nexus:
|
||||||
|
image: sonatype/nexus3:latest
|
||||||
|
container_name: site11_nexus
|
||||||
|
profiles: ["nexus"] # Only start with --profile nexus
|
||||||
|
ports:
|
||||||
|
- "8081:8081" # Nexus UI
|
||||||
|
- "8082:8082" # Docker hosted registry
|
||||||
|
- "8083:8083" # Docker proxy registry (cache)
|
||||||
|
- "8084:8084" # Docker group registry
|
||||||
|
volumes:
|
||||||
|
- nexus-data:/nexus-data
|
||||||
|
environment:
|
||||||
|
NEXUS_CONTEXT: /
|
||||||
|
INSTALL4J_ADD_VM_PARAMS: "-Xms2g -Xmx2g -XX:MaxDirectMemorySize=3g"
|
||||||
|
networks:
|
||||||
|
- site11_network
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "curl", "-f", "http://localhost:8081/"]
|
||||||
|
interval: 30s
|
||||||
|
timeout: 10s
|
||||||
|
retries: 3
|
||||||
|
|
||||||
|
# Redis for registry cache metadata (optional enhancement)
|
||||||
|
registry-redis:
|
||||||
|
image: redis:7-alpine
|
||||||
|
container_name: site11_registry_redis
|
||||||
|
profiles: ["registry-redis"]
|
||||||
|
volumes:
|
||||||
|
- registry-redis-data:/data
|
||||||
|
networks:
|
||||||
|
- site11_network
|
||||||
|
command: redis-server --appendonly yes
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "redis-cli", "ping"]
|
||||||
|
interval: 30s
|
||||||
|
timeout: 10s
|
||||||
|
retries: 3
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
registry-cache-data:
|
||||||
|
driver: local
|
||||||
|
harbor-data:
|
||||||
|
driver: local
|
||||||
|
harbor-config:
|
||||||
|
driver: local
|
||||||
|
nexus-data:
|
||||||
|
driver: local
|
||||||
|
registry-redis-data:
|
||||||
|
driver: local
|
||||||
|
|
||||||
|
networks:
|
||||||
|
site11_network:
|
||||||
|
external: true
|
||||||
140
docker-compose.kubernetes.yml
Normal file
140
docker-compose.kubernetes.yml
Normal file
@ -0,0 +1,140 @@
|
|||||||
|
version: '3.8'
|
||||||
|
|
||||||
|
# Site11 KIND Kubernetes 개발 환경
|
||||||
|
#
|
||||||
|
# 빠른 시작:
|
||||||
|
# docker-compose -f docker-compose.kubernetes.yml up -d
|
||||||
|
#
|
||||||
|
# 관리 명령어:
|
||||||
|
# docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh setup
|
||||||
|
# docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh status
|
||||||
|
# docker-compose -f docker-compose.kubernetes.yml logs -f monitor
|
||||||
|
|
||||||
|
services:
|
||||||
|
# KIND CLI 관리 서비스 (kind, kubectl, docker 모두 포함)
|
||||||
|
# Note: MongoDB와 Redis는 기존 docker-compose.yml에서 관리됩니다
|
||||||
|
kind-cli:
|
||||||
|
image: alpine:latest
|
||||||
|
container_name: site11-kind-cli
|
||||||
|
volumes:
|
||||||
|
- /var/run/docker.sock:/var/run/docker.sock
|
||||||
|
- ~/.kube:/root/.kube
|
||||||
|
- ./k8s:/k8s
|
||||||
|
- ./scripts:/scripts
|
||||||
|
networks:
|
||||||
|
- kind
|
||||||
|
working_dir: /scripts
|
||||||
|
entrypoint: /bin/sh
|
||||||
|
command: |
|
||||||
|
-c "
|
||||||
|
# Install required tools
|
||||||
|
apk add --no-cache docker-cli curl bash
|
||||||
|
|
||||||
|
# Install kubectl
|
||||||
|
curl -LO https://dl.k8s.io/release/$$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl
|
||||||
|
chmod +x kubectl && mv kubectl /usr/local/bin/
|
||||||
|
|
||||||
|
# Install kind
|
||||||
|
curl -Lo ./kind https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64
|
||||||
|
chmod +x kind && mv kind /usr/local/bin/
|
||||||
|
|
||||||
|
echo '';
|
||||||
|
echo '╔═══════════════════════════════════════╗';
|
||||||
|
echo '║ Site11 KIND Cluster Manager ║';
|
||||||
|
echo '╚═══════════════════════════════════════╝';
|
||||||
|
echo '';
|
||||||
|
echo '사용 가능한 명령어:';
|
||||||
|
echo '';
|
||||||
|
echo ' 전체 설정 (클러스터 생성 + 배포):';
|
||||||
|
echo ' docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh setup';
|
||||||
|
echo '';
|
||||||
|
echo ' 개별 명령어:';
|
||||||
|
echo ' docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh create';
|
||||||
|
echo ' docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh status';
|
||||||
|
echo ' docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh delete';
|
||||||
|
echo '';
|
||||||
|
echo ' kubectl 직접 사용:';
|
||||||
|
echo ' docker-compose -f docker-compose.kubernetes.yml exec kind-cli kubectl get pods -A';
|
||||||
|
echo '';
|
||||||
|
echo ' Shell 접속:';
|
||||||
|
echo ' docker-compose -f docker-compose.kubernetes.yml exec kind-cli bash';
|
||||||
|
echo '';
|
||||||
|
echo 'KIND CLI 준비 완료!';
|
||||||
|
tail -f /dev/null
|
||||||
|
"
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
# 클러스터 실시간 모니터링
|
||||||
|
monitor:
|
||||||
|
image: bitnami/kubectl:latest
|
||||||
|
container_name: site11-kind-monitor
|
||||||
|
volumes:
|
||||||
|
- ~/.kube:/root/.kube:ro
|
||||||
|
networks:
|
||||||
|
- kind
|
||||||
|
entrypoint: /bin/bash
|
||||||
|
command: |
|
||||||
|
-c "
|
||||||
|
while true; do
|
||||||
|
clear;
|
||||||
|
echo '╔═══════════════════════════════════════════════════╗';
|
||||||
|
echo '║ Site11 KIND Cluster Monitor ║';
|
||||||
|
echo '║ Updated: $$(date +"%Y-%m-%d %H:%M:%S") ║';
|
||||||
|
echo '╚═══════════════════════════════════════════════════╝';
|
||||||
|
echo '';
|
||||||
|
|
||||||
|
if kubectl cluster-info --context kind-site11-dev &>/dev/null; then
|
||||||
|
echo '✅ Cluster Status: Running';
|
||||||
|
echo '━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━';
|
||||||
|
echo '';
|
||||||
|
|
||||||
|
echo '📦 Nodes:';
|
||||||
|
kubectl get nodes --context kind-site11-dev 2>/dev/null | sed '1s/.*/ &/' | sed '1!s/.*/ &/' || echo ' No nodes';
|
||||||
|
echo '';
|
||||||
|
|
||||||
|
echo '🔧 Console Namespace (site11-console):';
|
||||||
|
kubectl get pods -n site11-console --context kind-site11-dev 2>/dev/null | sed '1s/.*/ &/' | sed '1!s/.*/ &/' || echo ' No pods';
|
||||||
|
echo '';
|
||||||
|
|
||||||
|
echo '📊 Services:';
|
||||||
|
kubectl get svc -n site11-console --context kind-site11-dev 2>/dev/null | sed '1s/.*/ &/' | sed '1!s/.*/ &/' || echo ' No services';
|
||||||
|
echo '';
|
||||||
|
|
||||||
|
echo '━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━';
|
||||||
|
echo '🌐 Access URLs:';
|
||||||
|
echo ' Frontend: http://localhost:3000';
|
||||||
|
echo ' Backend: http://localhost:8000';
|
||||||
|
else
|
||||||
|
echo '❌ Cluster Status: Not Running';
|
||||||
|
echo '━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━';
|
||||||
|
echo '';
|
||||||
|
echo '시작 방법:';
|
||||||
|
echo ' docker-compose -f docker-compose.kubernetes.yml exec kind-cli /scripts/kind-setup.sh setup';
|
||||||
|
fi;
|
||||||
|
|
||||||
|
echo '';
|
||||||
|
echo 'Next update in 30 seconds... (Press Ctrl+C to stop)';
|
||||||
|
sleep 30;
|
||||||
|
done
|
||||||
|
"
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
networks:
|
||||||
|
kind:
|
||||||
|
name: kind
|
||||||
|
external: true
|
||||||
|
|
||||||
|
# 참고:
|
||||||
|
# 1. KIND 클러스터 자체는 docker-compose로 직접 제어되지 않습니다
|
||||||
|
# 2. 이 파일은 KIND 클러스터 관리를 위한 헬퍼 컨테이너들을 제공합니다
|
||||||
|
# 3. 실제 클러스터 생성/삭제는 kind CLI를 사용해야 합니다
|
||||||
|
#
|
||||||
|
# KIND 클러스터 라이프사이클:
|
||||||
|
# 생성: kind create cluster --config k8s/kind-dev-cluster.yaml
|
||||||
|
# 삭제: kind delete cluster --name site11-dev
|
||||||
|
# 목록: kind get clusters
|
||||||
|
#
|
||||||
|
# docker-compose 명령어:
|
||||||
|
# 헬퍼 시작: docker-compose -f docker-compose.kubernetes.yml up -d
|
||||||
|
# 헬퍼 중지: docker-compose -f docker-compose.kubernetes.yml down
|
||||||
|
# 로그 확인: docker-compose -f docker-compose.kubernetes.yml logs -f kind-monitor
|
||||||
@ -665,6 +665,94 @@ services:
|
|||||||
networks:
|
networks:
|
||||||
- site11_network
|
- site11_network
|
||||||
|
|
||||||
|
# PostgreSQL for SAPIENS
|
||||||
|
sapiens-postgres:
|
||||||
|
image: postgres:16-alpine
|
||||||
|
container_name: ${COMPOSE_PROJECT_NAME}_sapiens_postgres
|
||||||
|
environment:
|
||||||
|
- POSTGRES_DB=sapiens_db
|
||||||
|
- POSTGRES_USER=sapiens_user
|
||||||
|
- POSTGRES_PASSWORD=sapiens_password
|
||||||
|
ports:
|
||||||
|
- "5433:5432"
|
||||||
|
volumes:
|
||||||
|
- ./data/sapiens-postgres:/var/lib/postgresql/data
|
||||||
|
networks:
|
||||||
|
- site11_network
|
||||||
|
restart: unless-stopped
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "pg_isready -U sapiens_user -d sapiens_db"]
|
||||||
|
interval: 10s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 5
|
||||||
|
|
||||||
|
# SAPIENS Web Platform
|
||||||
|
sapiens-web:
|
||||||
|
build:
|
||||||
|
context: ./services/sapiens-web
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
container_name: ${COMPOSE_PROJECT_NAME}_sapiens_web
|
||||||
|
ports:
|
||||||
|
- "3005:5000"
|
||||||
|
environment:
|
||||||
|
- NODE_ENV=development
|
||||||
|
- PORT=5000
|
||||||
|
- DATABASE_URL=postgresql://sapiens_user:sapiens_password@sapiens-postgres:5432/sapiens_db
|
||||||
|
- SESSION_SECRET=sapiens_dev_secret_key_change_in_production
|
||||||
|
volumes:
|
||||||
|
- ./services/sapiens-web:/app
|
||||||
|
- /app/node_modules
|
||||||
|
networks:
|
||||||
|
- site11_network
|
||||||
|
restart: unless-stopped
|
||||||
|
depends_on:
|
||||||
|
sapiens-postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
|
||||||
|
# PostgreSQL for SAPIENS Web2
|
||||||
|
sapiens-postgres2:
|
||||||
|
image: postgres:16-alpine
|
||||||
|
container_name: ${COMPOSE_PROJECT_NAME}_sapiens_postgres2
|
||||||
|
environment:
|
||||||
|
- POSTGRES_DB=sapiens_db2
|
||||||
|
- POSTGRES_USER=sapiens_user2
|
||||||
|
- POSTGRES_PASSWORD=sapiens_password2
|
||||||
|
ports:
|
||||||
|
- "5434:5432"
|
||||||
|
volumes:
|
||||||
|
- ./data/sapiens-postgres2:/var/lib/postgresql/data
|
||||||
|
networks:
|
||||||
|
- site11_network
|
||||||
|
restart: unless-stopped
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "pg_isready -U sapiens_user2 -d sapiens_db2"]
|
||||||
|
interval: 10s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 5
|
||||||
|
|
||||||
|
# SAPIENS Web2 Platform
|
||||||
|
sapiens-web2:
|
||||||
|
build:
|
||||||
|
context: ./services/sapiens-web2
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
container_name: ${COMPOSE_PROJECT_NAME}_sapiens_web2
|
||||||
|
ports:
|
||||||
|
- "3006:5000"
|
||||||
|
environment:
|
||||||
|
- NODE_ENV=development
|
||||||
|
- PORT=5000
|
||||||
|
- DATABASE_URL=postgresql://sapiens_user2:sapiens_password2@sapiens-postgres2:5432/sapiens_db2
|
||||||
|
- SESSION_SECRET=sapiens2_dev_secret_key_change_in_production
|
||||||
|
volumes:
|
||||||
|
- ./services/sapiens-web2:/app
|
||||||
|
- /app/node_modules
|
||||||
|
networks:
|
||||||
|
- site11_network
|
||||||
|
restart: unless-stopped
|
||||||
|
depends_on:
|
||||||
|
sapiens-postgres2:
|
||||||
|
condition: service_healthy
|
||||||
|
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
site11_network:
|
site11_network:
|
||||||
|
|||||||
397
docs/ARCHITECTURE_OVERVIEW.md
Normal file
397
docs/ARCHITECTURE_OVERVIEW.md
Normal file
@ -0,0 +1,397 @@
|
|||||||
|
# Site11 시스템 아키텍처 개요
|
||||||
|
|
||||||
|
## 📋 목차
|
||||||
|
- [전체 아키텍처](#전체-아키텍처)
|
||||||
|
- [마이크로서비스 구성](#마이크로서비스-구성)
|
||||||
|
- [데이터 플로우](#데이터-플로우)
|
||||||
|
- [기술 스택](#기술-스택)
|
||||||
|
- [확장성 고려사항](#확장성-고려사항)
|
||||||
|
|
||||||
|
## 전체 아키텍처
|
||||||
|
|
||||||
|
### 하이브리드 아키텍처 (현재)
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────┐
|
||||||
|
│ 외부 API │
|
||||||
|
│ DeepL | OpenAI | Claude | Google Search | RSS Feeds │
|
||||||
|
└────────────────────┬────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
┌─────────────────────┴────────────────────────────────────┐
|
||||||
|
│ Kubernetes Cluster │
|
||||||
|
│ ┌─────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Frontend Layer │ │
|
||||||
|
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
|
||||||
|
│ │ │ Console │ │ Images │ │ Users │ │ │
|
||||||
|
│ │ │ Frontend │ │ Frontend │ │ Frontend │ │ │
|
||||||
|
│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │
|
||||||
|
│ └─────────────────────────────────────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ┌─────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ API Gateway Layer │ │
|
||||||
|
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
|
||||||
|
│ │ │ Console │ │ Images │ │ Users │ │ │
|
||||||
|
│ │ │ Backend │ │ Backend │ │ Backend │ │ │
|
||||||
|
│ │ │ (Gateway) │ │ │ │ │ │ │
|
||||||
|
│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │
|
||||||
|
│ └─────────────────────────────────────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ┌─────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Pipeline Workers Layer │ │
|
||||||
|
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌─────────┐ │ │
|
||||||
|
│ │ │RSS │ │Google │ │AI Article│ │Image │ │ │
|
||||||
|
│ │ │Collector │ │Search │ │Generator │ │Generator│ │ │
|
||||||
|
│ │ └──────────┘ └──────────┘ └──────────┘ └─────────┘ │ │
|
||||||
|
│ │ ┌─────────────────────────────────────────────────┐ │ │
|
||||||
|
│ │ │ Translator │ │ │
|
||||||
|
│ │ │ (8 Languages Support) │ │ │
|
||||||
|
│ │ └─────────────────────────────────────────────────┘ │ │
|
||||||
|
│ └─────────────────────────────────────────────────────┘ │
|
||||||
|
└────────────────────┬────────────────────────────────────┘
|
||||||
|
│ host.docker.internal
|
||||||
|
┌─────────────────────┴────────────────────────────────────┐
|
||||||
|
│ Docker Compose Infrastructure │
|
||||||
|
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||||
|
│ │ MongoDB │ │ Redis │ │ Kafka │ │
|
||||||
|
│ │ (Primary │ │ (Cache & │ │ (Message │ │
|
||||||
|
│ │ Database) │ │ Queue) │ │ Broker) │ │
|
||||||
|
│ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||||||
|
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||||
|
│ │ Zookeeper │ │ Pipeline │ │ Pipeline │ │
|
||||||
|
│ │(Kafka Coord)│ │ Scheduler │ │ Monitor │ │
|
||||||
|
│ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||||||
|
│ ┌─────────────┐ ┌─────────────┐ │
|
||||||
|
│ │ Language │ │ Registry │ │
|
||||||
|
│ │ Sync │ │ Cache │ │
|
||||||
|
│ └─────────────┘ └─────────────┘ │
|
||||||
|
└──────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## 마이크로서비스 구성
|
||||||
|
|
||||||
|
### Console Services (API Gateway Pattern)
|
||||||
|
```yaml
|
||||||
|
Console Backend:
|
||||||
|
Purpose: API Gateway & Orchestration
|
||||||
|
Technology: FastAPI
|
||||||
|
Port: 8000
|
||||||
|
Features:
|
||||||
|
- Service Discovery
|
||||||
|
- Authentication & Authorization
|
||||||
|
- Request Routing
|
||||||
|
- Health Monitoring
|
||||||
|
|
||||||
|
Console Frontend:
|
||||||
|
Purpose: Admin Dashboard
|
||||||
|
Technology: React + Vite + TypeScript
|
||||||
|
Port: 80 (nginx)
|
||||||
|
Features:
|
||||||
|
- Service Health Dashboard
|
||||||
|
- Real-time Monitoring
|
||||||
|
- User Management UI
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pipeline Services (Event-Driven Architecture)
|
||||||
|
```yaml
|
||||||
|
RSS Collector:
|
||||||
|
Purpose: RSS Feed 수집
|
||||||
|
Scaling: 1-5 replicas
|
||||||
|
Queue: rss_collection
|
||||||
|
|
||||||
|
Google Search:
|
||||||
|
Purpose: Google 검색 결과 수집
|
||||||
|
Scaling: 1-5 replicas
|
||||||
|
Queue: google_search
|
||||||
|
|
||||||
|
AI Article Generator:
|
||||||
|
Purpose: AI 기반 콘텐츠 생성
|
||||||
|
Scaling: 2-10 replicas
|
||||||
|
Queue: ai_generation
|
||||||
|
APIs: OpenAI, Claude
|
||||||
|
|
||||||
|
Translator:
|
||||||
|
Purpose: 8개 언어 번역
|
||||||
|
Scaling: 3-10 replicas (높은 처리량)
|
||||||
|
Queue: translation
|
||||||
|
API: DeepL
|
||||||
|
|
||||||
|
Image Generator:
|
||||||
|
Purpose: 이미지 생성 및 최적화
|
||||||
|
Scaling: 2-10 replicas
|
||||||
|
Queue: image_generation
|
||||||
|
API: OpenAI DALL-E
|
||||||
|
```
|
||||||
|
|
||||||
|
### Infrastructure Services (Stateful)
|
||||||
|
```yaml
|
||||||
|
MongoDB:
|
||||||
|
Purpose: Primary Database
|
||||||
|
Collections:
|
||||||
|
- articles_ko (Korean articles)
|
||||||
|
- articles_en (English articles)
|
||||||
|
- articles_zh_cn, articles_zh_tw (Chinese)
|
||||||
|
- articles_ja (Japanese)
|
||||||
|
- articles_fr, articles_de, articles_es, articles_it (European)
|
||||||
|
|
||||||
|
Redis:
|
||||||
|
Purpose: Cache & Queue
|
||||||
|
Usage:
|
||||||
|
- Queue management (FIFO/Priority)
|
||||||
|
- Session storage
|
||||||
|
- Result caching
|
||||||
|
- Rate limiting
|
||||||
|
|
||||||
|
Kafka:
|
||||||
|
Purpose: Event Streaming
|
||||||
|
Topics:
|
||||||
|
- user-events
|
||||||
|
- oauth-events
|
||||||
|
- pipeline-events
|
||||||
|
- dead-letter-queue
|
||||||
|
|
||||||
|
Pipeline Scheduler:
|
||||||
|
Purpose: Workflow Orchestration
|
||||||
|
Features:
|
||||||
|
- Task scheduling
|
||||||
|
- Dependency management
|
||||||
|
- Error handling
|
||||||
|
- Retry logic
|
||||||
|
|
||||||
|
Pipeline Monitor:
|
||||||
|
Purpose: Real-time Monitoring
|
||||||
|
Features:
|
||||||
|
- Queue status
|
||||||
|
- Processing metrics
|
||||||
|
- Performance monitoring
|
||||||
|
- Alerting
|
||||||
|
```
|
||||||
|
|
||||||
|
## 데이터 플로우
|
||||||
|
|
||||||
|
### 콘텐츠 생성 플로우
|
||||||
|
```
|
||||||
|
1. Content Collection
|
||||||
|
RSS Feeds → RSS Collector → Redis Queue
|
||||||
|
Search Terms → Google Search → Redis Queue
|
||||||
|
|
||||||
|
2. Content Processing
|
||||||
|
Raw Content → AI Article Generator → Enhanced Articles
|
||||||
|
|
||||||
|
3. Multi-Language Translation
|
||||||
|
Korean Articles → Translator (DeepL) → 8 Languages
|
||||||
|
|
||||||
|
4. Image Generation
|
||||||
|
Article Content → Image Generator (DALL-E) → Optimized Images
|
||||||
|
|
||||||
|
5. Data Storage
|
||||||
|
Processed Content → MongoDB Collections (by language)
|
||||||
|
|
||||||
|
6. Language Synchronization
|
||||||
|
Language Sync Service → Monitors & balances translations
|
||||||
|
```
|
||||||
|
|
||||||
|
### 실시간 모니터링 플로우
|
||||||
|
```
|
||||||
|
1. Metrics Collection
|
||||||
|
Each Service → Pipeline Monitor → Real-time Dashboard
|
||||||
|
|
||||||
|
2. Health Monitoring
|
||||||
|
Services → Health Endpoints → Console Backend → Dashboard
|
||||||
|
|
||||||
|
3. Queue Monitoring
|
||||||
|
Redis Queues → Pipeline Monitor → Queue Status Display
|
||||||
|
|
||||||
|
4. Event Streaming
|
||||||
|
Service Events → Kafka → Event Consumer → Real-time Updates
|
||||||
|
```
|
||||||
|
|
||||||
|
## 기술 스택
|
||||||
|
|
||||||
|
### Backend Technologies
|
||||||
|
```yaml
|
||||||
|
API Framework: FastAPI (Python 3.11)
|
||||||
|
Database: MongoDB 7.0
|
||||||
|
Cache/Queue: Redis 7
|
||||||
|
Message Broker: Kafka 3.5 + Zookeeper 3.9
|
||||||
|
Container Runtime: Docker + Kubernetes
|
||||||
|
Registry: Docker Hub + Local Registry
|
||||||
|
```
|
||||||
|
|
||||||
|
### Frontend Technologies
|
||||||
|
```yaml
|
||||||
|
Framework: React 18
|
||||||
|
Build Tool: Vite 4
|
||||||
|
Language: TypeScript
|
||||||
|
UI Library: Material-UI v7
|
||||||
|
Bundler: Rollup (via Vite)
|
||||||
|
Web Server: Nginx (Production)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Infrastructure Technologies
|
||||||
|
```yaml
|
||||||
|
Orchestration: Kubernetes (Kind/Docker Desktop)
|
||||||
|
Container Platform: Docker 20.10+
|
||||||
|
Networking: Docker Networks + K8s Services
|
||||||
|
Storage: Docker Volumes + K8s PVCs
|
||||||
|
Monitoring: Custom Dashboard + kubectl
|
||||||
|
```
|
||||||
|
|
||||||
|
### External APIs
|
||||||
|
```yaml
|
||||||
|
Translation: DeepL API
|
||||||
|
AI Content: OpenAI GPT + Claude API
|
||||||
|
Image Generation: OpenAI DALL-E
|
||||||
|
Search: Google Custom Search API (SERP)
|
||||||
|
```
|
||||||
|
|
||||||
|
## 확장성 고려사항
|
||||||
|
|
||||||
|
### Horizontal Scaling (현재 구현됨)
|
||||||
|
```yaml
|
||||||
|
Auto-scaling Rules:
|
||||||
|
CPU > 70% → Scale Up
|
||||||
|
Memory > 80% → Scale Up
|
||||||
|
Queue Length > 100 → Scale Up
|
||||||
|
|
||||||
|
Scaling Limits:
|
||||||
|
Console: 2-10 replicas
|
||||||
|
Translator: 3-10 replicas (highest throughput)
|
||||||
|
AI Generator: 2-10 replicas
|
||||||
|
Others: 1-5 replicas
|
||||||
|
```
|
||||||
|
|
||||||
|
### Vertical Scaling
|
||||||
|
```yaml
|
||||||
|
Resource Allocation:
|
||||||
|
CPU Intensive: AI Generator, Image Generator
|
||||||
|
Memory Intensive: Translator (language models)
|
||||||
|
I/O Intensive: RSS Collector, Database operations
|
||||||
|
|
||||||
|
Resource Limits:
|
||||||
|
Request: 100m CPU, 256Mi RAM
|
||||||
|
Limit: 500m CPU, 512Mi RAM
|
||||||
|
```
|
||||||
|
|
||||||
|
### Database Scaling
|
||||||
|
```yaml
|
||||||
|
Current: Single MongoDB instance
|
||||||
|
Future Options:
|
||||||
|
- MongoDB Replica Set (HA)
|
||||||
|
- Sharding by language
|
||||||
|
- Read replicas for different regions
|
||||||
|
|
||||||
|
Indexing Strategy:
|
||||||
|
- Language-based indexing
|
||||||
|
- Timestamp-based partitioning
|
||||||
|
- Full-text search indexes
|
||||||
|
```
|
||||||
|
|
||||||
|
### Caching Strategy
|
||||||
|
```yaml
|
||||||
|
L1 Cache: Application-level (FastAPI)
|
||||||
|
L2 Cache: Redis (shared)
|
||||||
|
L3 Cache: Registry Cache (Docker images)
|
||||||
|
|
||||||
|
Cache Invalidation:
|
||||||
|
- TTL-based expiration
|
||||||
|
- Event-driven invalidation
|
||||||
|
- Manual cache warming
|
||||||
|
```
|
||||||
|
|
||||||
|
### API Rate Limiting
|
||||||
|
```yaml
|
||||||
|
External APIs:
|
||||||
|
DeepL: 500,000 chars/month
|
||||||
|
OpenAI: Usage-based billing
|
||||||
|
Google Search: 100 queries/day (free tier)
|
||||||
|
|
||||||
|
Rate Limiting Strategy:
|
||||||
|
- Redis-based rate limiting
|
||||||
|
- Queue-based buffering
|
||||||
|
- Priority queuing
|
||||||
|
- Circuit breaker pattern
|
||||||
|
```
|
||||||
|
|
||||||
|
### Future Architecture Considerations
|
||||||
|
|
||||||
|
#### Service Mesh (다음 단계)
|
||||||
|
```yaml
|
||||||
|
Technology: Istio or Linkerd
|
||||||
|
Benefits:
|
||||||
|
- Service-to-service encryption
|
||||||
|
- Traffic management
|
||||||
|
- Observability
|
||||||
|
- Circuit breaking
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Multi-Region Deployment
|
||||||
|
```yaml
|
||||||
|
Current: Single cluster
|
||||||
|
Future: Multi-region with:
|
||||||
|
- Regional MongoDB clusters
|
||||||
|
- CDN for static assets
|
||||||
|
- Geo-distributed caching
|
||||||
|
- Language-specific regions
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Event Sourcing
|
||||||
|
```yaml
|
||||||
|
Current: State-based
|
||||||
|
Future: Event-based with:
|
||||||
|
- Event store (EventStore or Kafka)
|
||||||
|
- CQRS pattern
|
||||||
|
- Aggregate reconstruction
|
||||||
|
- Audit trail
|
||||||
|
```
|
||||||
|
|
||||||
|
## 보안 아키텍처
|
||||||
|
|
||||||
|
### Authentication & Authorization
|
||||||
|
```yaml
|
||||||
|
Current: JWT-based authentication
|
||||||
|
Users: Demo users (admin/user)
|
||||||
|
Tokens: 30-minute expiration
|
||||||
|
|
||||||
|
Future:
|
||||||
|
- OAuth2 with external providers
|
||||||
|
- RBAC with granular permissions
|
||||||
|
- API key management
|
||||||
|
```
|
||||||
|
|
||||||
|
### Network Security
|
||||||
|
```yaml
|
||||||
|
K8s Network Policies: Not implemented
|
||||||
|
Service Mesh Security: Future consideration
|
||||||
|
Secrets Management: K8s Secrets + .env files
|
||||||
|
|
||||||
|
Future:
|
||||||
|
- HashiCorp Vault integration
|
||||||
|
- mTLS between services
|
||||||
|
- Network segmentation
|
||||||
|
```
|
||||||
|
|
||||||
|
## 성능 특성
|
||||||
|
|
||||||
|
### Throughput Metrics
|
||||||
|
```yaml
|
||||||
|
Translation: ~100 articles/minute (3 replicas)
|
||||||
|
AI Generation: ~50 articles/minute (2 replicas)
|
||||||
|
Image Generation: ~20 images/minute (2 replicas)
|
||||||
|
Total Processing: ~1000 articles/hour
|
||||||
|
```
|
||||||
|
|
||||||
|
### Latency Targets
|
||||||
|
```yaml
|
||||||
|
API Response: < 200ms
|
||||||
|
Translation: < 5s per article
|
||||||
|
AI Generation: < 30s per article
|
||||||
|
Image Generation: < 60s per image
|
||||||
|
End-to-end: < 2 minutes per complete article
|
||||||
|
```
|
||||||
|
|
||||||
|
### Resource Utilization
|
||||||
|
```yaml
|
||||||
|
CPU Usage: 60-80% under normal load
|
||||||
|
Memory Usage: 70-90% under normal load
|
||||||
|
Disk I/O: MongoDB primary bottleneck
|
||||||
|
Network I/O: External API calls
|
||||||
|
```
|
||||||
546
docs/CONSOLE_ARCHITECTURE.md
Normal file
546
docs/CONSOLE_ARCHITECTURE.md
Normal file
@ -0,0 +1,546 @@
|
|||||||
|
# Console Architecture Design
|
||||||
|
|
||||||
|
## 1. 시스템 개요
|
||||||
|
|
||||||
|
Site11 Console은 마이크로서비스 기반 뉴스 생성 파이프라인의 중앙 관리 시스템입니다.
|
||||||
|
|
||||||
|
### 핵심 기능
|
||||||
|
1. **인증 및 권한 관리** (OAuth2.0 + JWT)
|
||||||
|
2. **서비스 관리** (Microservices CRUD)
|
||||||
|
3. **뉴스 시스템** (키워드 기반 뉴스 생성 관리)
|
||||||
|
4. **파이프라인 관리** (실시간 모니터링 및 제어)
|
||||||
|
5. **대시보드** (시스템 현황 및 모니터링)
|
||||||
|
6. **통계 및 분석** (사용자, 서비스, 뉴스 생성 통계)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 시스템 아키텍처
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────┐
|
||||||
|
│ Console Frontend (React) │
|
||||||
|
│ ┌──────────┬──────────┬──────────┬──────────┬──────────┐ │
|
||||||
|
│ │ Auth │ Services │ News │ Pipeline │Dashboard │ │
|
||||||
|
│ │ Module │ Module │ Module │ Module │ Module │ │
|
||||||
|
│ └──────────┴──────────┴──────────┴──────────┴──────────┘ │
|
||||||
|
└─────────────────────────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
│ REST API + WebSocket
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────┐
|
||||||
|
│ Console Backend (FastAPI) │
|
||||||
|
│ ┌──────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ API Gateway Layer │ │
|
||||||
|
│ ├──────────┬──────────┬──────────┬──────────┬──────────┤ │
|
||||||
|
│ │ Auth │ Services │ News │ Pipeline │ Stats │ │
|
||||||
|
│ │ Service │ Manager │ Manager │ Manager │ Service │ │
|
||||||
|
│ └──────────┴──────────┴──────────┴──────────┴──────────┘ │
|
||||||
|
└─────────────────────────────────────────────────────────────┘
|
||||||
|
│ │ │
|
||||||
|
▼ ▼ ▼
|
||||||
|
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
||||||
|
│ MongoDB │ │ Redis │ │ Pipeline │
|
||||||
|
│ (Metadata) │ │ (Queue/ │ │ Workers │
|
||||||
|
│ │ │ Cache) │ │ │
|
||||||
|
└──────────────┘ └──────────────┘ └──────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 데이터 모델 설계
|
||||||
|
|
||||||
|
### 3.1 Users Collection
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"_id": "ObjectId",
|
||||||
|
"email": "user@example.com",
|
||||||
|
"username": "username",
|
||||||
|
"password_hash": "bcrypt_hash",
|
||||||
|
"full_name": "Full Name",
|
||||||
|
"role": "admin|editor|viewer",
|
||||||
|
"permissions": ["service:read", "news:write", "pipeline:manage"],
|
||||||
|
"oauth_providers": [
|
||||||
|
{
|
||||||
|
"provider": "google|github|azure",
|
||||||
|
"provider_user_id": "external_id",
|
||||||
|
"access_token": "encrypted_token",
|
||||||
|
"refresh_token": "encrypted_token"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"profile": {
|
||||||
|
"avatar_url": "https://...",
|
||||||
|
"department": "Engineering",
|
||||||
|
"timezone": "Asia/Seoul"
|
||||||
|
},
|
||||||
|
"status": "active|suspended|deleted",
|
||||||
|
"created_at": "2024-01-01T00:00:00Z",
|
||||||
|
"updated_at": "2024-01-01T00:00:00Z",
|
||||||
|
"last_login_at": "2024-01-01T00:00:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.2 Services Collection
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"_id": "ObjectId",
|
||||||
|
"service_id": "rss-collector",
|
||||||
|
"name": "RSS Collector Service",
|
||||||
|
"type": "pipeline_worker",
|
||||||
|
"category": "data_collection",
|
||||||
|
"description": "Collects news from RSS feeds",
|
||||||
|
"status": "running|stopped|error|deploying",
|
||||||
|
"deployment": {
|
||||||
|
"namespace": "site11-pipeline",
|
||||||
|
"deployment_name": "pipeline-rss-collector",
|
||||||
|
"replicas": {
|
||||||
|
"desired": 2,
|
||||||
|
"current": 2,
|
||||||
|
"ready": 2
|
||||||
|
},
|
||||||
|
"image": "yakenator/site11-rss-collector:latest",
|
||||||
|
"resources": {
|
||||||
|
"requests": {"cpu": "100m", "memory": "256Mi"},
|
||||||
|
"limits": {"cpu": "500m", "memory": "512Mi"}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"config": {
|
||||||
|
"env_vars": {
|
||||||
|
"REDIS_URL": "redis://...",
|
||||||
|
"MONGODB_URL": "mongodb://...",
|
||||||
|
"LOG_LEVEL": "INFO"
|
||||||
|
},
|
||||||
|
"queue_name": "rss_collection",
|
||||||
|
"batch_size": 10,
|
||||||
|
"worker_count": 2
|
||||||
|
},
|
||||||
|
"health": {
|
||||||
|
"endpoint": "/health",
|
||||||
|
"status": "healthy|unhealthy|unknown",
|
||||||
|
"last_check": "2024-01-01T00:00:00Z",
|
||||||
|
"uptime_seconds": 3600
|
||||||
|
},
|
||||||
|
"metrics": {
|
||||||
|
"requests_total": 1000,
|
||||||
|
"requests_failed": 10,
|
||||||
|
"avg_response_time_ms": 150,
|
||||||
|
"cpu_usage_percent": 45.5,
|
||||||
|
"memory_usage_mb": 256
|
||||||
|
},
|
||||||
|
"created_by": "user_id",
|
||||||
|
"created_at": "2024-01-01T00:00:00Z",
|
||||||
|
"updated_at": "2024-01-01T00:00:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.3 News Keywords Collection
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"_id": "ObjectId",
|
||||||
|
"keyword": "도널드 트럼프",
|
||||||
|
"keyword_type": "person|topic|company|location|custom",
|
||||||
|
"category": "politics|technology|business|sports|entertainment",
|
||||||
|
"languages": ["ko", "en", "ja", "zh_cn"],
|
||||||
|
"config": {
|
||||||
|
"enabled": true,
|
||||||
|
"priority": 1,
|
||||||
|
"collection_frequency": "hourly|daily|realtime",
|
||||||
|
"max_articles_per_day": 50,
|
||||||
|
"sources": [
|
||||||
|
{
|
||||||
|
"type": "rss",
|
||||||
|
"url": "https://...",
|
||||||
|
"enabled": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "google_search",
|
||||||
|
"query": "도널드 트럼프 news",
|
||||||
|
"enabled": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"processing_rules": {
|
||||||
|
"translate": true,
|
||||||
|
"target_languages": ["en", "ja", "zh_cn"],
|
||||||
|
"generate_image": true,
|
||||||
|
"sentiment_analysis": true,
|
||||||
|
"entity_extraction": true
|
||||||
|
},
|
||||||
|
"statistics": {
|
||||||
|
"total_articles_collected": 5000,
|
||||||
|
"total_articles_published": 4800,
|
||||||
|
"last_collection_at": "2024-01-01T00:00:00Z",
|
||||||
|
"success_rate": 96.0
|
||||||
|
},
|
||||||
|
"status": "active|paused|archived",
|
||||||
|
"tags": ["politics", "usa", "election"],
|
||||||
|
"created_by": "user_id",
|
||||||
|
"created_at": "2024-01-01T00:00:00Z",
|
||||||
|
"updated_at": "2024-01-01T00:00:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.4 Pipeline Jobs Collection
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"_id": "ObjectId",
|
||||||
|
"job_id": "job_20240101_001",
|
||||||
|
"job_type": "news_collection|translation|image_generation",
|
||||||
|
"keyword_id": "ObjectId",
|
||||||
|
"keyword": "도널드 트럼프",
|
||||||
|
"status": "pending|processing|completed|failed|cancelled",
|
||||||
|
"priority": 1,
|
||||||
|
"pipeline_stages": [
|
||||||
|
{
|
||||||
|
"stage": "rss_collection",
|
||||||
|
"status": "completed",
|
||||||
|
"worker_id": "rss-collector-pod-123",
|
||||||
|
"started_at": "2024-01-01T00:00:00Z",
|
||||||
|
"completed_at": "2024-01-01T00:00:10Z",
|
||||||
|
"duration_ms": 10000,
|
||||||
|
"result": {
|
||||||
|
"articles_found": 15,
|
||||||
|
"articles_processed": 15
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"stage": "google_search",
|
||||||
|
"status": "completed",
|
||||||
|
"worker_id": "google-search-pod-456",
|
||||||
|
"started_at": "2024-01-01T00:00:10Z",
|
||||||
|
"completed_at": "2024-01-01T00:00:20Z",
|
||||||
|
"duration_ms": 10000,
|
||||||
|
"result": {
|
||||||
|
"articles_found": 20,
|
||||||
|
"articles_processed": 18
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"stage": "translation",
|
||||||
|
"status": "processing",
|
||||||
|
"worker_id": "translator-pod-789",
|
||||||
|
"started_at": "2024-01-01T00:00:20Z",
|
||||||
|
"progress": {
|
||||||
|
"total": 33,
|
||||||
|
"completed": 20,
|
||||||
|
"percent": 60.6
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"stage": "ai_article_generation",
|
||||||
|
"status": "pending",
|
||||||
|
"worker_id": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"stage": "image_generation",
|
||||||
|
"status": "pending",
|
||||||
|
"worker_id": null
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"source": "scheduled|manual|api",
|
||||||
|
"triggered_by": "user_id",
|
||||||
|
"retry_count": 0,
|
||||||
|
"max_retries": 3
|
||||||
|
},
|
||||||
|
"errors": [],
|
||||||
|
"created_at": "2024-01-01T00:00:00Z",
|
||||||
|
"updated_at": "2024-01-01T00:00:20Z",
|
||||||
|
"completed_at": null
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.5 System Statistics Collection
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"_id": "ObjectId",
|
||||||
|
"date": "2024-01-01",
|
||||||
|
"hour": 14,
|
||||||
|
"metrics": {
|
||||||
|
"users": {
|
||||||
|
"total_active": 150,
|
||||||
|
"new_registrations": 5,
|
||||||
|
"active_sessions": 45
|
||||||
|
},
|
||||||
|
"services": {
|
||||||
|
"total": 7,
|
||||||
|
"running": 7,
|
||||||
|
"stopped": 0,
|
||||||
|
"error": 0,
|
||||||
|
"avg_cpu_usage": 45.5,
|
||||||
|
"avg_memory_usage": 512.0,
|
||||||
|
"total_requests": 10000,
|
||||||
|
"failed_requests": 50
|
||||||
|
},
|
||||||
|
"news": {
|
||||||
|
"keywords_active": 100,
|
||||||
|
"articles_collected": 500,
|
||||||
|
"articles_translated": 450,
|
||||||
|
"articles_published": 480,
|
||||||
|
"images_generated": 480,
|
||||||
|
"avg_processing_time_ms": 15000,
|
||||||
|
"success_rate": 96.0
|
||||||
|
},
|
||||||
|
"pipeline": {
|
||||||
|
"jobs_total": 150,
|
||||||
|
"jobs_completed": 140,
|
||||||
|
"jobs_failed": 5,
|
||||||
|
"jobs_running": 5,
|
||||||
|
"avg_job_duration_ms": 60000,
|
||||||
|
"queue_depth": {
|
||||||
|
"rss_collection": 10,
|
||||||
|
"google_search": 5,
|
||||||
|
"translation": 8,
|
||||||
|
"ai_generation": 12,
|
||||||
|
"image_generation": 15
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"created_at": "2024-01-01T14:00:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.6 Activity Logs Collection
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"_id": "ObjectId",
|
||||||
|
"user_id": "ObjectId",
|
||||||
|
"action": "service.start|news.create|pipeline.cancel|user.login",
|
||||||
|
"resource_type": "service|news_keyword|pipeline_job|user",
|
||||||
|
"resource_id": "ObjectId",
|
||||||
|
"details": {
|
||||||
|
"service_name": "rss-collector",
|
||||||
|
"previous_status": "stopped",
|
||||||
|
"new_status": "running"
|
||||||
|
},
|
||||||
|
"ip_address": "192.168.1.1",
|
||||||
|
"user_agent": "Mozilla/5.0...",
|
||||||
|
"status": "success|failure",
|
||||||
|
"error_message": null,
|
||||||
|
"created_at": "2024-01-01T00:00:00Z"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. API 설계
|
||||||
|
|
||||||
|
### 4.1 Authentication APIs
|
||||||
|
```
|
||||||
|
POST /api/v1/auth/register # 사용자 등록
|
||||||
|
POST /api/v1/auth/login # 로그인 (JWT 발급)
|
||||||
|
POST /api/v1/auth/refresh # Token 갱신
|
||||||
|
POST /api/v1/auth/logout # 로그아웃
|
||||||
|
GET /api/v1/auth/me # 현재 사용자 정보
|
||||||
|
POST /api/v1/auth/oauth/{provider} # OAuth 로그인 (Google, GitHub)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.2 Service Management APIs
|
||||||
|
```
|
||||||
|
GET /api/v1/services # 서비스 목록
|
||||||
|
GET /api/v1/services/{id} # 서비스 상세
|
||||||
|
POST /api/v1/services # 서비스 등록
|
||||||
|
PUT /api/v1/services/{id} # 서비스 수정
|
||||||
|
DELETE /api/v1/services/{id} # 서비스 삭제
|
||||||
|
POST /api/v1/services/{id}/start # 서비스 시작
|
||||||
|
POST /api/v1/services/{id}/stop # 서비스 중지
|
||||||
|
POST /api/v1/services/{id}/restart # 서비스 재시작
|
||||||
|
GET /api/v1/services/{id}/logs # 서비스 로그
|
||||||
|
GET /api/v1/services/{id}/metrics # 서비스 메트릭
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.3 News Keyword APIs
|
||||||
|
```
|
||||||
|
GET /api/v1/keywords # 키워드 목록
|
||||||
|
GET /api/v1/keywords/{id} # 키워드 상세
|
||||||
|
POST /api/v1/keywords # 키워드 생성
|
||||||
|
PUT /api/v1/keywords/{id} # 키워드 수정
|
||||||
|
DELETE /api/v1/keywords/{id} # 키워드 삭제
|
||||||
|
POST /api/v1/keywords/{id}/enable # 키워드 활성화
|
||||||
|
POST /api/v1/keywords/{id}/disable # 키워드 비활성화
|
||||||
|
GET /api/v1/keywords/{id}/stats # 키워드 통계
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.4 Pipeline Management APIs
|
||||||
|
```
|
||||||
|
GET /api/v1/pipelines # 파이프라인 작업 목록
|
||||||
|
GET /api/v1/pipelines/{id} # 파이프라인 작업 상세
|
||||||
|
POST /api/v1/pipelines # 파이프라인 작업 생성 (수동 트리거)
|
||||||
|
POST /api/v1/pipelines/{id}/cancel # 파이프라인 작업 취소
|
||||||
|
POST /api/v1/pipelines/{id}/retry # 파이프라인 작업 재시도
|
||||||
|
GET /api/v1/pipelines/queue # 큐 상태 조회
|
||||||
|
GET /api/v1/pipelines/realtime # 실시간 상태 (WebSocket)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.5 Dashboard APIs
|
||||||
|
```
|
||||||
|
GET /api/v1/dashboard/overview # 대시보드 개요
|
||||||
|
GET /api/v1/dashboard/services # 서비스 현황
|
||||||
|
GET /api/v1/dashboard/news # 뉴스 생성 현황
|
||||||
|
GET /api/v1/dashboard/pipeline # 파이프라인 현황
|
||||||
|
GET /api/v1/dashboard/alerts # 알림 및 경고
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.6 Statistics APIs
|
||||||
|
```
|
||||||
|
GET /api/v1/stats/users # 사용자 통계
|
||||||
|
GET /api/v1/stats/services # 서비스 통계
|
||||||
|
GET /api/v1/stats/news # 뉴스 통계
|
||||||
|
GET /api/v1/stats/pipeline # 파이프라인 통계
|
||||||
|
GET /api/v1/stats/trends # 트렌드 분석
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Frontend 페이지 구조
|
||||||
|
|
||||||
|
```
|
||||||
|
/
|
||||||
|
├── /login # 로그인 페이지
|
||||||
|
├── /register # 회원가입 페이지
|
||||||
|
├── /dashboard # 대시보드 (홈)
|
||||||
|
│ ├── Overview # 전체 현황
|
||||||
|
│ ├── Services Status # 서비스 상태
|
||||||
|
│ ├── News Generation # 뉴스 생성 현황
|
||||||
|
│ └── Pipeline Status # 파이프라인 현황
|
||||||
|
│
|
||||||
|
├── /services # 서비스 관리
|
||||||
|
│ ├── List # 서비스 목록
|
||||||
|
│ ├── Detail/{id} # 서비스 상세
|
||||||
|
│ ├── Create # 서비스 등록
|
||||||
|
│ ├── Edit/{id} # 서비스 수정
|
||||||
|
│ └── Logs/{id} # 서비스 로그
|
||||||
|
│
|
||||||
|
├── /keywords # 뉴스 키워드 관리
|
||||||
|
│ ├── List # 키워드 목록
|
||||||
|
│ ├── Detail/{id} # 키워드 상세
|
||||||
|
│ ├── Create # 키워드 생성
|
||||||
|
│ ├── Edit/{id} # 키워드 수정
|
||||||
|
│ └── Statistics/{id} # 키워드 통계
|
||||||
|
│
|
||||||
|
├── /pipeline # 파이프라인 관리
|
||||||
|
│ ├── Jobs # 작업 목록
|
||||||
|
│ ├── JobDetail/{id} # 작업 상세
|
||||||
|
│ ├── Monitor # 실시간 모니터링
|
||||||
|
│ └── Queue # 큐 상태
|
||||||
|
│
|
||||||
|
├── /statistics # 통계 및 분석
|
||||||
|
│ ├── Overview # 통계 개요
|
||||||
|
│ ├── Users # 사용자 통계
|
||||||
|
│ ├── Services # 서비스 통계
|
||||||
|
│ ├── News # 뉴스 통계
|
||||||
|
│ └── Trends # 트렌드 분석
|
||||||
|
│
|
||||||
|
└── /settings # 설정
|
||||||
|
├── Profile # 프로필
|
||||||
|
├── Security # 보안 설정
|
||||||
|
└── System # 시스템 설정
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. 기술 스택
|
||||||
|
|
||||||
|
### Backend
|
||||||
|
- **Framework**: FastAPI
|
||||||
|
- **Authentication**: OAuth2.0 + JWT (python-jose, passlib)
|
||||||
|
- **Database**: MongoDB (Motor - async driver)
|
||||||
|
- **Cache/Queue**: Redis
|
||||||
|
- **WebSocket**: FastAPI WebSocket
|
||||||
|
- **Kubernetes Client**: kubernetes-python
|
||||||
|
- **Validation**: Pydantic v2
|
||||||
|
|
||||||
|
### Frontend
|
||||||
|
- **Framework**: React 18 + TypeScript
|
||||||
|
- **State Management**: Redux Toolkit / Zustand
|
||||||
|
- **UI Library**: Material-UI v7 (MUI)
|
||||||
|
- **Routing**: React Router v6
|
||||||
|
- **API Client**: Axios / React Query
|
||||||
|
- **Real-time**: Socket.IO Client
|
||||||
|
- **Charts**: Recharts / Chart.js
|
||||||
|
- **Forms**: React Hook Form + Zod
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. 보안 고려사항
|
||||||
|
|
||||||
|
### 7.1 Authentication & Authorization
|
||||||
|
- JWT Token (Access + Refresh)
|
||||||
|
- OAuth2.0 (Google, GitHub, Azure AD)
|
||||||
|
- RBAC (Role-Based Access Control)
|
||||||
|
- Permission-based authorization
|
||||||
|
|
||||||
|
### 7.2 API Security
|
||||||
|
- Rate Limiting (per user/IP)
|
||||||
|
- CORS 설정
|
||||||
|
- Input Validation (Pydantic)
|
||||||
|
- SQL/NoSQL Injection 방어
|
||||||
|
- XSS/CSRF 방어
|
||||||
|
|
||||||
|
### 7.3 Data Security
|
||||||
|
- Password Hashing (bcrypt)
|
||||||
|
- Sensitive Data Encryption
|
||||||
|
- API Key Management (Secrets)
|
||||||
|
- Audit Logging
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. 구현 우선순위
|
||||||
|
|
||||||
|
### Phase 1: 기본 인프라 (Week 1-2)
|
||||||
|
1. ✅ Kubernetes 배포 완료
|
||||||
|
2. 🔄 Authentication System (OAuth2.0 + JWT)
|
||||||
|
3. 🔄 User Management (CRUD)
|
||||||
|
4. 🔄 Permission System (RBAC)
|
||||||
|
|
||||||
|
### Phase 2: 서비스 관리 (Week 3)
|
||||||
|
1. Service Management (CRUD)
|
||||||
|
2. Service Control (Start/Stop/Restart)
|
||||||
|
3. Service Monitoring (Health/Metrics)
|
||||||
|
4. Service Logs Viewer
|
||||||
|
|
||||||
|
### Phase 3: 뉴스 시스템 (Week 4)
|
||||||
|
1. Keyword Management (CRUD)
|
||||||
|
2. Keyword Configuration
|
||||||
|
3. Keyword Statistics
|
||||||
|
4. Article Management
|
||||||
|
|
||||||
|
### Phase 4: 파이프라인 관리 (Week 5)
|
||||||
|
1. Pipeline Job Tracking
|
||||||
|
2. Queue Management
|
||||||
|
3. Real-time Monitoring (WebSocket)
|
||||||
|
4. Pipeline Control (Cancel/Retry)
|
||||||
|
|
||||||
|
### Phase 5: 대시보드 & 통계 (Week 6)
|
||||||
|
1. Dashboard Overview
|
||||||
|
2. Real-time Status
|
||||||
|
3. Statistics & Analytics
|
||||||
|
4. Trend Analysis
|
||||||
|
|
||||||
|
### Phase 6: 최적화 & 테스트 (Week 7-8)
|
||||||
|
1. Performance Optimization
|
||||||
|
2. Unit/Integration Tests
|
||||||
|
3. Load Testing
|
||||||
|
4. Documentation
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. 다음 단계
|
||||||
|
|
||||||
|
현재 작업: **Phase 1 - Authentication System 구현**
|
||||||
|
|
||||||
|
1. Backend: Auth 모듈 구현
|
||||||
|
- JWT 토큰 발급/검증
|
||||||
|
- OAuth2.0 Provider 연동
|
||||||
|
- User CRUD API
|
||||||
|
- Permission System
|
||||||
|
|
||||||
|
2. Frontend: Auth UI 구현
|
||||||
|
- Login/Register 페이지
|
||||||
|
- OAuth 로그인 버튼
|
||||||
|
- Protected Routes
|
||||||
|
- User Context/Store
|
||||||
|
|
||||||
|
3. Database: Collections 생성
|
||||||
|
- Users Collection
|
||||||
|
- Sessions Collection (Redis)
|
||||||
|
- Activity Logs Collection
|
||||||
342
docs/DEPLOYMENT_GUIDE.md
Normal file
342
docs/DEPLOYMENT_GUIDE.md
Normal file
@ -0,0 +1,342 @@
|
|||||||
|
# Site11 배포 가이드
|
||||||
|
|
||||||
|
## 📋 목차
|
||||||
|
- [배포 아키텍처](#배포-아키텍처)
|
||||||
|
- [배포 옵션](#배포-옵션)
|
||||||
|
- [하이브리드 배포 (권장)](#하이브리드-배포-권장)
|
||||||
|
- [포트 구성](#포트-구성)
|
||||||
|
- [Health Check](#health-check)
|
||||||
|
- [문제 해결](#문제-해결)
|
||||||
|
|
||||||
|
## 배포 아키텍처
|
||||||
|
|
||||||
|
### 현재 구성: 하이브리드 아키텍처
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────┐
|
||||||
|
│ 사용자 브라우저 │
|
||||||
|
└────────────┬────────────────────┬──────────────────────┘
|
||||||
|
│ │
|
||||||
|
localhost:8080 localhost:8000
|
||||||
|
│ │
|
||||||
|
┌────────┴──────────┐ ┌──────┴──────────┐
|
||||||
|
│ kubectl │ │ kubectl │
|
||||||
|
│ port-forward │ │ port-forward │
|
||||||
|
└────────┬──────────┘ └──────┬──────────┘
|
||||||
|
│ │
|
||||||
|
┌────────┴──────────────────┴──────────┐
|
||||||
|
│ Kubernetes Cluster (Kind) │
|
||||||
|
│ │
|
||||||
|
│ ┌──────────────┐ ┌──────────────┐ │
|
||||||
|
│ │ Console │ │ Console │ │
|
||||||
|
│ │ Frontend │ │ Backend │ │
|
||||||
|
│ │ Service:3000 │ │ Service:8000 │ │
|
||||||
|
│ └──────┬───────┘ └──────┬───────┘ │
|
||||||
|
│ │ │ │
|
||||||
|
│ ┌──────┴───────┐ ┌──────┴───────┐ │
|
||||||
|
│ │ nginx:80 │ │ FastAPI:8000 │ │
|
||||||
|
│ │ (Pod) │ │ (Pod) │ │
|
||||||
|
│ └──────────────┘ └──────┬───────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ┌─────────────────────────┴───────┐ │
|
||||||
|
│ │ Pipeline Workers (5 Deployments) │ │
|
||||||
|
│ └──────────────┬──────────────────┘ │
|
||||||
|
└─────────────────┼──────────────────┘
|
||||||
|
│
|
||||||
|
host.docker.internal
|
||||||
|
│
|
||||||
|
┌─────────────────┴──────────────────┐
|
||||||
|
│ Docker Compose Infrastructure │
|
||||||
|
│ │
|
||||||
|
│ MongoDB | Redis | Kafka | Zookeeper│
|
||||||
|
│ Pipeline Scheduler | Monitor │
|
||||||
|
└──────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## 배포 옵션
|
||||||
|
|
||||||
|
### 옵션 1: 하이브리드 배포 (현재/권장)
|
||||||
|
- **Docker Compose**: 인프라 서비스 (MongoDB, Redis, Kafka)
|
||||||
|
- **Kubernetes**: 애플리케이션 및 파이프라인 워커
|
||||||
|
- **장점**: 프로덕션 환경과 유사, 확장성 우수
|
||||||
|
- **단점**: 설정 복잡도 높음
|
||||||
|
|
||||||
|
### 옵션 2: 전체 Docker Compose
|
||||||
|
- **모든 서비스를 Docker Compose로 실행**
|
||||||
|
- **장점**: 설정 간단, 로컬 개발에 최적
|
||||||
|
- **단점**: 오토스케일링 제한
|
||||||
|
|
||||||
|
### 옵션 3: 전체 Kubernetes
|
||||||
|
- **모든 서비스를 Kubernetes로 실행**
|
||||||
|
- **장점**: 완전한 클라우드 네이티브
|
||||||
|
- **단점**: 로컬 리소스 많이 필요
|
||||||
|
|
||||||
|
## 하이브리드 배포 (권장)
|
||||||
|
|
||||||
|
### 1. 인프라 시작 (Docker Compose)
|
||||||
|
```bash
|
||||||
|
# Docker Compose로 인프라 서비스 시작
|
||||||
|
docker-compose -f docker-compose-hybrid.yml up -d
|
||||||
|
|
||||||
|
# 상태 확인
|
||||||
|
docker-compose -f docker-compose-hybrid.yml ps
|
||||||
|
|
||||||
|
# 서비스 확인
|
||||||
|
docker ps | grep -E "mongodb|redis|kafka|zookeeper|scheduler|monitor"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Kubernetes 클러스터 준비
|
||||||
|
```bash
|
||||||
|
# Docker Desktop Kubernetes 활성화 또는 Kind 사용
|
||||||
|
# Docker Desktop: Preferences → Kubernetes → Enable Kubernetes
|
||||||
|
|
||||||
|
# 네임스페이스 생성
|
||||||
|
kubectl create namespace site11-pipeline
|
||||||
|
|
||||||
|
# ConfigMap 및 Secrets 생성
|
||||||
|
kubectl -n site11-pipeline apply -f k8s/pipeline/configmap.yaml
|
||||||
|
kubectl -n site11-pipeline apply -f k8s/pipeline/secrets.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. 애플리케이션 배포 (Docker Hub)
|
||||||
|
```bash
|
||||||
|
# Docker Hub에 이미지 푸시
|
||||||
|
export DOCKER_HUB_USER=yakenator
|
||||||
|
./deploy-dockerhub.sh
|
||||||
|
|
||||||
|
# Kubernetes에 배포
|
||||||
|
cd k8s/pipeline
|
||||||
|
for yaml in *-dockerhub.yaml; do
|
||||||
|
kubectl apply -f $yaml
|
||||||
|
done
|
||||||
|
|
||||||
|
# 배포 확인
|
||||||
|
kubectl -n site11-pipeline get deployments
|
||||||
|
kubectl -n site11-pipeline get pods
|
||||||
|
kubectl -n site11-pipeline get services
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Port Forwarding 설정
|
||||||
|
```bash
|
||||||
|
# 자동 스크립트 사용
|
||||||
|
./scripts/start-k8s-port-forward.sh
|
||||||
|
|
||||||
|
# 또는 수동 설정
|
||||||
|
kubectl -n site11-pipeline port-forward service/console-frontend 8080:3000 &
|
||||||
|
kubectl -n site11-pipeline port-forward service/console-backend 8000:8000 &
|
||||||
|
```
|
||||||
|
|
||||||
|
## 포트 구성
|
||||||
|
|
||||||
|
### 하이브리드 배포 포트 매핑
|
||||||
|
| 서비스 | 로컬 포트 | Service 포트 | Pod 포트 | 설명 |
|
||||||
|
|--------|----------|-------------|---------|------|
|
||||||
|
| Console Frontend | 8080 | 3000 | 80 | nginx 정적 파일 서빙 |
|
||||||
|
| Console Backend | 8000 | 8000 | 8000 | FastAPI API Gateway |
|
||||||
|
| Pipeline Monitor | 8100 | - | 8100 | Docker 직접 노출 |
|
||||||
|
| Pipeline Scheduler | 8099 | - | 8099 | Docker 직접 노출 |
|
||||||
|
| MongoDB | 27017 | - | 27017 | Docker 내부 |
|
||||||
|
| Redis | 6379 | - | 6379 | Docker 내부 |
|
||||||
|
| Kafka | 9092 | - | 9092 | Docker 내부 |
|
||||||
|
|
||||||
|
### Port Forward 체인
|
||||||
|
```
|
||||||
|
사용자 → localhost:8080 → kubectl port-forward → K8s Service:3000 → Pod nginx:80
|
||||||
|
```
|
||||||
|
|
||||||
|
## Health Check
|
||||||
|
|
||||||
|
### Console 서비스 Health Check
|
||||||
|
```bash
|
||||||
|
# Console Backend Health
|
||||||
|
curl http://localhost:8000/health
|
||||||
|
curl http://localhost:8000/api/health
|
||||||
|
|
||||||
|
# Console Frontend Health (HTML 응답)
|
||||||
|
curl http://localhost:8080/
|
||||||
|
|
||||||
|
# Users Service Health (via Console Backend)
|
||||||
|
curl http://localhost:8000/api/users/health
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pipeline 서비스 Health Check
|
||||||
|
```bash
|
||||||
|
# Pipeline Monitor
|
||||||
|
curl http://localhost:8100/health
|
||||||
|
|
||||||
|
# Pipeline Scheduler
|
||||||
|
curl http://localhost:8099/health
|
||||||
|
```
|
||||||
|
|
||||||
|
### Kubernetes Health Check
|
||||||
|
```bash
|
||||||
|
# Pod 상태
|
||||||
|
kubectl -n site11-pipeline get pods -o wide
|
||||||
|
|
||||||
|
# 서비스 엔드포인트
|
||||||
|
kubectl -n site11-pipeline get endpoints
|
||||||
|
|
||||||
|
# HPA 상태
|
||||||
|
kubectl -n site11-pipeline get hpa
|
||||||
|
|
||||||
|
# 이벤트 확인
|
||||||
|
kubectl -n site11-pipeline get events --sort-by='.lastTimestamp'
|
||||||
|
```
|
||||||
|
|
||||||
|
## 스케일링
|
||||||
|
|
||||||
|
### Horizontal Pod Autoscaler (HPA)
|
||||||
|
| 서비스 | 최소 | 최대 | CPU 목표 | 메모리 목표 |
|
||||||
|
|--------|-----|------|---------|------------|
|
||||||
|
| Console Frontend | 2 | 10 | 70% | 80% |
|
||||||
|
| Console Backend | 2 | 10 | 70% | 80% |
|
||||||
|
| RSS Collector | 1 | 5 | 70% | 80% |
|
||||||
|
| Google Search | 1 | 5 | 70% | 80% |
|
||||||
|
| Translator | 3 | 10 | 70% | 80% |
|
||||||
|
| AI Generator | 2 | 10 | 70% | 80% |
|
||||||
|
| Image Generator | 2 | 10 | 70% | 80% |
|
||||||
|
|
||||||
|
### 수동 스케일링
|
||||||
|
```bash
|
||||||
|
# 특정 디플로이먼트 스케일 조정
|
||||||
|
kubectl -n site11-pipeline scale deployment/pipeline-translator --replicas=5
|
||||||
|
|
||||||
|
# 모든 파이프라인 워커 스케일 업
|
||||||
|
for deploy in rss-collector google-search translator ai-article-generator image-generator; do
|
||||||
|
kubectl -n site11-pipeline scale deployment/pipeline-$deploy --replicas=3
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
## 모니터링
|
||||||
|
|
||||||
|
### 실시간 모니터링
|
||||||
|
```bash
|
||||||
|
# Pod 리소스 사용량
|
||||||
|
kubectl -n site11-pipeline top pods
|
||||||
|
|
||||||
|
# 로그 스트리밍
|
||||||
|
kubectl -n site11-pipeline logs -f deployment/console-backend
|
||||||
|
kubectl -n site11-pipeline logs -f deployment/pipeline-translator
|
||||||
|
|
||||||
|
# HPA 상태 감시
|
||||||
|
watch -n 2 kubectl -n site11-pipeline get hpa
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pipeline 모니터링
|
||||||
|
```bash
|
||||||
|
# Pipeline Monitor 웹 UI
|
||||||
|
open http://localhost:8100
|
||||||
|
|
||||||
|
# Queue 상태 확인
|
||||||
|
docker exec -it site11_redis redis-cli
|
||||||
|
> LLEN queue:translation
|
||||||
|
> LLEN queue:ai_generation
|
||||||
|
> LLEN queue:image_generation
|
||||||
|
```
|
||||||
|
|
||||||
|
## 문제 해결
|
||||||
|
|
||||||
|
### Pod가 시작되지 않을 때
|
||||||
|
```bash
|
||||||
|
# Pod 상세 정보
|
||||||
|
kubectl -n site11-pipeline describe pod <pod-name>
|
||||||
|
|
||||||
|
# 이미지 풀 에러 확인
|
||||||
|
kubectl -n site11-pipeline get events | grep -i pull
|
||||||
|
|
||||||
|
# 해결: Docker Hub 이미지 다시 푸시
|
||||||
|
docker push yakenator/site11-<service>:latest
|
||||||
|
kubectl -n site11-pipeline rollout restart deployment/<service>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Port Forward 연결 끊김
|
||||||
|
```bash
|
||||||
|
# 기존 port-forward 종료
|
||||||
|
pkill -f "kubectl.*port-forward"
|
||||||
|
|
||||||
|
# 다시 시작
|
||||||
|
./scripts/start-k8s-port-forward.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### 인프라 서비스 연결 실패
|
||||||
|
```bash
|
||||||
|
# Docker 네트워크 확인
|
||||||
|
docker network ls | grep site11
|
||||||
|
|
||||||
|
# K8s Pod에서 연결 테스트
|
||||||
|
kubectl -n site11-pipeline exec -it <pod-name> -- bash
|
||||||
|
> apt update && apt install -y netcat
|
||||||
|
> nc -zv host.docker.internal 6379 # Redis
|
||||||
|
> nc -zv host.docker.internal 27017 # MongoDB
|
||||||
|
```
|
||||||
|
|
||||||
|
### Health Check 실패
|
||||||
|
```bash
|
||||||
|
# Console Backend 로그 확인
|
||||||
|
kubectl -n site11-pipeline logs deployment/console-backend --tail=50
|
||||||
|
|
||||||
|
# 엔드포인트 직접 테스트
|
||||||
|
kubectl -n site11-pipeline exec -it deployment/console-backend -- curl localhost:8000/health
|
||||||
|
```
|
||||||
|
|
||||||
|
## 정리 및 초기화
|
||||||
|
|
||||||
|
### 전체 정리
|
||||||
|
```bash
|
||||||
|
# Kubernetes 리소스 삭제
|
||||||
|
kubectl delete namespace site11-pipeline
|
||||||
|
|
||||||
|
# Docker Compose 정리
|
||||||
|
docker-compose -f docker-compose-hybrid.yml down
|
||||||
|
|
||||||
|
# 볼륨 포함 완전 정리 (주의!)
|
||||||
|
docker-compose -f docker-compose-hybrid.yml down -v
|
||||||
|
```
|
||||||
|
|
||||||
|
### 선택적 정리
|
||||||
|
```bash
|
||||||
|
# 특정 디플로이먼트만 삭제
|
||||||
|
kubectl -n site11-pipeline delete deployment <name>
|
||||||
|
|
||||||
|
# 특정 Docker 서비스만 중지
|
||||||
|
docker-compose -f docker-compose-hybrid.yml stop mongodb
|
||||||
|
```
|
||||||
|
|
||||||
|
## 백업 및 복구
|
||||||
|
|
||||||
|
### MongoDB 백업
|
||||||
|
```bash
|
||||||
|
# 백업
|
||||||
|
docker exec site11_mongodb mongodump --archive=/tmp/backup.archive
|
||||||
|
docker cp site11_mongodb:/tmp/backup.archive ./backups/mongodb-$(date +%Y%m%d).archive
|
||||||
|
|
||||||
|
# 복구
|
||||||
|
docker cp ./backups/mongodb-20240101.archive site11_mongodb:/tmp/
|
||||||
|
docker exec site11_mongodb mongorestore --archive=/tmp/mongodb-20240101.archive
|
||||||
|
```
|
||||||
|
|
||||||
|
### 전체 설정 백업
|
||||||
|
```bash
|
||||||
|
# 설정 파일 백업
|
||||||
|
tar -czf config-backup-$(date +%Y%m%d).tar.gz \
|
||||||
|
k8s/ \
|
||||||
|
docker-compose*.yml \
|
||||||
|
.env \
|
||||||
|
registry/
|
||||||
|
```
|
||||||
|
|
||||||
|
## 다음 단계
|
||||||
|
|
||||||
|
1. **프로덕션 준비**
|
||||||
|
- Ingress Controller 설정
|
||||||
|
- SSL/TLS 인증서
|
||||||
|
- 외부 모니터링 통합
|
||||||
|
|
||||||
|
2. **성능 최적화**
|
||||||
|
- Registry Cache 활성화
|
||||||
|
- 빌드 캐시 최적화
|
||||||
|
- 리소스 리밋 조정
|
||||||
|
|
||||||
|
3. **보안 강화**
|
||||||
|
- Network Policy 적용
|
||||||
|
- RBAC 설정
|
||||||
|
- Secrets 암호화
|
||||||
393
docs/KIND_SETUP.md
Normal file
393
docs/KIND_SETUP.md
Normal file
@ -0,0 +1,393 @@
|
|||||||
|
# KIND (Kubernetes IN Docker) 개발 환경 설정
|
||||||
|
|
||||||
|
## 개요
|
||||||
|
Docker Desktop의 Kubernetes 대신 KIND를 사용하여 개발 환경을 구성합니다.
|
||||||
|
|
||||||
|
### KIND 선택 이유
|
||||||
|
1. **독립성**: Docker Desktop Kubernetes와 별도로 관리
|
||||||
|
2. **재현성**: 설정 파일로 클러스터 구성 관리
|
||||||
|
3. **멀티 노드**: 실제 프로덕션과 유사한 멀티 노드 환경
|
||||||
|
4. **빠른 재시작**: 필요시 클러스터 삭제/재생성 용이
|
||||||
|
5. **리소스 관리**: 노드별 리소스 할당 가능
|
||||||
|
|
||||||
|
## 사전 요구사항
|
||||||
|
|
||||||
|
### 1. KIND 설치
|
||||||
|
```bash
|
||||||
|
# macOS (Homebrew)
|
||||||
|
brew install kind
|
||||||
|
|
||||||
|
# 또는 직접 다운로드
|
||||||
|
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-darwin-amd64
|
||||||
|
chmod +x ./kind
|
||||||
|
sudo mv ./kind /usr/local/bin/kind
|
||||||
|
|
||||||
|
# 설치 확인
|
||||||
|
kind version
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. kubectl 설치
|
||||||
|
```bash
|
||||||
|
# macOS (Homebrew)
|
||||||
|
brew install kubectl
|
||||||
|
|
||||||
|
# 설치 확인
|
||||||
|
kubectl version --client
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Docker 실행 확인
|
||||||
|
```bash
|
||||||
|
docker ps
|
||||||
|
# Docker가 실행 중이어야 합니다
|
||||||
|
```
|
||||||
|
|
||||||
|
## 클러스터 구성
|
||||||
|
|
||||||
|
### 5-Node 클러스터 설정 파일
|
||||||
|
파일 위치: `k8s/kind-dev-cluster.yaml`
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
kind: Cluster
|
||||||
|
apiVersion: kind.x-k8s.io/v1alpha4
|
||||||
|
name: site11-dev
|
||||||
|
|
||||||
|
# 노드 구성
|
||||||
|
nodes:
|
||||||
|
# Control Plane (마스터 노드)
|
||||||
|
- role: control-plane
|
||||||
|
kubeadmConfigPatches:
|
||||||
|
- |
|
||||||
|
kind: InitConfiguration
|
||||||
|
nodeRegistration:
|
||||||
|
kubeletExtraArgs:
|
||||||
|
node-labels: "node-type=control-plane"
|
||||||
|
extraPortMappings:
|
||||||
|
# Console Frontend
|
||||||
|
- containerPort: 30080
|
||||||
|
hostPort: 3000
|
||||||
|
protocol: TCP
|
||||||
|
# Console Backend
|
||||||
|
- containerPort: 30081
|
||||||
|
hostPort: 8000
|
||||||
|
protocol: TCP
|
||||||
|
|
||||||
|
# Worker Node 1 (Console 서비스용)
|
||||||
|
- role: worker
|
||||||
|
labels:
|
||||||
|
workload: console
|
||||||
|
node-type: worker
|
||||||
|
kubeadmConfigPatches:
|
||||||
|
- |
|
||||||
|
kind: JoinConfiguration
|
||||||
|
nodeRegistration:
|
||||||
|
kubeletExtraArgs:
|
||||||
|
node-labels: "workload=console"
|
||||||
|
|
||||||
|
# Worker Node 2 (Pipeline 서비스용 - 수집)
|
||||||
|
- role: worker
|
||||||
|
labels:
|
||||||
|
workload: pipeline-collector
|
||||||
|
node-type: worker
|
||||||
|
kubeadmConfigPatches:
|
||||||
|
- |
|
||||||
|
kind: JoinConfiguration
|
||||||
|
nodeRegistration:
|
||||||
|
kubeletExtraArgs:
|
||||||
|
node-labels: "workload=pipeline-collector"
|
||||||
|
|
||||||
|
# Worker Node 3 (Pipeline 서비스용 - 처리)
|
||||||
|
- role: worker
|
||||||
|
labels:
|
||||||
|
workload: pipeline-processor
|
||||||
|
node-type: worker
|
||||||
|
kubeadmConfigPatches:
|
||||||
|
- |
|
||||||
|
kind: JoinConfiguration
|
||||||
|
nodeRegistration:
|
||||||
|
kubeletExtraArgs:
|
||||||
|
node-labels: "workload=pipeline-processor"
|
||||||
|
|
||||||
|
# Worker Node 4 (Pipeline 서비스용 - 생성)
|
||||||
|
- role: worker
|
||||||
|
labels:
|
||||||
|
workload: pipeline-generator
|
||||||
|
node-type: worker
|
||||||
|
kubeadmConfigPatches:
|
||||||
|
- |
|
||||||
|
kind: JoinConfiguration
|
||||||
|
nodeRegistration:
|
||||||
|
kubeletExtraArgs:
|
||||||
|
node-labels: "workload=pipeline-generator"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 노드 역할 분담
|
||||||
|
- **Control Plane**: 클러스터 관리, API 서버
|
||||||
|
- **Worker 1 (console)**: Console Backend, Console Frontend
|
||||||
|
- **Worker 2 (pipeline-collector)**: RSS Collector, Google Search
|
||||||
|
- **Worker 3 (pipeline-processor)**: Translator
|
||||||
|
- **Worker 4 (pipeline-generator)**: AI Article Generator, Image Generator
|
||||||
|
|
||||||
|
## 클러스터 관리 명령어
|
||||||
|
|
||||||
|
### 클러스터 생성
|
||||||
|
```bash
|
||||||
|
# KIND 클러스터 생성
|
||||||
|
kind create cluster --config k8s/kind-dev-cluster.yaml
|
||||||
|
|
||||||
|
# 생성 확인
|
||||||
|
kubectl cluster-info --context kind-site11-dev
|
||||||
|
kubectl get nodes
|
||||||
|
```
|
||||||
|
|
||||||
|
### 클러스터 삭제
|
||||||
|
```bash
|
||||||
|
# 클러스터 삭제
|
||||||
|
kind delete cluster --name site11-dev
|
||||||
|
|
||||||
|
# 모든 KIND 클러스터 확인
|
||||||
|
kind get clusters
|
||||||
|
```
|
||||||
|
|
||||||
|
### 컨텍스트 전환
|
||||||
|
```bash
|
||||||
|
# KIND 클러스터로 전환
|
||||||
|
kubectl config use-context kind-site11-dev
|
||||||
|
|
||||||
|
# 현재 컨텍스트 확인
|
||||||
|
kubectl config current-context
|
||||||
|
|
||||||
|
# 모든 컨텍스트 보기
|
||||||
|
kubectl config get-contexts
|
||||||
|
```
|
||||||
|
|
||||||
|
## 서비스 배포
|
||||||
|
|
||||||
|
### 1. Namespace 생성
|
||||||
|
```bash
|
||||||
|
# Console namespace
|
||||||
|
kubectl create namespace site11-console
|
||||||
|
|
||||||
|
# Pipeline namespace
|
||||||
|
kubectl create namespace site11-pipeline
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. ConfigMap 및 Secret 배포
|
||||||
|
```bash
|
||||||
|
# Pipeline 설정
|
||||||
|
kubectl apply -f k8s/pipeline/configmap-dockerhub.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. 서비스 배포
|
||||||
|
```bash
|
||||||
|
# Console 서비스
|
||||||
|
kubectl apply -f k8s/console/console-backend.yaml
|
||||||
|
kubectl apply -f k8s/console/console-frontend.yaml
|
||||||
|
|
||||||
|
# Pipeline 서비스
|
||||||
|
kubectl apply -f k8s/pipeline/rss-collector-dockerhub.yaml
|
||||||
|
kubectl apply -f k8s/pipeline/google-search-dockerhub.yaml
|
||||||
|
kubectl apply -f k8s/pipeline/translator-dockerhub.yaml
|
||||||
|
kubectl apply -f k8s/pipeline/ai-article-generator-dockerhub.yaml
|
||||||
|
kubectl apply -f k8s/pipeline/image-generator-dockerhub.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. 배포 확인
|
||||||
|
```bash
|
||||||
|
# Pod 상태 확인
|
||||||
|
kubectl -n site11-console get pods -o wide
|
||||||
|
kubectl -n site11-pipeline get pods -o wide
|
||||||
|
|
||||||
|
# Service 확인
|
||||||
|
kubectl -n site11-console get svc
|
||||||
|
kubectl -n site11-pipeline get svc
|
||||||
|
|
||||||
|
# 노드별 Pod 분포 확인
|
||||||
|
kubectl get pods -A -o wide
|
||||||
|
```
|
||||||
|
|
||||||
|
## 접속 방법
|
||||||
|
|
||||||
|
### NodePort 방식 (권장)
|
||||||
|
KIND 클러스터는 NodePort를 통해 서비스를 노출합니다.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# Console Frontend Service 예시
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: console-frontend
|
||||||
|
spec:
|
||||||
|
type: NodePort
|
||||||
|
ports:
|
||||||
|
- port: 80
|
||||||
|
targetPort: 80
|
||||||
|
nodePort: 30080 # http://localhost:3000
|
||||||
|
selector:
|
||||||
|
app: console-frontend
|
||||||
|
```
|
||||||
|
|
||||||
|
접속:
|
||||||
|
- Console Frontend: http://localhost:3000
|
||||||
|
- Console Backend: http://localhost:8000
|
||||||
|
|
||||||
|
### Port Forward 방식 (대안)
|
||||||
|
```bash
|
||||||
|
# Console Backend
|
||||||
|
kubectl -n site11-console port-forward svc/console-backend 8000:8000 &
|
||||||
|
|
||||||
|
# Console Frontend
|
||||||
|
kubectl -n site11-console port-forward svc/console-frontend 3000:80 &
|
||||||
|
```
|
||||||
|
|
||||||
|
## 모니터링
|
||||||
|
|
||||||
|
### 클러스터 상태
|
||||||
|
```bash
|
||||||
|
# 노드 상태
|
||||||
|
kubectl get nodes
|
||||||
|
|
||||||
|
# 전체 리소스
|
||||||
|
kubectl get all -A
|
||||||
|
|
||||||
|
# 특정 노드의 Pod
|
||||||
|
kubectl get pods -A -o wide | grep <node-name>
|
||||||
|
```
|
||||||
|
|
||||||
|
### 로그 확인
|
||||||
|
```bash
|
||||||
|
# Pod 로그
|
||||||
|
kubectl -n site11-console logs <pod-name>
|
||||||
|
|
||||||
|
# 실시간 로그
|
||||||
|
kubectl -n site11-console logs -f <pod-name>
|
||||||
|
|
||||||
|
# 이전 컨테이너 로그
|
||||||
|
kubectl -n site11-console logs <pod-name> --previous
|
||||||
|
```
|
||||||
|
|
||||||
|
### 리소스 사용량
|
||||||
|
```bash
|
||||||
|
# 노드 리소스
|
||||||
|
kubectl top nodes
|
||||||
|
|
||||||
|
# Pod 리소스
|
||||||
|
kubectl top pods -A
|
||||||
|
```
|
||||||
|
|
||||||
|
## 트러블슈팅
|
||||||
|
|
||||||
|
### 이미지 로드 문제
|
||||||
|
KIND는 로컬 이미지를 자동으로 로드하지 않습니다.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 로컬 이미지를 KIND로 로드
|
||||||
|
kind load docker-image yakenator/site11-console-backend:latest --name site11-dev
|
||||||
|
kind load docker-image yakenator/site11-console-frontend:latest --name site11-dev
|
||||||
|
|
||||||
|
# 또는 imagePullPolicy: Always 사용 (Docker Hub에서 자동 pull)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pod가 시작하지 않는 경우
|
||||||
|
```bash
|
||||||
|
# Pod 상태 확인
|
||||||
|
kubectl -n site11-console describe pod <pod-name>
|
||||||
|
|
||||||
|
# 이벤트 확인
|
||||||
|
kubectl -n site11-console get events --sort-by='.lastTimestamp'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 네트워크 문제
|
||||||
|
```bash
|
||||||
|
# Service endpoint 확인
|
||||||
|
kubectl -n site11-console get endpoints
|
||||||
|
|
||||||
|
# DNS 테스트
|
||||||
|
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup console-backend.site11-console.svc.cluster.local
|
||||||
|
```
|
||||||
|
|
||||||
|
## 개발 워크플로우
|
||||||
|
|
||||||
|
### 1. 코드 변경 후 재배포
|
||||||
|
```bash
|
||||||
|
# Docker 이미지 빌드
|
||||||
|
docker build -t yakenator/site11-console-backend:latest -f services/console/backend/Dockerfile services/console/backend
|
||||||
|
|
||||||
|
# Docker Hub에 푸시
|
||||||
|
docker push yakenator/site11-console-backend:latest
|
||||||
|
|
||||||
|
# Pod 재시작 (새 이미지 pull)
|
||||||
|
kubectl -n site11-console rollout restart deployment console-backend
|
||||||
|
|
||||||
|
# 또는 Pod 삭제 (자동 재생성)
|
||||||
|
kubectl -n site11-console delete pod -l app=console-backend
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. 로컬 개발 (빠른 테스트)
|
||||||
|
```bash
|
||||||
|
# 로컬에서 서비스 실행
|
||||||
|
cd services/console/backend
|
||||||
|
uvicorn app.main:app --reload --port 8000
|
||||||
|
|
||||||
|
# KIND 클러스터의 MongoDB 접속
|
||||||
|
kubectl -n site11-console port-forward svc/mongodb 27017:27017
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. 클러스터 리셋
|
||||||
|
```bash
|
||||||
|
# 전체 재생성
|
||||||
|
kind delete cluster --name site11-dev
|
||||||
|
kind create cluster --config k8s/kind-dev-cluster.yaml
|
||||||
|
|
||||||
|
# 서비스 재배포
|
||||||
|
kubectl apply -f k8s/console/
|
||||||
|
kubectl apply -f k8s/pipeline/
|
||||||
|
```
|
||||||
|
|
||||||
|
## 성능 최적화
|
||||||
|
|
||||||
|
### 노드 리소스 제한 (선택사항)
|
||||||
|
```yaml
|
||||||
|
nodes:
|
||||||
|
- role: worker
|
||||||
|
extraMounts:
|
||||||
|
- hostPath: /path/to/data
|
||||||
|
containerPath: /data
|
||||||
|
kubeadmConfigPatches:
|
||||||
|
- |
|
||||||
|
kind: JoinConfiguration
|
||||||
|
nodeRegistration:
|
||||||
|
kubeletExtraArgs:
|
||||||
|
max-pods: "50"
|
||||||
|
cpu-manager-policy: "static"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 이미지 Pull 정책
|
||||||
|
```yaml
|
||||||
|
# Deployment에서 설정
|
||||||
|
spec:
|
||||||
|
template:
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: console-backend
|
||||||
|
image: yakenator/site11-console-backend:latest
|
||||||
|
imagePullPolicy: Always # 항상 최신 이미지
|
||||||
|
```
|
||||||
|
|
||||||
|
## 백업 및 복원
|
||||||
|
|
||||||
|
### 클러스터 설정 백업
|
||||||
|
```bash
|
||||||
|
# 현재 리소스 백업
|
||||||
|
kubectl get all -A -o yaml > backup-$(date +%Y%m%d).yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
### 복원
|
||||||
|
```bash
|
||||||
|
# 백업에서 복원
|
||||||
|
kubectl apply -f backup-20251028.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
## 참고 자료
|
||||||
|
- KIND 공식 문서: https://kind.sigs.k8s.io/
|
||||||
|
- Kubernetes 공식 문서: https://kubernetes.io/docs/
|
||||||
|
- KIND GitHub: https://github.com/kubernetes-sigs/kind
|
||||||
339
docs/PROGRESS.md
339
docs/PROGRESS.md
@ -5,123 +5,312 @@
|
|||||||
|
|
||||||
## Current Status
|
## Current Status
|
||||||
- **Date Started**: 2025-09-09
|
- **Date Started**: 2025-09-09
|
||||||
- **Current Phase**: Step 3 Complete ✅
|
- **Last Updated**: 2025-10-28
|
||||||
- **Next Action**: Step 4 - Frontend Skeleton
|
- **Current Phase**: KIND Cluster Setup Complete ✅
|
||||||
|
- **Next Action**: Phase 2 - Frontend UI Implementation
|
||||||
|
|
||||||
## Completed Checkpoints
|
## Completed Checkpoints
|
||||||
|
|
||||||
|
### Phase 1: Authentication System (OAuth2.0 + JWT) ✅
|
||||||
|
**Completed Date**: 2025-10-28
|
||||||
|
|
||||||
|
#### Backend (FastAPI + MongoDB)
|
||||||
|
✅ JWT token system (access + refresh tokens)
|
||||||
|
✅ User authentication and registration
|
||||||
|
✅ Password hashing with bcrypt
|
||||||
|
✅ Protected endpoints with JWT middleware
|
||||||
|
✅ Token refresh mechanism
|
||||||
|
✅ Role-Based Access Control (RBAC) structure
|
||||||
|
✅ MongoDB integration with Motor (async driver)
|
||||||
|
✅ Pydantic v2 models and schemas
|
||||||
|
✅ Docker image built and pushed
|
||||||
|
✅ Deployed to Kubernetes (site11-pipeline namespace)
|
||||||
|
|
||||||
|
**API Endpoints**:
|
||||||
|
- POST `/api/auth/register` - User registration
|
||||||
|
- POST `/api/auth/login` - User login (returns access + refresh tokens)
|
||||||
|
- GET `/api/auth/me` - Get current user (protected)
|
||||||
|
- POST `/api/auth/refresh` - Refresh access token
|
||||||
|
- POST `/api/auth/logout` - Logout
|
||||||
|
|
||||||
|
**Docker Image**: `yakenator/site11-console-backend:latest`
|
||||||
|
|
||||||
|
#### Frontend (React + TypeScript + Material-UI)
|
||||||
|
✅ Login page component
|
||||||
|
✅ Register page component
|
||||||
|
✅ AuthContext for global state management
|
||||||
|
✅ API client with Axios interceptors
|
||||||
|
✅ Automatic token refresh on 401
|
||||||
|
✅ Protected routes implementation
|
||||||
|
✅ User info display in navigation bar
|
||||||
|
✅ Logout functionality
|
||||||
|
✅ Docker image built and pushed
|
||||||
|
✅ Deployed to Kubernetes (site11-pipeline namespace)
|
||||||
|
|
||||||
|
**Docker Image**: `yakenator/site11-console-frontend:latest`
|
||||||
|
|
||||||
|
#### Files Created/Modified
|
||||||
|
|
||||||
|
**Backend Files**:
|
||||||
|
- `/services/console/backend/app/core/config.py` - Settings with pydantic-settings
|
||||||
|
- `/services/console/backend/app/core/security.py` - JWT & bcrypt password hashing
|
||||||
|
- `/services/console/backend/app/db/mongodb.py` - MongoDB connection manager
|
||||||
|
- `/services/console/backend/app/models/user.py` - User model with Pydantic v2
|
||||||
|
- `/services/console/backend/app/schemas/auth.py` - Auth request/response schemas
|
||||||
|
- `/services/console/backend/app/services/user_service.py` - User business logic
|
||||||
|
- `/services/console/backend/app/routes/auth.py` - Authentication endpoints
|
||||||
|
- `/services/console/backend/requirements.txt` - Updated with Motor, bcrypt
|
||||||
|
|
||||||
|
**Frontend Files**:
|
||||||
|
- `/services/console/frontend/src/types/auth.ts` - TypeScript types
|
||||||
|
- `/services/console/frontend/src/api/auth.ts` - API client with interceptors
|
||||||
|
- `/services/console/frontend/src/contexts/AuthContext.tsx` - Auth state management
|
||||||
|
- `/services/console/frontend/src/pages/Login.tsx` - Login page
|
||||||
|
- `/services/console/frontend/src/pages/Register.tsx` - Register page
|
||||||
|
- `/services/console/frontend/src/components/ProtectedRoute.tsx` - Route guard
|
||||||
|
- `/services/console/frontend/src/components/Layout.tsx` - Updated with logout
|
||||||
|
- `/services/console/frontend/src/App.tsx` - Router configuration
|
||||||
|
- `/services/console/frontend/src/vite-env.d.ts` - Vite types
|
||||||
|
|
||||||
|
**Documentation**:
|
||||||
|
- `/docs/CONSOLE_ARCHITECTURE.md` - Complete system architecture
|
||||||
|
|
||||||
|
#### Technical Achievements
|
||||||
|
- Fixed bcrypt 72-byte limit issue by using native bcrypt library
|
||||||
|
- Resolved Pydantic v2 compatibility (PyObjectId, ConfigDict)
|
||||||
|
- Implemented automatic token refresh with axios interceptors
|
||||||
|
- Protected routes with loading states
|
||||||
|
- Nginx reverse proxy configuration for API
|
||||||
|
|
||||||
|
#### Testing Results
|
||||||
|
All authentication endpoints tested and working:
|
||||||
|
- ✅ User registration with validation
|
||||||
|
- ✅ User login with JWT tokens
|
||||||
|
- ✅ Protected endpoint access with token
|
||||||
|
- ✅ Token refresh mechanism
|
||||||
|
- ✅ Invalid credentials rejection
|
||||||
|
- ✅ Duplicate email prevention
|
||||||
|
- ✅ Unauthorized access blocking
|
||||||
|
|
||||||
|
### Phase 2: Service Management CRUD 🔄
|
||||||
|
**Started Date**: 2025-10-28
|
||||||
|
**Status**: Backend Complete, Frontend In Progress
|
||||||
|
|
||||||
|
#### Backend (FastAPI + MongoDB) ✅
|
||||||
|
✅ Service model with comprehensive fields
|
||||||
|
✅ Service CRUD API endpoints (Create, Read, Update, Delete)
|
||||||
|
✅ Health check mechanism with httpx
|
||||||
|
✅ Response time measurement
|
||||||
|
✅ Status tracking (healthy/unhealthy/unknown)
|
||||||
|
✅ Service type categorization (backend, frontend, database, etc.)
|
||||||
|
|
||||||
|
**API Endpoints**:
|
||||||
|
- GET `/api/services` - Get all services
|
||||||
|
- POST `/api/services` - Create new service
|
||||||
|
- GET `/api/services/{id}` - Get service by ID
|
||||||
|
- PUT `/api/services/{id}` - Update service
|
||||||
|
- DELETE `/api/services/{id}` - Delete service
|
||||||
|
- POST `/api/services/{id}/health-check` - Check specific service health
|
||||||
|
- POST `/api/services/health-check/all` - Check all services health
|
||||||
|
|
||||||
|
**Files Created**:
|
||||||
|
- `/services/console/backend/app/models/service.py` - Service model
|
||||||
|
- `/services/console/backend/app/schemas/service.py` - Service schemas
|
||||||
|
- `/services/console/backend/app/services/service_service.py` - Business logic
|
||||||
|
- `/services/console/backend/app/routes/services.py` - API routes
|
||||||
|
|
||||||
|
#### Frontend (React + TypeScript) 🔄
|
||||||
|
✅ TypeScript type definitions
|
||||||
|
✅ Service API client
|
||||||
|
⏳ Services list page (pending)
|
||||||
|
⏳ Add/Edit service modal (pending)
|
||||||
|
⏳ Health status display (pending)
|
||||||
|
|
||||||
|
**Files Created**:
|
||||||
|
- `/services/console/frontend/src/types/service.ts` - TypeScript types
|
||||||
|
- `/services/console/frontend/src/api/service.ts` - API client
|
||||||
|
|
||||||
|
### KIND Cluster Setup (Local Development Environment) ✅
|
||||||
|
**Completed Date**: 2025-10-28
|
||||||
|
|
||||||
|
#### Infrastructure Setup
|
||||||
|
✅ KIND (Kubernetes IN Docker) 5-node cluster
|
||||||
|
✅ Cluster configuration with role-based workers
|
||||||
|
✅ NodePort mappings for console access (30080, 30081)
|
||||||
|
✅ Namespace separation (site11-console, site11-pipeline)
|
||||||
|
✅ MongoDB and Redis deployed in cluster
|
||||||
|
✅ Console backend and frontend deployed with NodePort services
|
||||||
|
✅ All 4 pods running successfully
|
||||||
|
|
||||||
|
#### Management Tools
|
||||||
|
✅ `kind-setup.sh` script for cluster management
|
||||||
|
✅ `docker-compose.kubernetes.yml` for monitoring
|
||||||
|
✅ Comprehensive documentation (KUBERNETES.md, KIND_SETUP.md)
|
||||||
|
|
||||||
|
#### Kubernetes Resources Created
|
||||||
|
- **Cluster Config**: `/k8s/kind-dev-cluster.yaml`
|
||||||
|
- **Console MongoDB/Redis**: `/k8s/kind/console-mongodb-redis.yaml`
|
||||||
|
- **Console Backend**: `/k8s/kind/console-backend.yaml`
|
||||||
|
- **Console Frontend**: `/k8s/kind/console-frontend.yaml`
|
||||||
|
- **Management Script**: `/scripts/kind-setup.sh`
|
||||||
|
- **Docker Compose**: `/docker-compose.kubernetes.yml`
|
||||||
|
- **Documentation**: `/KUBERNETES.md`
|
||||||
|
|
||||||
|
#### Verification Results
|
||||||
|
✅ Cluster created with 5 nodes (all Ready)
|
||||||
|
✅ Console namespace with 4 running pods
|
||||||
|
✅ NodePort services accessible (3000, 8000)
|
||||||
|
✅ Frontend login/register tested successfully
|
||||||
|
✅ Backend API health check passed
|
||||||
|
✅ Authentication system working in KIND cluster
|
||||||
|
|
||||||
|
### Earlier Checkpoints
|
||||||
✅ Project structure planning (CLAUDE.md)
|
✅ Project structure planning (CLAUDE.md)
|
||||||
✅ Implementation plan created (docs/PLAN.md)
|
✅ Implementation plan created (docs/PLAN.md)
|
||||||
✅ Progressive approach defined
|
✅ Progressive approach defined
|
||||||
✅ Step 1: Minimal Foundation - Docker + Console Hello World
|
✅ Step 1: Minimal Foundation - Docker + Console Hello World
|
||||||
- docker-compose.yml created
|
|
||||||
- console/backend with FastAPI
|
|
||||||
- Running on port 8011
|
|
||||||
✅ Step 2: Add First Service (Users)
|
✅ Step 2: Add First Service (Users)
|
||||||
- Users service with CRUD operations
|
|
||||||
- Console API Gateway routing to Users
|
|
||||||
- Service communication verified
|
|
||||||
- Test: curl http://localhost:8011/api/users/users
|
|
||||||
✅ Step 3: Database Integration
|
✅ Step 3: Database Integration
|
||||||
- MongoDB and Redis containers added
|
|
||||||
- Users service using MongoDB with Beanie ODM
|
|
||||||
- Data persistence verified
|
|
||||||
- MongoDB IDs: 68c126c0bbbe52be68495933
|
|
||||||
|
|
||||||
## Active Working Files
|
## Active Working Files
|
||||||
```
|
```
|
||||||
현재 작업 중인 주요 파일:
|
주요 작업 파일:
|
||||||
|
- /services/console/backend/ (Console Backend - FastAPI)
|
||||||
|
- /services/console/frontend/ (Console Frontend - React + TypeScript)
|
||||||
|
- /docs/CONSOLE_ARCHITECTURE.md (시스템 아키텍처)
|
||||||
- /docs/PLAN.md (구현 계획)
|
- /docs/PLAN.md (구현 계획)
|
||||||
- /CLAUDE.md (아키텍처 가이드)
|
|
||||||
- /docs/PROGRESS.md (이 파일)
|
- /docs/PROGRESS.md (이 파일)
|
||||||
|
- /CLAUDE.md (개발 가이드라인)
|
||||||
```
|
```
|
||||||
|
|
||||||
## Next Immediate Steps
|
## Deployment Status
|
||||||
|
|
||||||
|
### KIND Cluster: site11-dev ✅
|
||||||
|
**Cluster Created**: 2025-10-28
|
||||||
|
**Nodes**: 5 (1 control-plane + 4 workers)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 다음 작업 시작 명령
|
# Console Namespace
|
||||||
# Step 1: Create docker-compose.yml
|
kubectl -n site11-console get pods
|
||||||
# Step 2: Create console/backend/main.py
|
# Status: 4/4 Running (mongodb, redis, console-backend, console-frontend)
|
||||||
# Step 3: Test with docker-compose up
|
|
||||||
|
# Cluster Status
|
||||||
|
./scripts/kind-setup.sh status
|
||||||
|
|
||||||
|
# Management
|
||||||
|
./scripts/kind-setup.sh {create|delete|deploy-console|status|logs|access|setup}
|
||||||
```
|
```
|
||||||
|
|
||||||
## Code Snippets Ready to Use
|
### Access URLs (NodePort)
|
||||||
|
- Frontend: http://localhost:3000 (NodePort 30080)
|
||||||
|
- Backend API: http://localhost:8000 (NodePort 30081)
|
||||||
|
- Backend Health: http://localhost:8000/health
|
||||||
|
- API Docs: http://localhost:8000/docs
|
||||||
|
|
||||||
### 1. Minimal docker-compose.yml
|
### Monitoring
|
||||||
```yaml
|
```bash
|
||||||
version: '3.8'
|
# Start monitoring
|
||||||
services:
|
docker-compose -f docker-compose.kubernetes.yml up -d
|
||||||
console:
|
docker-compose -f docker-compose.kubernetes.yml logs -f kind-monitor
|
||||||
build: ./console/backend
|
|
||||||
ports:
|
|
||||||
- "8000:8000"
|
|
||||||
environment:
|
|
||||||
- ENV=development
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2. Console main.py starter
|
## Next Immediate Steps (Phase 2)
|
||||||
```python
|
|
||||||
from fastapi import FastAPI
|
|
||||||
app = FastAPI(title="Console API Gateway")
|
|
||||||
|
|
||||||
@app.get("/health")
|
### Service Management CRUD
|
||||||
async def health():
|
```
|
||||||
return {"status": "healthy", "service": "console"}
|
1. Backend API for service management
|
||||||
|
- Service model (name, url, status, health_endpoint)
|
||||||
|
- CRUD endpoints
|
||||||
|
- Health check mechanism
|
||||||
|
|
||||||
|
2. Frontend Service Management UI
|
||||||
|
- Service list page
|
||||||
|
- Add/Edit service form
|
||||||
|
- Service status display
|
||||||
|
- Health monitoring
|
||||||
|
|
||||||
|
3. Service Discovery & Registry
|
||||||
|
- Auto-discovery of services
|
||||||
|
- Heartbeat mechanism
|
||||||
|
- Status dashboard
|
||||||
```
|
```
|
||||||
|
|
||||||
## Important Decisions Made
|
## Important Decisions Made
|
||||||
1. **Architecture**: API Gateway Pattern with Console as orchestrator
|
1. **Architecture**: API Gateway Pattern with Console as orchestrator
|
||||||
2. **Tech Stack**: FastAPI + React + MongoDB + Redis + Docker
|
2. **Tech Stack**: FastAPI + React + MongoDB + Redis + Docker + Kubernetes
|
||||||
3. **Approach**: Progressive implementation (simple to complex)
|
3. **Authentication**: JWT with access/refresh tokens
|
||||||
4. **First Service**: Users service after Console
|
4. **Password Security**: bcrypt (not passlib)
|
||||||
|
5. **Frontend State**: React Context API (not Redux)
|
||||||
|
6. **API Client**: Axios with interceptors for token management
|
||||||
|
7. **Deployment**: Kubernetes on Docker Desktop
|
||||||
|
8. **Docker Registry**: Docker Hub (yakenator)
|
||||||
|
|
||||||
## Questions to Ask When Resuming
|
## Questions to Ask When Resuming
|
||||||
새로운 세션에서 이어서 작업할 때 확인할 사항:
|
새로운 세션에서 이어서 작업할 때 확인할 사항:
|
||||||
1. "PROGRESS.md 파일을 확인했나요?"
|
1. "Phase 1 (Authentication) 완료 확인?"
|
||||||
2. "마지막으로 완료한 Step은 무엇인가요?"
|
2. "Kubernetes 클러스터 정상 동작 중?"
|
||||||
3. "현재 에러나 블로킹 이슈가 있나요?"
|
3. "다음 Phase 2 (Service Management) 시작할까요?"
|
||||||
|
|
||||||
## Git Commits Pattern
|
## Git Workflow
|
||||||
각 Step 완료 시 커밋 메시지:
|
```bash
|
||||||
```
|
# Current branch
|
||||||
Step X: [간단한 설명]
|
main
|
||||||
- 구현 내용 1
|
|
||||||
- 구현 내용 2
|
|
||||||
```
|
|
||||||
|
|
||||||
## Directory Structure Snapshot
|
# Commit pattern
|
||||||
```
|
git add .
|
||||||
site11/
|
git commit -m "feat: Phase 1 - Complete authentication system
|
||||||
├── CLAUDE.md ✅ Created
|
|
||||||
├── docs/
|
- Backend: JWT auth with FastAPI + MongoDB
|
||||||
│ ├── PLAN.md ✅ Created
|
- Frontend: Login/Register with React + TypeScript
|
||||||
│ └── PROGRESS.md ✅ Created (this file)
|
- Docker images built and deployed to Kubernetes
|
||||||
├── console/ 🔄 Next
|
- All authentication endpoints tested
|
||||||
│ └── backend/
|
|
||||||
│ └── main.py
|
🤖 Generated with Claude Code
|
||||||
└── docker-compose.yml 🔄 Next
|
Co-Authored-By: Claude <noreply@anthropic.com>"
|
||||||
|
|
||||||
|
git push origin main
|
||||||
```
|
```
|
||||||
|
|
||||||
## Context Recovery Commands
|
## Context Recovery Commands
|
||||||
새 세션에서 빠르게 상황 파악하기:
|
새 세션에서 빠르게 상황 파악하기:
|
||||||
```bash
|
```bash
|
||||||
# 1. 현재 구조 확인
|
# 1. 현재 구조 확인
|
||||||
ls -la
|
ls -la services/console/
|
||||||
|
|
||||||
# 2. 진행 상황 확인
|
# 2. 진행 상황 확인
|
||||||
cat docs/PROGRESS.md
|
cat docs/PROGRESS.md | grep "Current Phase"
|
||||||
|
|
||||||
# 3. 다음 단계 확인
|
# 3. Kubernetes 상태 확인
|
||||||
grep "Step" docs/PLAN.md | head -5
|
kubectl -n site11-pipeline get pods
|
||||||
|
|
||||||
# 4. 실행 중인 컨테이너 확인
|
# 4. Docker 이미지 확인
|
||||||
docker ps
|
docker images | grep console
|
||||||
|
|
||||||
|
# 5. Git 상태 확인
|
||||||
|
git status
|
||||||
|
git log --oneline -5
|
||||||
```
|
```
|
||||||
|
|
||||||
## Error Log
|
## Troubleshooting Log
|
||||||
문제 발생 시 여기에 기록:
|
|
||||||
- (아직 없음)
|
### Issue 1: Bcrypt 72-byte limit
|
||||||
|
**Error**: `ValueError: password cannot be longer than 72 bytes`
|
||||||
|
**Solution**: Replaced `passlib[bcrypt]` with native `bcrypt==4.1.2`
|
||||||
|
**Status**: ✅ Resolved
|
||||||
|
|
||||||
|
### Issue 2: Pydantic v2 incompatibility
|
||||||
|
**Error**: `__modify_schema__` not supported
|
||||||
|
**Solution**: Updated to `__get_pydantic_core_schema__` and `model_config = ConfigDict(...)`
|
||||||
|
**Status**: ✅ Resolved
|
||||||
|
|
||||||
|
### Issue 3: Port forwarding disconnections
|
||||||
|
**Error**: Lost connection to pod
|
||||||
|
**Solution**: Kill kubectl processes and restart port forwarding
|
||||||
|
**Status**: ⚠️ Known issue (Kubernetes restarts)
|
||||||
|
|
||||||
## Notes for Next Session
|
## Notes for Next Session
|
||||||
- Step 1부터 시작
|
- Phase 1 완료! Authentication 시스템 완전히 작동함
|
||||||
- docker-compose.yml 생성 필요
|
- 모든 코드는 services/console/ 디렉토리에 있음
|
||||||
- console/backend/main.py 생성 필요
|
- Docker 이미지는 yakenator/site11-console-* 로 푸시됨
|
||||||
- 모든 문서 파일은 대문자.md 형식으로 생성 (예: README.md, SETUP.md)
|
- Kubernetes에 배포되어 있음 (site11-pipeline namespace)
|
||||||
|
- Phase 2: Service Management CRUD 구현 시작 가능
|
||||||
|
|||||||
300
docs/QUICK_REFERENCE.md
Normal file
300
docs/QUICK_REFERENCE.md
Normal file
@ -0,0 +1,300 @@
|
|||||||
|
# Site11 빠른 참조 가이드
|
||||||
|
|
||||||
|
## 🚀 빠른 시작
|
||||||
|
|
||||||
|
### 전체 시스템 시작
|
||||||
|
```bash
|
||||||
|
# 1. 인프라 시작 (Docker)
|
||||||
|
docker-compose -f docker-compose-hybrid.yml up -d
|
||||||
|
|
||||||
|
# 2. 애플리케이션 배포 (Kubernetes)
|
||||||
|
./deploy-dockerhub.sh
|
||||||
|
|
||||||
|
# 3. 포트 포워딩 시작
|
||||||
|
./scripts/start-k8s-port-forward.sh
|
||||||
|
|
||||||
|
# 4. 상태 확인
|
||||||
|
./scripts/status-check.sh
|
||||||
|
|
||||||
|
# 5. 브라우저에서 확인
|
||||||
|
open http://localhost:8080
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📊 주요 엔드포인트
|
||||||
|
|
||||||
|
| 서비스 | URL | 설명 |
|
||||||
|
|--------|-----|------|
|
||||||
|
| Console Frontend | http://localhost:8080 | 관리 대시보드 |
|
||||||
|
| Console Backend | http://localhost:8000 | API Gateway |
|
||||||
|
| Health Check | http://localhost:8000/health | 백엔드 상태 |
|
||||||
|
| API Health | http://localhost:8000/api/health | API 상태 |
|
||||||
|
| Users Health | http://localhost:8000/api/users/health | 사용자 서비스 상태 |
|
||||||
|
| Pipeline Monitor | http://localhost:8100 | 파이프라인 모니터링 |
|
||||||
|
| Pipeline Scheduler | http://localhost:8099 | 스케줄러 상태 |
|
||||||
|
|
||||||
|
## 🔧 주요 명령어
|
||||||
|
|
||||||
|
### Docker 관리
|
||||||
|
```bash
|
||||||
|
# 전체 서비스 상태
|
||||||
|
docker-compose -f docker-compose-hybrid.yml ps
|
||||||
|
|
||||||
|
# 특정 서비스 로그
|
||||||
|
docker-compose -f docker-compose-hybrid.yml logs -f pipeline-scheduler
|
||||||
|
|
||||||
|
# 서비스 재시작
|
||||||
|
docker-compose -f docker-compose-hybrid.yml restart mongodb
|
||||||
|
|
||||||
|
# 정리
|
||||||
|
docker-compose -f docker-compose-hybrid.yml down
|
||||||
|
```
|
||||||
|
|
||||||
|
### Kubernetes 관리
|
||||||
|
```bash
|
||||||
|
# Pod 상태 확인
|
||||||
|
kubectl -n site11-pipeline get pods
|
||||||
|
|
||||||
|
# 서비스 상태 확인
|
||||||
|
kubectl -n site11-pipeline get services
|
||||||
|
|
||||||
|
# HPA 상태 확인
|
||||||
|
kubectl -n site11-pipeline get hpa
|
||||||
|
|
||||||
|
# 특정 Pod 로그
|
||||||
|
kubectl -n site11-pipeline logs -f deployment/console-backend
|
||||||
|
|
||||||
|
# Pod 재시작
|
||||||
|
kubectl -n site11-pipeline rollout restart deployment/console-backend
|
||||||
|
```
|
||||||
|
|
||||||
|
### 시스템 상태 확인
|
||||||
|
```bash
|
||||||
|
# 전체 상태 체크
|
||||||
|
./scripts/status-check.sh
|
||||||
|
|
||||||
|
# 포트 포워딩 상태
|
||||||
|
ps aux | grep "kubectl.*port-forward"
|
||||||
|
|
||||||
|
# 리소스 사용량
|
||||||
|
kubectl -n site11-pipeline top pods
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🗃️ 데이터베이스 관리
|
||||||
|
|
||||||
|
### MongoDB
|
||||||
|
```bash
|
||||||
|
# MongoDB 접속
|
||||||
|
docker exec -it site11_mongodb mongosh
|
||||||
|
|
||||||
|
# 데이터베이스 사용
|
||||||
|
use ai_writer_db
|
||||||
|
|
||||||
|
# 컬렉션 목록
|
||||||
|
show collections
|
||||||
|
|
||||||
|
# 기사 수 확인
|
||||||
|
db.articles_ko.countDocuments()
|
||||||
|
|
||||||
|
# 언어별 동기화 상태 확인
|
||||||
|
docker exec site11_mongodb mongosh ai_writer_db --quiet --eval '
|
||||||
|
var ko_count = db.articles_ko.countDocuments({});
|
||||||
|
var collections = ["articles_en", "articles_zh_cn", "articles_zh_tw", "articles_ja"];
|
||||||
|
collections.forEach(function(coll) {
|
||||||
|
var count = db[coll].countDocuments({});
|
||||||
|
print(coll + ": " + count + " (" + (ko_count - count) + " missing)");
|
||||||
|
});'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Redis (큐 관리)
|
||||||
|
```bash
|
||||||
|
# Redis CLI 접속
|
||||||
|
docker exec -it site11_redis redis-cli
|
||||||
|
|
||||||
|
# 큐 길이 확인
|
||||||
|
LLEN queue:translation
|
||||||
|
LLEN queue:ai_generation
|
||||||
|
LLEN queue:image_generation
|
||||||
|
|
||||||
|
# 큐 내용 확인 (첫 번째 항목)
|
||||||
|
LINDEX queue:translation 0
|
||||||
|
|
||||||
|
# 큐 비우기 (주의!)
|
||||||
|
DEL queue:translation
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🔄 파이프라인 관리
|
||||||
|
|
||||||
|
### 언어 동기화
|
||||||
|
```bash
|
||||||
|
# 수동 동기화 실행
|
||||||
|
docker exec -it site11_language_sync python language_sync.py sync
|
||||||
|
|
||||||
|
# 특정 언어만 동기화
|
||||||
|
docker exec -it site11_language_sync python language_sync.py sync --target-lang en
|
||||||
|
|
||||||
|
# 동기화 상태 확인
|
||||||
|
docker exec -it site11_language_sync python language_sync.py status
|
||||||
|
```
|
||||||
|
|
||||||
|
### 파이프라인 작업 실행
|
||||||
|
```bash
|
||||||
|
# RSS 수집 작업 추가
|
||||||
|
docker exec -it site11_pipeline_scheduler python -c "
|
||||||
|
import redis
|
||||||
|
r = redis.Redis(host='redis', port=6379)
|
||||||
|
r.lpush('queue:rss_collection', '{\"url\": \"https://example.com/rss\"}')
|
||||||
|
"
|
||||||
|
|
||||||
|
# 번역 작업 상태 확인
|
||||||
|
./scripts/status-check.sh | grep -A 10 "Queue Status"
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🛠️ 문제 해결
|
||||||
|
|
||||||
|
### 포트 충돌
|
||||||
|
```bash
|
||||||
|
# 포트 사용 중인 프로세스 확인
|
||||||
|
lsof -i :8080
|
||||||
|
lsof -i :8000
|
||||||
|
|
||||||
|
# 포트 포워딩 재시작
|
||||||
|
pkill -f "kubectl.*port-forward"
|
||||||
|
./scripts/start-k8s-port-forward.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pod 시작 실패
|
||||||
|
```bash
|
||||||
|
# Pod 상세 정보 확인
|
||||||
|
kubectl -n site11-pipeline describe pod <pod-name>
|
||||||
|
|
||||||
|
# 이벤트 확인
|
||||||
|
kubectl -n site11-pipeline get events --sort-by='.lastTimestamp'
|
||||||
|
|
||||||
|
# 이미지 풀 재시도
|
||||||
|
kubectl -n site11-pipeline delete pod <pod-name>
|
||||||
|
```
|
||||||
|
|
||||||
|
### 서비스 연결 실패
|
||||||
|
```bash
|
||||||
|
# 네트워크 연결 테스트
|
||||||
|
kubectl -n site11-pipeline exec -it deployment/console-backend -- bash
|
||||||
|
> curl host.docker.internal:6379 # Redis
|
||||||
|
> curl host.docker.internal:27017 # MongoDB
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📈 모니터링
|
||||||
|
|
||||||
|
### 실시간 모니터링
|
||||||
|
```bash
|
||||||
|
# 전체 시스템 상태 실시간 확인
|
||||||
|
watch -n 5 './scripts/status-check.sh'
|
||||||
|
|
||||||
|
# Kubernetes 리소스 모니터링
|
||||||
|
watch -n 2 'kubectl -n site11-pipeline get pods,hpa'
|
||||||
|
|
||||||
|
# 큐 상태 모니터링
|
||||||
|
watch -n 5 'docker exec site11_redis redis-cli info replication'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 로그 모니터링
|
||||||
|
```bash
|
||||||
|
# 전체 Docker 로그
|
||||||
|
docker-compose -f docker-compose-hybrid.yml logs -f
|
||||||
|
|
||||||
|
# 전체 Kubernetes 로그
|
||||||
|
kubectl -n site11-pipeline logs -f -l app=console-backend
|
||||||
|
|
||||||
|
# 에러만 필터링
|
||||||
|
kubectl -n site11-pipeline logs -f deployment/console-backend | grep ERROR
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🔐 인증 정보
|
||||||
|
|
||||||
|
### Console 로그인
|
||||||
|
- **URL**: http://localhost:8080
|
||||||
|
- **Admin**: admin / admin123
|
||||||
|
- **User**: user / user123
|
||||||
|
|
||||||
|
### Harbor Registry (옵션)
|
||||||
|
- **URL**: http://localhost:8880
|
||||||
|
- **Admin**: admin / Harbor12345
|
||||||
|
|
||||||
|
### Nexus Repository (옵션)
|
||||||
|
- **URL**: http://localhost:8081
|
||||||
|
- **Admin**: admin / (초기 비밀번호는 컨테이너에서 확인)
|
||||||
|
|
||||||
|
## 🏗️ 개발 도구
|
||||||
|
|
||||||
|
### 이미지 빌드
|
||||||
|
```bash
|
||||||
|
# 개별 서비스 빌드
|
||||||
|
docker-compose build console-backend
|
||||||
|
|
||||||
|
# 전체 빌드
|
||||||
|
docker-compose build
|
||||||
|
|
||||||
|
# 캐시 사용 빌드
|
||||||
|
./scripts/build-with-cache.sh console-backend
|
||||||
|
```
|
||||||
|
|
||||||
|
### 레지스트리 관리
|
||||||
|
```bash
|
||||||
|
# 레지스트리 캐시 시작
|
||||||
|
docker-compose -f docker-compose-registry-cache.yml up -d
|
||||||
|
|
||||||
|
# 캐시 상태 확인
|
||||||
|
./scripts/manage-registry.sh status
|
||||||
|
|
||||||
|
# 캐시 정리
|
||||||
|
./scripts/manage-registry.sh clean
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📚 유용한 스크립트
|
||||||
|
|
||||||
|
| 스크립트 | 설명 |
|
||||||
|
|----------|------|
|
||||||
|
| `./scripts/status-check.sh` | 전체 시스템 상태 확인 |
|
||||||
|
| `./scripts/start-k8s-port-forward.sh` | Kubernetes 포트 포워딩 시작 |
|
||||||
|
| `./scripts/setup-registry-cache.sh` | Docker 레지스트리 캐시 설정 |
|
||||||
|
| `./scripts/backup-mongodb.sh` | MongoDB 백업 |
|
||||||
|
| `./deploy-dockerhub.sh` | Docker Hub 배포 |
|
||||||
|
| `./deploy-local.sh` | 로컬 레지스트리 배포 |
|
||||||
|
|
||||||
|
## 🔍 디버깅 팁
|
||||||
|
|
||||||
|
### Console Frontend 연결 문제
|
||||||
|
```bash
|
||||||
|
# nginx 설정 확인
|
||||||
|
kubectl -n site11-pipeline exec deployment/console-frontend -- cat /etc/nginx/conf.d/default.conf
|
||||||
|
|
||||||
|
# 환경 변수 확인
|
||||||
|
kubectl -n site11-pipeline exec deployment/console-frontend -- env | grep VITE
|
||||||
|
```
|
||||||
|
|
||||||
|
### Console Backend API 문제
|
||||||
|
```bash
|
||||||
|
# FastAPI 로그 확인
|
||||||
|
kubectl -n site11-pipeline logs deployment/console-backend --tail=50
|
||||||
|
|
||||||
|
# 헬스 체크 직접 호출
|
||||||
|
kubectl -n site11-pipeline exec deployment/console-backend -- curl localhost:8000/health
|
||||||
|
```
|
||||||
|
|
||||||
|
### 파이프라인 작업 막힘
|
||||||
|
```bash
|
||||||
|
# 큐 상태 상세 확인
|
||||||
|
docker exec site11_redis redis-cli info stats
|
||||||
|
|
||||||
|
# 워커 프로세스 확인
|
||||||
|
kubectl -n site11-pipeline top pods | grep pipeline
|
||||||
|
|
||||||
|
# 메모리 사용량 확인
|
||||||
|
kubectl -n site11-pipeline describe pod <pipeline-pod-name>
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📞 지원 및 문의
|
||||||
|
|
||||||
|
- **문서**: `/docs` 디렉토리
|
||||||
|
- **이슈 트래커**: http://gitea.yakenator.io/aimond/site11/issues
|
||||||
|
- **로그 위치**: `docker-compose logs` 또는 `kubectl logs`
|
||||||
|
- **설정 파일**: `k8s/pipeline/`, `docker-compose*.yml`
|
||||||
285
docs/REGISTRY_CACHE.md
Normal file
285
docs/REGISTRY_CACHE.md
Normal file
@ -0,0 +1,285 @@
|
|||||||
|
# Docker Registry Cache 구성 가이드
|
||||||
|
|
||||||
|
## 개요
|
||||||
|
Docker Registry Cache를 사용하면 이미지 빌드 및 배포 속도를 크게 개선할 수 있습니다.
|
||||||
|
|
||||||
|
## 주요 이점
|
||||||
|
|
||||||
|
### 1. 빌드 속도 향상
|
||||||
|
- **기본 이미지 캐싱**: Python, Node.js 등 베이스 이미지를 로컬에 캐시
|
||||||
|
- **레이어 재사용**: 동일한 레이어를 여러 서비스에서 공유
|
||||||
|
- **네트워크 대역폭 절감**: Docker Hub에서 반복 다운로드 방지
|
||||||
|
|
||||||
|
### 2. CI/CD 효율성
|
||||||
|
- **빌드 시간 단축**: 캐시된 이미지로 50-80% 빌드 시간 감소
|
||||||
|
- **안정성 향상**: Docker Hub rate limit 회피
|
||||||
|
- **비용 절감**: 네트워크 트래픽 감소
|
||||||
|
|
||||||
|
### 3. 개발 환경 개선
|
||||||
|
- **오프라인 작업 가능**: 캐시된 이미지로 인터넷 없이 작업
|
||||||
|
- **일관된 이미지 버전**: 팀 전체가 동일한 캐시 사용
|
||||||
|
|
||||||
|
## 구성 옵션
|
||||||
|
|
||||||
|
### 옵션 1: 기본 Registry Cache (권장)
|
||||||
|
```bash
|
||||||
|
# 시작
|
||||||
|
docker-compose -f docker-compose-registry-cache.yml up -d registry-cache
|
||||||
|
|
||||||
|
# 설정
|
||||||
|
./scripts/setup-registry-cache.sh
|
||||||
|
|
||||||
|
# 확인
|
||||||
|
curl http://localhost:5000/v2/_catalog
|
||||||
|
```
|
||||||
|
|
||||||
|
**장점:**
|
||||||
|
- 가볍고 빠름
|
||||||
|
- 설정이 간단
|
||||||
|
- 리소스 사용량 적음
|
||||||
|
|
||||||
|
**단점:**
|
||||||
|
- UI 없음
|
||||||
|
- 기본적인 기능만 제공
|
||||||
|
|
||||||
|
### 옵션 2: Harbor Registry
|
||||||
|
```bash
|
||||||
|
# Harbor 프로필로 시작
|
||||||
|
docker-compose -f docker-compose-registry-cache.yml --profile harbor up -d
|
||||||
|
|
||||||
|
# 접속
|
||||||
|
open http://localhost:8880
|
||||||
|
# 계정: admin / Harbor12345
|
||||||
|
```
|
||||||
|
|
||||||
|
**장점:**
|
||||||
|
- 웹 UI 제공
|
||||||
|
- 보안 스캐닝
|
||||||
|
- RBAC 지원
|
||||||
|
- 복제 기능
|
||||||
|
|
||||||
|
**단점:**
|
||||||
|
- 리소스 사용량 많음
|
||||||
|
- 설정 복잡
|
||||||
|
|
||||||
|
### 옵션 3: Nexus Repository
|
||||||
|
```bash
|
||||||
|
# Nexus 프로필로 시작
|
||||||
|
docker-compose -f docker-compose-registry-cache.yml --profile nexus up -d
|
||||||
|
|
||||||
|
# 접속
|
||||||
|
open http://localhost:8081
|
||||||
|
# 초기 비밀번호: docker exec site11_nexus cat /nexus-data/admin.password
|
||||||
|
```
|
||||||
|
|
||||||
|
**장점:**
|
||||||
|
- 다양한 저장소 형식 지원 (Docker, Maven, NPM 등)
|
||||||
|
- 강력한 프록시 캐시
|
||||||
|
- 세밀한 권한 관리
|
||||||
|
|
||||||
|
**단점:**
|
||||||
|
- 초기 설정 필요
|
||||||
|
- 메모리 사용량 높음 (최소 2GB)
|
||||||
|
|
||||||
|
## 사용 방법
|
||||||
|
|
||||||
|
### 1. 캐시를 통한 이미지 빌드
|
||||||
|
```bash
|
||||||
|
# 기존 방식
|
||||||
|
docker build -t site11-service:latest .
|
||||||
|
|
||||||
|
# 캐시 활용 방식
|
||||||
|
./scripts/build-with-cache.sh service-name
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. BuildKit 캐시 마운트 활용
|
||||||
|
```dockerfile
|
||||||
|
# Dockerfile 예제
|
||||||
|
FROM python:3.11-slim
|
||||||
|
|
||||||
|
# 캐시 마운트로 pip 패키지 캐싱
|
||||||
|
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Multi-stage 빌드 최적화
|
||||||
|
```dockerfile
|
||||||
|
# 빌드 스테이지 캐싱
|
||||||
|
FROM localhost:5000/python:3.11-slim as builder
|
||||||
|
WORKDIR /app
|
||||||
|
COPY requirements.txt .
|
||||||
|
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||||
|
pip install --user -r requirements.txt
|
||||||
|
|
||||||
|
# 런타임 스테이지
|
||||||
|
FROM localhost:5000/python:3.11-slim
|
||||||
|
WORKDIR /app
|
||||||
|
COPY --from=builder /root/.local /root/.local
|
||||||
|
COPY . .
|
||||||
|
```
|
||||||
|
|
||||||
|
## Kubernetes와 통합
|
||||||
|
|
||||||
|
### 1. K8s 클러스터 설정
|
||||||
|
```yaml
|
||||||
|
# configmap for containerd
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: containerd-config
|
||||||
|
namespace: kube-system
|
||||||
|
data:
|
||||||
|
config.toml: |
|
||||||
|
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
|
||||||
|
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
|
||||||
|
endpoint = ["http://host.docker.internal:5000"]
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Pod 설정
|
||||||
|
```yaml
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Pod
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: app
|
||||||
|
image: localhost:5000/site11-service:latest
|
||||||
|
imagePullPolicy: Always
|
||||||
|
```
|
||||||
|
|
||||||
|
## 모니터링
|
||||||
|
|
||||||
|
### 캐시 상태 확인
|
||||||
|
```bash
|
||||||
|
# 캐시된 이미지 목록
|
||||||
|
./scripts/manage-registry.sh status
|
||||||
|
|
||||||
|
# 캐시 크기
|
||||||
|
./scripts/manage-registry.sh size
|
||||||
|
|
||||||
|
# 실시간 로그
|
||||||
|
./scripts/manage-registry.sh logs
|
||||||
|
```
|
||||||
|
|
||||||
|
### 메트릭 수집
|
||||||
|
```yaml
|
||||||
|
# Prometheus 설정 예제
|
||||||
|
scrape_configs:
|
||||||
|
- job_name: 'docker-registry'
|
||||||
|
static_configs:
|
||||||
|
- targets: ['localhost:5000']
|
||||||
|
metrics_path: '/metrics'
|
||||||
|
```
|
||||||
|
|
||||||
|
## 최적화 팁
|
||||||
|
|
||||||
|
### 1. 레이어 캐싱 최적화
|
||||||
|
- 자주 변경되지 않는 명령을 먼저 실행
|
||||||
|
- COPY 명령 최소화
|
||||||
|
- .dockerignore 활용
|
||||||
|
|
||||||
|
### 2. 빌드 캐시 전략
|
||||||
|
```bash
|
||||||
|
# 캐시 export
|
||||||
|
docker buildx build \
|
||||||
|
--cache-to type=registry,ref=localhost:5000/cache:latest \
|
||||||
|
.
|
||||||
|
|
||||||
|
# 캐시 import
|
||||||
|
docker buildx build \
|
||||||
|
--cache-from type=registry,ref=localhost:5000/cache:latest \
|
||||||
|
.
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. 가비지 컬렉션
|
||||||
|
```bash
|
||||||
|
# 수동 정리
|
||||||
|
./scripts/manage-registry.sh clean
|
||||||
|
|
||||||
|
# 자동 정리 (config.yml에 설정됨)
|
||||||
|
# 12시간마다 자동 실행
|
||||||
|
```
|
||||||
|
|
||||||
|
## 문제 해결
|
||||||
|
|
||||||
|
### Registry 접근 불가
|
||||||
|
```bash
|
||||||
|
# 방화벽 확인
|
||||||
|
sudo iptables -L | grep 5000
|
||||||
|
|
||||||
|
# Docker 데몬 재시작
|
||||||
|
sudo systemctl restart docker
|
||||||
|
```
|
||||||
|
|
||||||
|
### 캐시 미스 발생
|
||||||
|
```bash
|
||||||
|
# 캐시 재구성
|
||||||
|
docker buildx prune -f
|
||||||
|
docker buildx create --use
|
||||||
|
```
|
||||||
|
|
||||||
|
### 디스크 공간 부족
|
||||||
|
```bash
|
||||||
|
# 오래된 이미지 정리
|
||||||
|
docker system prune -a --volumes
|
||||||
|
|
||||||
|
# Registry 가비지 컬렉션
|
||||||
|
docker exec site11_registry_cache \
|
||||||
|
registry garbage-collect /etc/docker/registry/config.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
## 성능 벤치마크
|
||||||
|
|
||||||
|
### 테스트 환경
|
||||||
|
- macOS M1 Pro
|
||||||
|
- Docker Desktop 4.x
|
||||||
|
- 16GB RAM
|
||||||
|
|
||||||
|
### 결과
|
||||||
|
| 작업 | 캐시 없음 | 캐시 사용 | 개선율 |
|
||||||
|
|------|---------|----------|--------|
|
||||||
|
| Python 서비스 빌드 | 120s | 35s | 71% |
|
||||||
|
| Node.js 프론트엔드 | 90s | 25s | 72% |
|
||||||
|
| 전체 스택 빌드 | 15m | 4m | 73% |
|
||||||
|
|
||||||
|
## 보안 고려사항
|
||||||
|
|
||||||
|
### 1. Registry 인증
|
||||||
|
```yaml
|
||||||
|
# Basic Auth 설정
|
||||||
|
auth:
|
||||||
|
htpasswd:
|
||||||
|
realm: basic-realm
|
||||||
|
path: /auth/htpasswd
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. TLS 설정
|
||||||
|
```yaml
|
||||||
|
# TLS 활성화
|
||||||
|
http:
|
||||||
|
addr: :5000
|
||||||
|
tls:
|
||||||
|
certificate: /certs/domain.crt
|
||||||
|
key: /certs/domain.key
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. 접근 제어
|
||||||
|
```yaml
|
||||||
|
# IP 화이트리스트
|
||||||
|
http:
|
||||||
|
addr: :5000
|
||||||
|
host: 127.0.0.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## 다음 단계
|
||||||
|
|
||||||
|
1. **프로덕션 배포**
|
||||||
|
- AWS ECR 또는 GCP Artifact Registry 연동
|
||||||
|
- CDN 통합
|
||||||
|
|
||||||
|
2. **고가용성**
|
||||||
|
- Registry 클러스터링
|
||||||
|
- 백업 및 복구 전략
|
||||||
|
|
||||||
|
3. **자동화**
|
||||||
|
- GitHub Actions 통합
|
||||||
|
- ArgoCD 연동
|
||||||
769
docs/TECHNICAL_INTERVIEW.md
Normal file
769
docs/TECHNICAL_INTERVIEW.md
Normal file
@ -0,0 +1,769 @@
|
|||||||
|
# Site11 프로젝트 기술 면접 가이드
|
||||||
|
|
||||||
|
## 프로젝트 개요
|
||||||
|
- **아키텍처**: API Gateway 패턴 기반 마이크로서비스
|
||||||
|
- **기술 스택**: FastAPI, React 18, TypeScript, MongoDB, Redis, Docker, Kubernetes
|
||||||
|
- **도메인**: 뉴스/미디어 플랫폼 (다국가/다언어 지원)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 백엔드 아키텍처 (5문항)
|
||||||
|
|
||||||
|
### Q1. API Gateway vs Service Mesh
|
||||||
|
|
||||||
|
**질문**: Console이 API Gateway 역할을 합니다. Service Mesh(Istio)와 비교했을 때 장단점은?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> **API Gateway 패턴 (현재)**:
|
||||||
|
> - ✅ 중앙화된 인증/라우팅, 구축 간단, 단일 진입점
|
||||||
|
> - ❌ SPOF 가능성, 병목 위험, Gateway 변경 시 전체 영향
|
||||||
|
>
|
||||||
|
> **Service Mesh (Istio)**:
|
||||||
|
> - ✅ 서비스 간 직접 통신(낮은 지연), mTLS 자동, 세밀한 트래픽 제어
|
||||||
|
> - ❌ 학습 곡선, 리소스 오버헤드(Sidecar), 복잡한 디버깅
|
||||||
|
>
|
||||||
|
> **선택 기준**:
|
||||||
|
> - 30개 이하 서비스 → API Gateway
|
||||||
|
> - 50개 이상, 복잡한 통신 패턴 → Service Mesh
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Q2. FastAPI 비동기 처리
|
||||||
|
|
||||||
|
**질문**: `async/await` 사용 시기와 Motor vs PyMongo 선택 이유는?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> **동작 차이**:
|
||||||
|
> ```python
|
||||||
|
> # Sync: 요청 1(50ms) → 요청 2(50ms) = 총 100ms
|
||||||
|
> # Async: 요청 1 & 요청 2 병행 처리 = 총 ~50ms
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **Motor (Async) 추천**:
|
||||||
|
> - I/O bound 작업(DB, API 호출)에 적합
|
||||||
|
> - 동시 요청 시 처리량 증가
|
||||||
|
> - FastAPI의 비동기 특성과 완벽 호환
|
||||||
|
>
|
||||||
|
> **PyMongo (Sync) 사용**:
|
||||||
|
> - CPU bound 작업(이미지 처리, 데이터 분석)
|
||||||
|
> - Sync 전용 라이브러리 사용 시
|
||||||
|
>
|
||||||
|
> **주의**: `time.sleep()`은 전체 이벤트 루프 블로킹 → `asyncio.sleep()` 사용
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Q3. 마이크로서비스 간 통신
|
||||||
|
|
||||||
|
**질문**: REST API, Redis Pub/Sub, gRPC 각각 언제 사용?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> | 방식 | 사용 시기 | 특징 |
|
||||||
|
> |------|----------|------|
|
||||||
|
> | **REST** | 즉시 응답 필요, 데이터 조회 | Synchronous, 구현 간단 |
|
||||||
|
> | **Pub/Sub** | 이벤트 알림, 여러 서비스 반응 | Asynchronous, Loose coupling |
|
||||||
|
> | **gRPC** | 내부 서비스 통신, 고성능 | HTTP/2, Protobuf, 타입 안정성 |
|
||||||
|
>
|
||||||
|
> **예시**:
|
||||||
|
> - 사용자 조회 → REST (즉시 응답)
|
||||||
|
> - 사용자 생성 알림 → Pub/Sub (비동기 처리)
|
||||||
|
> - 마이크로서비스 간 내부 호출 → gRPC (성능)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Q4. 데이터베이스 전략
|
||||||
|
|
||||||
|
**질문**: Shared MongoDB Instance vs Separate Instances 장단점?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> **현재 전략 (Shared Instance, Separate DBs)**:
|
||||||
|
> ```
|
||||||
|
> MongoDB (site11-mongodb:27017)
|
||||||
|
> ├── console_db
|
||||||
|
> ├── users_db
|
||||||
|
> └── news_api_db
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **장점**: 운영 단순, 리소스 효율, 백업 간편, 비용 절감
|
||||||
|
> **단점**: 격리 부족, 확장성 제한, 장애 전파, 리소스 경합
|
||||||
|
>
|
||||||
|
> **Separate Instances**:
|
||||||
|
> - 장점: 완전 격리, 독립 확장, 장애 격리
|
||||||
|
> - 단점: 운영 복잡, 비용 증가, 트랜잭션 불가
|
||||||
|
>
|
||||||
|
> **서비스 간 데이터 접근**:
|
||||||
|
> - ❌ 직접 DB 접근 금지
|
||||||
|
> - ✅ API 호출 또는 Data Duplication (비정규화)
|
||||||
|
> - ✅ Event-driven 동기화
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Q5. JWT 인증 및 보안
|
||||||
|
|
||||||
|
**질문**: Access Token vs Refresh Token 차이와 탈취 대응 방안?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> | 구분 | Access Token | Refresh Token |
|
||||||
|
> |------|--------------|---------------|
|
||||||
|
> | 목적 | API 접근 권한 | Access Token 재발급 |
|
||||||
|
> | 만료 | 짧음 (15분-1시간) | 길음 (7일-30일) |
|
||||||
|
> | 저장 | 메모리 | HttpOnly Cookie |
|
||||||
|
> | 탈취 시 | 제한적 피해 | 심각한 피해 |
|
||||||
|
>
|
||||||
|
> **탈취 대응**:
|
||||||
|
> 1. **Refresh Token Rotation**: 재발급 시 새로운 토큰 쌍 생성
|
||||||
|
> 2. **Blacklist**: Redis에 로그아웃된 토큰 저장
|
||||||
|
> 3. **Device Binding**: 디바이스 ID로 제한
|
||||||
|
> 4. **IP/User-Agent 검증**: 비정상 접근 탐지
|
||||||
|
>
|
||||||
|
> **서비스 간 통신 보안**:
|
||||||
|
> - Service Token (API Key)
|
||||||
|
> - mTLS (Production)
|
||||||
|
> - Network Policy (Kubernetes)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 프론트엔드 (4문항)
|
||||||
|
|
||||||
|
### Q6. React 18 주요 변화
|
||||||
|
|
||||||
|
**질문**: Concurrent Rendering과 Automatic Batching 설명?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> **1. Concurrent Rendering**:
|
||||||
|
> ```tsx
|
||||||
|
> const [query, setQuery] = useState('');
|
||||||
|
> const [isPending, startTransition] = useTransition();
|
||||||
|
>
|
||||||
|
> // 긴급 업데이트 (사용자 입력)
|
||||||
|
> setQuery(e.target.value);
|
||||||
|
>
|
||||||
|
> // 비긴급 업데이트 (검색 결과) - 중단 가능
|
||||||
|
> startTransition(() => {
|
||||||
|
> fetchSearchResults(e.target.value);
|
||||||
|
> });
|
||||||
|
> ```
|
||||||
|
> → 사용자 입력이 항상 부드럽게 유지
|
||||||
|
>
|
||||||
|
> **2. Automatic Batching**:
|
||||||
|
> ```tsx
|
||||||
|
> // React 17: fetch 콜백에서 2번 리렌더링
|
||||||
|
> fetch('/api').then(() => {
|
||||||
|
> setCount(c => c + 1); // 리렌더링 1
|
||||||
|
> setFlag(f => !f); // 리렌더링 2
|
||||||
|
> });
|
||||||
|
>
|
||||||
|
> // React 18: 자동 배칭으로 1번만 리렌더링
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **기타**: `Suspense`, `useDeferredValue`, `useId`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Q7. TypeScript 활용
|
||||||
|
|
||||||
|
**질문**: Backend API 타입을 Frontend에서 안전하게 사용하는 방법?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> **방법 1: OpenAPI 코드 생성** (추천)
|
||||||
|
> ```bash
|
||||||
|
> npm install openapi-typescript-codegen
|
||||||
|
> openapi --input http://localhost:8000/openapi.json --output ./src/api/generated
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> ```typescript
|
||||||
|
> // 자동 생성된 타입 사용
|
||||||
|
> import { ArticlesService, Article } from '@/api/generated';
|
||||||
|
>
|
||||||
|
> const articles = await ArticlesService.getArticles({
|
||||||
|
> category: 'tech', // ✅ 타입 체크
|
||||||
|
> limit: 10
|
||||||
|
> });
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **방법 2: tRPC** (TypeScript 풀스택)
|
||||||
|
> ```typescript
|
||||||
|
> // Backend
|
||||||
|
> export const appRouter = t.router({
|
||||||
|
> articles: {
|
||||||
|
> list: t.procedure.input(z.object({...})).query(...)
|
||||||
|
> }
|
||||||
|
> });
|
||||||
|
>
|
||||||
|
> // Frontend - End-to-end 타입 안정성
|
||||||
|
> const { data } = trpc.articles.list.useQuery({ category: 'tech' });
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **방법 3: 수동 타입 정의** (작은 프로젝트)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Q8. 상태 관리
|
||||||
|
|
||||||
|
**질문**: Context API, Redux, Zustand, React Query 각각 언제 사용?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> | 도구 | 사용 시기 | 특징 |
|
||||||
|
> |------|----------|------|
|
||||||
|
> | **Context API** | 전역 테마, 인증 상태 | 내장, 리렌더링 주의 |
|
||||||
|
> | **Redux** | 복잡한 상태, Time-travel | Boilerplate 많음, DevTools |
|
||||||
|
> | **Zustand** | 간단한 전역 상태 | 경량, 간결, 리렌더링 최적화 |
|
||||||
|
> | **React Query** | 서버 상태 | 캐싱, 리페칭, 낙관적 업데이트 |
|
||||||
|
>
|
||||||
|
> **핵심**: 전역 상태 vs 서버 상태 구분
|
||||||
|
> - 전역 UI 상태 → Zustand/Redux
|
||||||
|
> - 서버 데이터 → React Query
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Q9. Material-UI 최적화
|
||||||
|
|
||||||
|
**질문**: 번들 사이즈 최적화와 테마 커스터마이징 방법?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> **번들 최적화**:
|
||||||
|
> ```tsx
|
||||||
|
> // ❌ 전체 import
|
||||||
|
> import { Button, TextField } from '@mui/material';
|
||||||
|
>
|
||||||
|
> // ✅ Tree shaking
|
||||||
|
> import Button from '@mui/material/Button';
|
||||||
|
> import TextField from '@mui/material/TextField';
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **Code Splitting**:
|
||||||
|
> ```tsx
|
||||||
|
> const Dashboard = lazy(() => import('./pages/Dashboard'));
|
||||||
|
>
|
||||||
|
> <Suspense fallback={<Loading />}>
|
||||||
|
> <Dashboard />
|
||||||
|
> </Suspense>
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **테마 커스터마이징**:
|
||||||
|
> ```tsx
|
||||||
|
> import { createTheme, ThemeProvider } from '@mui/material/styles';
|
||||||
|
>
|
||||||
|
> const theme = createTheme({
|
||||||
|
> palette: {
|
||||||
|
> mode: 'dark',
|
||||||
|
> primary: { main: '#1976d2' },
|
||||||
|
> },
|
||||||
|
> });
|
||||||
|
>
|
||||||
|
> <ThemeProvider theme={theme}>
|
||||||
|
> <App />
|
||||||
|
> </ThemeProvider>
|
||||||
|
> ```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. DevOps & 인프라 (6문항)
|
||||||
|
|
||||||
|
### Q10. Docker Multi-stage Build
|
||||||
|
|
||||||
|
**질문**: Multi-stage build의 장점과 각 stage 역할은?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> ```dockerfile
|
||||||
|
> # Stage 1: Builder (빌드 환경)
|
||||||
|
> FROM node:18-alpine AS builder
|
||||||
|
> WORKDIR /app
|
||||||
|
> COPY package.json ./
|
||||||
|
> RUN npm install
|
||||||
|
> COPY . .
|
||||||
|
> RUN npm run build
|
||||||
|
>
|
||||||
|
> # Stage 2: Production (런타임)
|
||||||
|
> FROM nginx:alpine
|
||||||
|
> COPY --from=builder /app/dist /usr/share/nginx/html
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **장점**:
|
||||||
|
> - 빌드 도구 제외 → 이미지 크기 90% 감소
|
||||||
|
> - Layer caching → 빌드 속도 향상
|
||||||
|
> - 보안 강화 → 소스코드 미포함
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Q11. Kubernetes 배포 전략
|
||||||
|
|
||||||
|
**질문**: Rolling Update, Blue/Green, Canary 차이와 선택 기준?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> | 전략 | 특징 | 적합한 경우 |
|
||||||
|
> |------|------|------------|
|
||||||
|
> | **Rolling Update** | 점진적 교체 | 일반 배포, Zero-downtime |
|
||||||
|
> | **Blue/Green** | 전체 전환 후 스위칭 | 빠른 롤백 필요 |
|
||||||
|
> | **Canary** | 일부 트래픽 테스트 | 위험한 변경, A/B 테스트 |
|
||||||
|
>
|
||||||
|
> **News API 같은 중요 서비스**: Canary (10% → 50% → 100%)
|
||||||
|
>
|
||||||
|
> **Probe 설정**:
|
||||||
|
> ```yaml
|
||||||
|
> livenessProbe: # 재시작 판단
|
||||||
|
> httpGet:
|
||||||
|
> path: /health
|
||||||
|
> readinessProbe: # 트래픽 차단 판단
|
||||||
|
> httpGet:
|
||||||
|
> path: /ready
|
||||||
|
> ```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Q12. 서비스 헬스체크
|
||||||
|
|
||||||
|
**질문**: Liveness Probe vs Readiness Probe 차이?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> | Probe | 실패 시 동작 | 실패 조건 예시 |
|
||||||
|
> |-------|-------------|---------------|
|
||||||
|
> | **Liveness** | Pod 재시작 | 데드락, 메모리 누수 |
|
||||||
|
> | **Readiness** | 트래픽 차단 | DB 연결 실패, 초기화 중 |
|
||||||
|
>
|
||||||
|
> **구현**:
|
||||||
|
> ```python
|
||||||
|
> @app.get("/health") # Liveness
|
||||||
|
> async def health():
|
||||||
|
> return {"status": "ok"}
|
||||||
|
>
|
||||||
|
> @app.get("/ready") # Readiness
|
||||||
|
> async def ready():
|
||||||
|
> # DB 연결 확인
|
||||||
|
> if not await db.ping():
|
||||||
|
> raise HTTPException(503)
|
||||||
|
> return {"status": "ready"}
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **Startup Probe**: 초기 구동이 느린 앱 (DB 마이그레이션 등)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Q13. 외부 DB 연결
|
||||||
|
|
||||||
|
**질문**: MongoDB/Redis를 클러스터 외부에서 운영하는 이유?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> **현재 전략 (외부 운영)**:
|
||||||
|
> - ✅ 데이터 영속성 (클러스터 재생성 시 보존)
|
||||||
|
> - ✅ 관리 용이 (단일 인스턴스)
|
||||||
|
> - ✅ 개발 환경 공유
|
||||||
|
>
|
||||||
|
> **StatefulSet (내부 운영)**:
|
||||||
|
> - ✅ Kubernetes 통합 관리
|
||||||
|
> - ✅ 자동 스케일링
|
||||||
|
> - ❌ PV 관리 복잡
|
||||||
|
> - ❌ 백업/복구 부담
|
||||||
|
>
|
||||||
|
> **선택 기준**:
|
||||||
|
> - 개발/스테이징 → 외부 (간편)
|
||||||
|
> - 프로덕션 → Managed Service (RDS, Atlas) 추천
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Q14. Docker Compose vs Kubernetes
|
||||||
|
|
||||||
|
**질문**: 언제 Docker Compose만으로 충분하고 언제 Kubernetes 필요?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> | 기능 | Docker Compose | Kubernetes |
|
||||||
|
> |------|---------------|-----------|
|
||||||
|
> | 컨테이너 실행 | ✅ | ✅ |
|
||||||
|
> | Auto-scaling | ❌ | ✅ |
|
||||||
|
> | Self-healing | ❌ | ✅ |
|
||||||
|
> | Load Balancing | 기본적 | 고급 |
|
||||||
|
> | 배포 전략 | 단순 | 다양 (Rolling, Canary) |
|
||||||
|
> | 멀티 호스트 | ❌ | ✅ |
|
||||||
|
>
|
||||||
|
> **Docker Compose 충분**:
|
||||||
|
> - 단일 서버
|
||||||
|
> - 소규모 서비스 (< 10개)
|
||||||
|
> - 개발/테스트 환경
|
||||||
|
>
|
||||||
|
> **Kubernetes 필요**:
|
||||||
|
> - 고가용성 (HA)
|
||||||
|
> - 자동 확장
|
||||||
|
> - 수십~수백 개 서비스
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Q15. 모니터링 및 로깅
|
||||||
|
|
||||||
|
**질문**: 마이크로서비스 환경에서 로그 수집 및 모니터링 방법?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> **로깅 스택**:
|
||||||
|
> - **ELK**: Elasticsearch + Logstash + Kibana
|
||||||
|
> - **EFK**: Elasticsearch + Fluentd + Kibana
|
||||||
|
> - **Loki**: Grafana Loki (경량)
|
||||||
|
>
|
||||||
|
> **모니터링**:
|
||||||
|
> - **Prometheus**: 메트릭 수집
|
||||||
|
> - **Grafana**: 대시보드
|
||||||
|
> - **Jaeger/Zipkin**: Distributed Tracing
|
||||||
|
>
|
||||||
|
> **Correlation ID**:
|
||||||
|
> ```python
|
||||||
|
> @app.middleware("http")
|
||||||
|
> async def add_correlation_id(request: Request, call_next):
|
||||||
|
> correlation_id = request.headers.get("X-Correlation-ID") or str(uuid.uuid4())
|
||||||
|
> request.state.correlation_id = correlation_id
|
||||||
|
>
|
||||||
|
> # 모든 로그에 포함
|
||||||
|
> logger.info(f"Request {correlation_id}: {request.url}")
|
||||||
|
>
|
||||||
|
> response = await call_next(request)
|
||||||
|
> response.headers["X-Correlation-ID"] = correlation_id
|
||||||
|
> return response
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **3가지 관찰성**:
|
||||||
|
> - Metrics (숫자): CPU, 메모리, 요청 수
|
||||||
|
> - Logs (텍스트): 이벤트, 에러
|
||||||
|
> - Traces (흐름): 요청 경로 추적
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 데이터 및 API 설계 (3문항)
|
||||||
|
|
||||||
|
### Q16. RESTful API 설계
|
||||||
|
|
||||||
|
**질문**: News API 엔드포인트를 RESTful하게 설계하면?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> ```
|
||||||
|
> GET /api/v1/outlets # 언론사 목록
|
||||||
|
> GET /api/v1/outlets/{outlet_id} # 언론사 상세
|
||||||
|
> GET /api/v1/outlets/{outlet_id}/articles # 특정 언론사 기사
|
||||||
|
>
|
||||||
|
> GET /api/v1/articles # 기사 목록
|
||||||
|
> GET /api/v1/articles/{article_id} # 기사 상세
|
||||||
|
> POST /api/v1/articles # 기사 생성
|
||||||
|
> PUT /api/v1/articles/{article_id} # 기사 수정
|
||||||
|
> DELETE /api/v1/articles/{article_id} # 기사 삭제
|
||||||
|
>
|
||||||
|
> # 쿼리 파라미터
|
||||||
|
> GET /api/v1/articles?category=tech&limit=10&offset=0
|
||||||
|
>
|
||||||
|
> # 다국어 지원
|
||||||
|
> GET /api/v1/ko/articles # URL prefix
|
||||||
|
> GET /api/v1/articles (Accept-Language: ko-KR) # Header
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **RESTful 원칙**:
|
||||||
|
> 1. 리소스 중심 (명사 사용)
|
||||||
|
> 2. HTTP 메소드 의미 준수
|
||||||
|
> 3. Stateless
|
||||||
|
> 4. 계층적 구조
|
||||||
|
> 5. HATEOAS (선택)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Q17. MongoDB 스키마 설계
|
||||||
|
|
||||||
|
**질문**: Outlets-Articles-Keywords 관계를 MongoDB에서 모델링?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> **방법 1: Embedding** (Read 최적화)
|
||||||
|
> ```json
|
||||||
|
> {
|
||||||
|
> "_id": "article123",
|
||||||
|
> "title": "Breaking News",
|
||||||
|
> "outlet": {
|
||||||
|
> "id": "outlet456",
|
||||||
|
> "name": "TechCrunch",
|
||||||
|
> "logo": "url"
|
||||||
|
> },
|
||||||
|
> "keywords": ["AI", "Machine Learning"]
|
||||||
|
> }
|
||||||
|
> ```
|
||||||
|
> - ✅ 1번의 쿼리로 모든 데이터
|
||||||
|
> - ❌ Outlet 정보 변경 시 모든 Article 업데이트
|
||||||
|
>
|
||||||
|
> **방법 2: Referencing** (Write 최적화)
|
||||||
|
> ```json
|
||||||
|
> {
|
||||||
|
> "_id": "article123",
|
||||||
|
> "title": "Breaking News",
|
||||||
|
> "outlet_id": "outlet456",
|
||||||
|
> "keyword_ids": ["kw1", "kw2"]
|
||||||
|
> }
|
||||||
|
> ```
|
||||||
|
> - ✅ 데이터 일관성
|
||||||
|
> - ❌ 조회 시 여러 쿼리 필요 (JOIN)
|
||||||
|
>
|
||||||
|
> **하이브리드** (추천):
|
||||||
|
> ```json
|
||||||
|
> {
|
||||||
|
> "_id": "article123",
|
||||||
|
> "title": "Breaking News",
|
||||||
|
> "outlet_id": "outlet456",
|
||||||
|
> "outlet_name": "TechCrunch", // 자주 조회되는 필드만 복제
|
||||||
|
> "keywords": ["AI", "ML"] // 배열 embedding
|
||||||
|
> }
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **인덱싱**:
|
||||||
|
> ```python
|
||||||
|
> db.articles.create_index([("outlet_id", 1), ("published_at", -1)])
|
||||||
|
> db.articles.create_index([("keywords", 1)])
|
||||||
|
> ```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Q18. 페이지네이션 전략
|
||||||
|
|
||||||
|
**질문**: Offset-based vs Cursor-based Pagination 차이?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> **Offset-based** (전통적):
|
||||||
|
> ```python
|
||||||
|
> # GET /api/articles?page=2&page_size=10
|
||||||
|
> skip = (page - 1) * page_size
|
||||||
|
> articles = db.articles.find().skip(skip).limit(page_size)
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> - ✅ 구현 간단, 페이지 번호 표시
|
||||||
|
> - ❌ 대량 데이터에서 느림 (SKIP 연산)
|
||||||
|
> - ❌ 실시간 데이터 변경 시 중복/누락
|
||||||
|
>
|
||||||
|
> **Cursor-based** (무한 스크롤):
|
||||||
|
> ```python
|
||||||
|
> # GET /api/articles?cursor=article123&limit=10
|
||||||
|
> articles = db.articles.find({
|
||||||
|
> "_id": {"$lt": ObjectId(cursor)}
|
||||||
|
> }).sort("_id", -1).limit(10)
|
||||||
|
>
|
||||||
|
> # Response
|
||||||
|
> {
|
||||||
|
> "items": [...],
|
||||||
|
> "next_cursor": "article110"
|
||||||
|
> }
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> - ✅ 빠른 성능 (인덱스 활용)
|
||||||
|
> - ✅ 실시간 데이터 일관성
|
||||||
|
> - ❌ 특정 페이지 이동 불가
|
||||||
|
>
|
||||||
|
> **선택 기준**:
|
||||||
|
> - 페이지 번호 필요 → Offset
|
||||||
|
> - 무한 스크롤, 대량 데이터 → Cursor
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. 문제 해결 및 확장성 (2문항)
|
||||||
|
|
||||||
|
### Q19. 대규모 트래픽 처리
|
||||||
|
|
||||||
|
**질문**: 순간 트래픽 10배 증가 시 대응 방안?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> **1. 캐싱 (Redis)**:
|
||||||
|
> ```python
|
||||||
|
> @app.get("/api/articles/{article_id}")
|
||||||
|
> async def get_article(article_id: str):
|
||||||
|
> # Cache-aside 패턴
|
||||||
|
> cached = await redis.get(f"article:{article_id}")
|
||||||
|
> if cached:
|
||||||
|
> return json.loads(cached)
|
||||||
|
>
|
||||||
|
> article = await db.articles.find_one({"_id": article_id})
|
||||||
|
> await redis.setex(f"article:{article_id}", 3600, json.dumps(article))
|
||||||
|
> return article
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **2. Auto-scaling (HPA)**:
|
||||||
|
> ```yaml
|
||||||
|
> apiVersion: autoscaling/v2
|
||||||
|
> kind: HorizontalPodAutoscaler
|
||||||
|
> metadata:
|
||||||
|
> name: news-api-hpa
|
||||||
|
> spec:
|
||||||
|
> scaleTargetRef:
|
||||||
|
> apiVersion: apps/v1
|
||||||
|
> kind: Deployment
|
||||||
|
> name: news-api
|
||||||
|
> minReplicas: 2
|
||||||
|
> maxReplicas: 10
|
||||||
|
> metrics:
|
||||||
|
> - type: Resource
|
||||||
|
> resource:
|
||||||
|
> name: cpu
|
||||||
|
> target:
|
||||||
|
> type: Utilization
|
||||||
|
> averageUtilization: 70
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **3. Rate Limiting**:
|
||||||
|
> ```python
|
||||||
|
> from slowapi import Limiter
|
||||||
|
>
|
||||||
|
> limiter = Limiter(key_func=get_remote_address)
|
||||||
|
>
|
||||||
|
> @app.get("/api/articles")
|
||||||
|
> @limiter.limit("100/minute")
|
||||||
|
> async def list_articles():
|
||||||
|
> ...
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **4. Circuit Breaker** (장애 전파 방지):
|
||||||
|
> ```python
|
||||||
|
> from circuitbreaker import circuit
|
||||||
|
>
|
||||||
|
> @circuit(failure_threshold=5, recovery_timeout=60)
|
||||||
|
> async def call_external_service():
|
||||||
|
> ...
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **5. CDN**: 정적 리소스 (이미지, CSS, JS)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Q20. 장애 시나리오 대응
|
||||||
|
|
||||||
|
**질문**: MongoDB 다운/서비스 무응답/Redis 메모리 가득 시 대응?
|
||||||
|
|
||||||
|
> [!success]- 모범 답안
|
||||||
|
>
|
||||||
|
> **1. MongoDB 다운**:
|
||||||
|
> ```python
|
||||||
|
> @app.get("/api/articles")
|
||||||
|
> async def list_articles():
|
||||||
|
> try:
|
||||||
|
> articles = await db.articles.find().to_list(10)
|
||||||
|
> return articles
|
||||||
|
> except Exception as e:
|
||||||
|
> # Graceful degradation
|
||||||
|
> logger.error(f"DB error: {e}")
|
||||||
|
>
|
||||||
|
> # Fallback: 캐시에서 반환
|
||||||
|
> cached = await redis.get("articles:fallback")
|
||||||
|
> if cached:
|
||||||
|
> return {"data": json.loads(cached), "source": "cache"}
|
||||||
|
>
|
||||||
|
> # 최후: 기본 메시지
|
||||||
|
> raise HTTPException(503, "Service temporarily unavailable")
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **2. 마이크로서비스 무응답**:
|
||||||
|
> ```python
|
||||||
|
> from circuitbreaker import circuit
|
||||||
|
>
|
||||||
|
> @circuit(failure_threshold=3, recovery_timeout=30)
|
||||||
|
> async def call_user_service(user_id):
|
||||||
|
> async with httpx.AsyncClient(timeout=5.0) as client:
|
||||||
|
> response = await client.get(f"http://users-service/users/{user_id}")
|
||||||
|
> return response.json()
|
||||||
|
>
|
||||||
|
> # Circuit Open 시 Fallback
|
||||||
|
> try:
|
||||||
|
> user = await call_user_service(user_id)
|
||||||
|
> except CircuitBreakerError:
|
||||||
|
> # 기본 사용자 정보 반환
|
||||||
|
> user = {"id": user_id, "name": "Unknown"}
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **3. Redis 메모리 가득**:
|
||||||
|
> ```conf
|
||||||
|
> # redis.conf
|
||||||
|
> maxmemory 2gb
|
||||||
|
> maxmemory-policy allkeys-lru # LRU eviction
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> ```python
|
||||||
|
> # 중요도 기반 TTL
|
||||||
|
> await redis.setex("hot_article:123", 3600, data) # 1시간
|
||||||
|
> await redis.setex("old_article:456", 300, data) # 5분
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **Health Check 자동 재시작**:
|
||||||
|
> ```yaml
|
||||||
|
> livenessProbe:
|
||||||
|
> httpGet:
|
||||||
|
> path: /health
|
||||||
|
> failureThreshold: 3
|
||||||
|
> periodSeconds: 10
|
||||||
|
> ```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 평가 기준
|
||||||
|
|
||||||
|
### 초급 (Junior) - 5-8개 정답
|
||||||
|
- 기본 개념 이해
|
||||||
|
- 공식 문서 참고하여 구현 가능
|
||||||
|
- 가이드 있으면 개발 가능
|
||||||
|
|
||||||
|
### 중급 (Mid-level) - 9-14개 정답
|
||||||
|
- 아키텍처 패턴 이해
|
||||||
|
- 트레이드오프 판단 가능
|
||||||
|
- 독립적으로 서비스 설계 및 구현
|
||||||
|
- 기본 DevOps 작업 가능
|
||||||
|
|
||||||
|
### 고급 (Senior) - 15-20개 정답
|
||||||
|
- 시스템 전체 설계 가능
|
||||||
|
- 성능/확장성/보안 고려한 의사결정
|
||||||
|
- 장애 대응 및 모니터링 전략
|
||||||
|
- 팀 리딩 및 기술 멘토링
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 실무 과제 (선택)
|
||||||
|
|
||||||
|
### 과제: Comments 서비스 추가
|
||||||
|
기사에 댓글 기능을 추가하는 마이크로서비스 구현
|
||||||
|
|
||||||
|
**요구사항**:
|
||||||
|
1. Backend API (FastAPI)
|
||||||
|
- CRUD 엔드포인트
|
||||||
|
- 대댓글(nested comments) 지원
|
||||||
|
- 페이지네이션
|
||||||
|
2. Frontend UI (React + TypeScript)
|
||||||
|
- 댓글 목록/작성/수정/삭제
|
||||||
|
- Material-UI 사용
|
||||||
|
3. DevOps
|
||||||
|
- Dockerfile 작성
|
||||||
|
- Kubernetes 배포
|
||||||
|
- Console과 연동
|
||||||
|
|
||||||
|
**평가 요소**:
|
||||||
|
- 코드 품질 (타입 안정성, 에러 핸들링)
|
||||||
|
- API 설계 (RESTful 원칙)
|
||||||
|
- 성능 고려 (인덱싱, 캐싱)
|
||||||
|
- Git 커밋 메시지
|
||||||
|
|
||||||
|
**소요 시간**: 4-6시간
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 면접 진행 Tips
|
||||||
|
|
||||||
|
1. **깊이 있는 질문**: "이전 프로젝트에서는 어떻게 해결했나요?"
|
||||||
|
2. **화이트보드 세션**: 아키텍처 다이어그램 그리기
|
||||||
|
3. **코드 리뷰**: 기존 코드 개선점 찾기
|
||||||
|
4. **시나리오 기반**: "만약 ~한 상황이라면?"
|
||||||
|
5. **후속 질문**: 답변에 따라 심화 질문
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**작성일**: 2025-10-28
|
||||||
|
**프로젝트**: Site11 Microservices Platform
|
||||||
|
**대상**: Full-stack Developer
|
||||||
185
k8s/AUTOSCALING-GUIDE.md
Normal file
185
k8s/AUTOSCALING-GUIDE.md
Normal file
@ -0,0 +1,185 @@
|
|||||||
|
# AUTOSCALING-GUIDE
|
||||||
|
|
||||||
|
## 로컬 환경에서 오토스케일링 테스트
|
||||||
|
|
||||||
|
### 현재 환경
|
||||||
|
- Docker Desktop K8s: 4개 노드 (1 control-plane, 3 workers)
|
||||||
|
- HPA 설정: CPU 70%, Memory 80% 기준
|
||||||
|
- Pod 확장: 2-10 replicas
|
||||||
|
|
||||||
|
### Cluster Autoscaler 대안
|
||||||
|
|
||||||
|
#### 1. **HPA (Horizontal Pod Autoscaler)** ✅ 현재 사용중
|
||||||
|
```bash
|
||||||
|
# HPA 상태 확인
|
||||||
|
kubectl -n site11-pipeline get hpa
|
||||||
|
|
||||||
|
# 메트릭 서버 설치 (필요시)
|
||||||
|
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
|
||||||
|
|
||||||
|
# 부하 테스트
|
||||||
|
kubectl apply -f load-test.yaml
|
||||||
|
|
||||||
|
# 스케일링 관찰
|
||||||
|
kubectl -n site11-pipeline get hpa -w
|
||||||
|
kubectl -n site11-pipeline get pods -w
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 2. **VPA (Vertical Pod Autoscaler)**
|
||||||
|
Pod의 리소스 요청을 자동 조정
|
||||||
|
```bash
|
||||||
|
# VPA 설치
|
||||||
|
git clone https://github.com/kubernetes/autoscaler.git
|
||||||
|
cd autoscaler/vertical-pod-autoscaler
|
||||||
|
./hack/vpa-up.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 3. **Kind 다중 노드 시뮬레이션**
|
||||||
|
```bash
|
||||||
|
# 다중 노드 클러스터 생성
|
||||||
|
kind create cluster --config kind-multi-node.yaml
|
||||||
|
|
||||||
|
# 노드 추가 (수동)
|
||||||
|
docker run -d --name site11-worker4 \
|
||||||
|
--network kind \
|
||||||
|
kindest/node:v1.27.3
|
||||||
|
|
||||||
|
# 노드 제거
|
||||||
|
kubectl drain site11-worker4 --ignore-daemonsets
|
||||||
|
kubectl delete node site11-worker4
|
||||||
|
```
|
||||||
|
|
||||||
|
### 프로덕션 환경 (AWS EKS)
|
||||||
|
|
||||||
|
#### Cluster Autoscaler 설정
|
||||||
|
```yaml
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: cluster-autoscaler
|
||||||
|
namespace: kube-system
|
||||||
|
spec:
|
||||||
|
template:
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.27.0
|
||||||
|
name: cluster-autoscaler
|
||||||
|
command:
|
||||||
|
- ./cluster-autoscaler
|
||||||
|
- --v=4
|
||||||
|
- --stderrthreshold=info
|
||||||
|
- --cloud-provider=aws
|
||||||
|
- --skip-nodes-with-local-storage=false
|
||||||
|
- --expander=least-waste
|
||||||
|
- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/site11-cluster
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Karpenter (더 빠른 대안)
|
||||||
|
```yaml
|
||||||
|
apiVersion: karpenter.sh/v1alpha5
|
||||||
|
kind: Provisioner
|
||||||
|
metadata:
|
||||||
|
name: default
|
||||||
|
spec:
|
||||||
|
requirements:
|
||||||
|
- key: karpenter.sh/capacity-type
|
||||||
|
operator: In
|
||||||
|
values: ["spot", "on-demand"]
|
||||||
|
- key: node.kubernetes.io/instance-type
|
||||||
|
operator: In
|
||||||
|
values: ["t3.medium", "t3.large", "t3.xlarge"]
|
||||||
|
limits:
|
||||||
|
resources:
|
||||||
|
cpu: 1000
|
||||||
|
memory: 1000Gi
|
||||||
|
ttlSecondsAfterEmpty: 30
|
||||||
|
```
|
||||||
|
|
||||||
|
### 부하 테스트 시나리오
|
||||||
|
|
||||||
|
#### 1. CPU 부하 생성
|
||||||
|
```bash
|
||||||
|
kubectl run -n site11-pipeline stress-cpu \
|
||||||
|
--image=progrium/stress \
|
||||||
|
--restart=Never \
|
||||||
|
-- --cpu 2 --timeout 60s
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 2. 메모리 부하 생성
|
||||||
|
```bash
|
||||||
|
kubectl run -n site11-pipeline stress-memory \
|
||||||
|
--image=progrium/stress \
|
||||||
|
--restart=Never \
|
||||||
|
-- --vm 2 --vm-bytes 256M --timeout 60s
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 3. HTTP 부하 생성
|
||||||
|
```bash
|
||||||
|
# Apache Bench 사용
|
||||||
|
kubectl run -n site11-pipeline ab-test \
|
||||||
|
--image=httpd \
|
||||||
|
--restart=Never \
|
||||||
|
-- ab -n 10000 -c 100 http://console-backend:8000/
|
||||||
|
```
|
||||||
|
|
||||||
|
### 모니터링
|
||||||
|
|
||||||
|
#### 실시간 모니터링
|
||||||
|
```bash
|
||||||
|
# Pod 자동 스케일링 관찰
|
||||||
|
watch -n 1 'kubectl -n site11-pipeline get pods | grep Running | wc -l'
|
||||||
|
|
||||||
|
# 리소스 사용량
|
||||||
|
kubectl top nodes
|
||||||
|
kubectl -n site11-pipeline top pods
|
||||||
|
|
||||||
|
# HPA 상태
|
||||||
|
kubectl -n site11-pipeline describe hpa
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Grafana/Prometheus (선택사항)
|
||||||
|
```bash
|
||||||
|
# Prometheus Stack 설치
|
||||||
|
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
|
||||||
|
helm install monitoring prometheus-community/kube-prometheus-stack
|
||||||
|
```
|
||||||
|
|
||||||
|
### 로컬 테스트 권장사항
|
||||||
|
|
||||||
|
1. **현재 Docker Desktop에서 가능한 것:**
|
||||||
|
- HPA 기반 Pod 자동 스케일링 ✅
|
||||||
|
- 부하 테스트를 통한 스케일링 검증 ✅
|
||||||
|
- 4개 노드에 Pod 분산 배치 ✅
|
||||||
|
|
||||||
|
2. **제한사항:**
|
||||||
|
- 실제 노드 자동 추가/제거 ❌
|
||||||
|
- Spot Instance 시뮬레이션 ❌
|
||||||
|
- 실제 비용 최적화 테스트 ❌
|
||||||
|
|
||||||
|
3. **대안:**
|
||||||
|
- Minikube: `minikube node add` 명령으로 노드 추가 가능
|
||||||
|
- Kind: 수동으로 노드 컨테이너 추가 가능
|
||||||
|
- K3s: 가벼운 멀티노드 클러스터 구성 가능
|
||||||
|
|
||||||
|
### 실습 예제
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. 현재 상태 확인
|
||||||
|
kubectl -n site11-pipeline get hpa
|
||||||
|
kubectl -n site11-pipeline get pods | wc -l
|
||||||
|
|
||||||
|
# 2. 부하 생성
|
||||||
|
kubectl apply -f load-test.yaml
|
||||||
|
|
||||||
|
# 3. 스케일링 관찰 (별도 터미널)
|
||||||
|
kubectl -n site11-pipeline get hpa -w
|
||||||
|
|
||||||
|
# 4. Pod 증가 확인
|
||||||
|
kubectl -n site11-pipeline get pods -w
|
||||||
|
|
||||||
|
# 5. 부하 중지
|
||||||
|
kubectl -n site11-pipeline delete pod load-generator
|
||||||
|
|
||||||
|
# 6. 스케일 다운 관찰 (5분 후)
|
||||||
|
kubectl -n site11-pipeline get pods
|
||||||
|
```
|
||||||
103
k8s/AWS-DEPLOYMENT.md
Normal file
103
k8s/AWS-DEPLOYMENT.md
Normal file
@ -0,0 +1,103 @@
|
|||||||
|
# AWS Production Deployment Architecture
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
Production deployment on AWS with external managed services and EKS for workloads.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### External Infrastructure (AWS Managed Services)
|
||||||
|
- **RDS MongoDB Compatible**: DocumentDB or MongoDB Atlas
|
||||||
|
- **ElastiCache**: Redis for caching and queues
|
||||||
|
- **Amazon MSK**: Managed Kafka for event streaming
|
||||||
|
- **Amazon ECR**: Container registry
|
||||||
|
- **S3**: Object storage (replaces MinIO)
|
||||||
|
- **OpenSearch**: Search engine (replaces Solr)
|
||||||
|
|
||||||
|
### EKS Workloads (Kubernetes)
|
||||||
|
- Pipeline workers (auto-scaling)
|
||||||
|
- API services
|
||||||
|
- Frontend applications
|
||||||
|
|
||||||
|
## Local Development Setup (AWS Simulation)
|
||||||
|
|
||||||
|
### 1. Infrastructure Layer (Docker Compose)
|
||||||
|
Simulates AWS managed services locally:
|
||||||
|
```yaml
|
||||||
|
# docker-compose-infra.yml
|
||||||
|
services:
|
||||||
|
mongodb: # Simulates DocumentDB
|
||||||
|
redis: # Simulates ElastiCache
|
||||||
|
kafka: # Simulates MSK
|
||||||
|
registry: # Simulates ECR
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. K8s Layer (Local Kubernetes)
|
||||||
|
Deploy workloads that will run on EKS:
|
||||||
|
```yaml
|
||||||
|
# K8s deployments
|
||||||
|
- pipeline-rss-collector
|
||||||
|
- pipeline-google-search
|
||||||
|
- pipeline-translator
|
||||||
|
- pipeline-ai-article-generator
|
||||||
|
- pipeline-image-generator
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Configuration
|
||||||
|
|
||||||
|
### Development (Local)
|
||||||
|
```yaml
|
||||||
|
# External services on host machine
|
||||||
|
MONGODB_URL: "mongodb://host.docker.internal:27017"
|
||||||
|
REDIS_URL: "redis://host.docker.internal:6379"
|
||||||
|
KAFKA_BROKERS: "host.docker.internal:9092"
|
||||||
|
REGISTRY_URL: "host.docker.internal:5555"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Production (AWS)
|
||||||
|
```yaml
|
||||||
|
# AWS managed services
|
||||||
|
MONGODB_URL: "mongodb://documentdb.region.amazonaws.com:27017"
|
||||||
|
REDIS_URL: "redis://cache.xxxxx.cache.amazonaws.com:6379"
|
||||||
|
KAFKA_BROKERS: "kafka.region.amazonaws.com:9092"
|
||||||
|
REGISTRY_URL: "xxxxx.dkr.ecr.region.amazonaws.com"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deployment Steps
|
||||||
|
|
||||||
|
### Local Development
|
||||||
|
1. Start infrastructure (Docker Compose)
|
||||||
|
2. Push images to local registry
|
||||||
|
3. Deploy to local K8s
|
||||||
|
4. Use host.docker.internal for service discovery
|
||||||
|
|
||||||
|
### AWS Production
|
||||||
|
1. Infrastructure provisioned via Terraform/CloudFormation
|
||||||
|
2. Push images to ECR
|
||||||
|
3. Deploy to EKS
|
||||||
|
4. Use AWS service endpoints
|
||||||
|
|
||||||
|
## Benefits of This Approach
|
||||||
|
1. **Cost Optimization**: Managed services reduce operational overhead
|
||||||
|
2. **Scalability**: Auto-scaling for K8s workloads
|
||||||
|
3. **High Availability**: AWS managed services provide built-in HA
|
||||||
|
4. **Security**: VPC isolation, IAM roles, secrets management
|
||||||
|
5. **Monitoring**: CloudWatch integration
|
||||||
|
|
||||||
|
## Migration Path
|
||||||
|
1. Local development with Docker Compose + K8s
|
||||||
|
2. Stage environment on AWS with smaller instances
|
||||||
|
3. Production deployment with full scaling
|
||||||
|
|
||||||
|
## Cost Considerations
|
||||||
|
- **DocumentDB**: ~$200/month (minimum)
|
||||||
|
- **ElastiCache**: ~$50/month (t3.micro)
|
||||||
|
- **MSK**: ~$140/month (kafka.t3.small)
|
||||||
|
- **EKS**: ~$73/month (cluster) + EC2 costs
|
||||||
|
- **ECR**: ~$10/month (storage)
|
||||||
|
|
||||||
|
## Security Best Practices
|
||||||
|
1. Use AWS Secrets Manager for API keys
|
||||||
|
2. VPC endpoints for service communication
|
||||||
|
3. IAM roles for service accounts (IRSA)
|
||||||
|
4. Network policies in K8s
|
||||||
|
5. Encryption at rest and in transit
|
||||||
198
k8s/K8S-DEPLOYMENT-GUIDE.md
Normal file
198
k8s/K8S-DEPLOYMENT-GUIDE.md
Normal file
@ -0,0 +1,198 @@
|
|||||||
|
# K8S-DEPLOYMENT-GUIDE
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
Site11 파이프라인 시스템의 K8s 배포 가이드입니다. AWS 프로덕션 환경과 유사하게 인프라는 K8s 외부에, 워커들은 K8s 내부에 배포합니다.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────┐
|
||||||
|
│ Docker Compose │
|
||||||
|
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||||
|
│ │ MongoDB │ │ Redis │ │ Kafka │ │
|
||||||
|
│ └──────────┘ └──────────┘ └──────────┘ │
|
||||||
|
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||||
|
│ │Scheduler │ │ Monitor │ │Lang Sync │ │
|
||||||
|
│ └──────────┘ └──────────┘ └──────────┘ │
|
||||||
|
└─────────────────────────────────────────────────┘
|
||||||
|
↕
|
||||||
|
┌─────────────────────────────────────────────────┐
|
||||||
|
│ Kubernetes │
|
||||||
|
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||||
|
│ │ RSS │ │ Search │ │Translator│ │
|
||||||
|
│ └──────────┘ └──────────┘ └──────────┘ │
|
||||||
|
│ ┌──────────┐ ┌──────────┐ │
|
||||||
|
│ │ AI Gen │ │Image Gen │ │
|
||||||
|
│ └──────────┘ └──────────┘ │
|
||||||
|
└─────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deployment Options
|
||||||
|
|
||||||
|
### Option 1: Docker Hub (Recommended)
|
||||||
|
가장 간단하고 안정적인 방법입니다.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Docker Hub 계정 설정
|
||||||
|
export DOCKER_HUB_USER=your-username
|
||||||
|
|
||||||
|
# 2. Docker Hub 로그인
|
||||||
|
docker login
|
||||||
|
|
||||||
|
# 3. 배포 실행
|
||||||
|
cd k8s/pipeline
|
||||||
|
./deploy-dockerhub.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
**장점:**
|
||||||
|
- 설정이 간단함
|
||||||
|
- 어떤 K8s 클러스터에서도 작동
|
||||||
|
- 이미지 버전 관리 용이
|
||||||
|
|
||||||
|
**단점:**
|
||||||
|
- Docker Hub 계정 필요
|
||||||
|
- 이미지 업로드 시간 소요
|
||||||
|
|
||||||
|
### Option 2: Local Registry
|
||||||
|
로컬 개발 환경용 (복잡함)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. 로컬 레지스트리 시작
|
||||||
|
docker-compose -f docker-compose-hybrid.yml up -d registry
|
||||||
|
|
||||||
|
# 2. 이미지 태그 및 푸시
|
||||||
|
./deploy-local.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
**장점:**
|
||||||
|
- 인터넷 연결 불필요
|
||||||
|
- 빠른 이미지 전송
|
||||||
|
|
||||||
|
**단점:**
|
||||||
|
- Docker Desktop K8s 제한사항
|
||||||
|
- 추가 설정 필요
|
||||||
|
|
||||||
|
### Option 3: Kind Cluster
|
||||||
|
고급 사용자용
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Kind 클러스터 생성
|
||||||
|
kind create cluster --config kind-config.yaml
|
||||||
|
|
||||||
|
# 2. 이미지 로드 및 배포
|
||||||
|
./deploy-kind.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
**장점:**
|
||||||
|
- 완전한 K8s 환경
|
||||||
|
- 로컬 이미지 직접 사용 가능
|
||||||
|
|
||||||
|
**단점:**
|
||||||
|
- Kind 설치 필요
|
||||||
|
- 리소스 사용량 높음
|
||||||
|
|
||||||
|
## Infrastructure Setup
|
||||||
|
|
||||||
|
### 1. Start Infrastructure Services
|
||||||
|
```bash
|
||||||
|
# 인프라 서비스 시작 (MongoDB, Redis, Kafka, etc.)
|
||||||
|
docker-compose -f docker-compose-hybrid.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Verify Infrastructure
|
||||||
|
```bash
|
||||||
|
# 서비스 상태 확인
|
||||||
|
docker ps | grep site11
|
||||||
|
|
||||||
|
# 로그 확인
|
||||||
|
docker-compose -f docker-compose-hybrid.yml logs -f
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common Issues
|
||||||
|
|
||||||
|
### Issue 1: ImagePullBackOff
|
||||||
|
**원인:** K8s가 이미지를 찾을 수 없음
|
||||||
|
**해결:** Docker Hub 사용 또는 Kind 클러스터 사용
|
||||||
|
|
||||||
|
### Issue 2: Connection to External Services Failed
|
||||||
|
**원인:** K8s Pod에서 Docker 서비스 접근 불가
|
||||||
|
**해결:** `host.docker.internal` 사용 확인
|
||||||
|
|
||||||
|
### Issue 3: Pods Not Starting
|
||||||
|
**원인:** 리소스 부족
|
||||||
|
**해결:** 리소스 limits 조정 또는 노드 추가
|
||||||
|
|
||||||
|
## Monitoring
|
||||||
|
|
||||||
|
### View Pod Status
|
||||||
|
```bash
|
||||||
|
kubectl -n site11-pipeline get pods -w
|
||||||
|
```
|
||||||
|
|
||||||
|
### View Logs
|
||||||
|
```bash
|
||||||
|
# 특정 서비스 로그
|
||||||
|
kubectl -n site11-pipeline logs -f deployment/pipeline-translator
|
||||||
|
|
||||||
|
# 모든 Pod 로그
|
||||||
|
kubectl -n site11-pipeline logs -l app=pipeline-translator
|
||||||
|
```
|
||||||
|
|
||||||
|
### Check Auto-scaling
|
||||||
|
```bash
|
||||||
|
kubectl -n site11-pipeline get hpa
|
||||||
|
```
|
||||||
|
|
||||||
|
### Monitor Queue Status
|
||||||
|
```bash
|
||||||
|
docker-compose -f docker-compose-hybrid.yml logs -f pipeline-monitor
|
||||||
|
```
|
||||||
|
|
||||||
|
## Scaling
|
||||||
|
|
||||||
|
### Manual Scaling
|
||||||
|
```bash
|
||||||
|
# Scale up
|
||||||
|
kubectl -n site11-pipeline scale deployment pipeline-translator --replicas=5
|
||||||
|
|
||||||
|
# Scale down
|
||||||
|
kubectl -n site11-pipeline scale deployment pipeline-translator --replicas=2
|
||||||
|
```
|
||||||
|
|
||||||
|
### Auto-scaling Configuration
|
||||||
|
HPA는 CPU 70%, Memory 80% 기준으로 자동 확장됩니다.
|
||||||
|
|
||||||
|
## Cleanup
|
||||||
|
|
||||||
|
### Remove K8s Resources
|
||||||
|
```bash
|
||||||
|
kubectl delete namespace site11-pipeline
|
||||||
|
```
|
||||||
|
|
||||||
|
### Stop Infrastructure
|
||||||
|
```bash
|
||||||
|
docker-compose -f docker-compose-hybrid.yml down
|
||||||
|
```
|
||||||
|
|
||||||
|
### Remove Kind Cluster (if used)
|
||||||
|
```bash
|
||||||
|
kind delete cluster --name site11-cluster
|
||||||
|
```
|
||||||
|
|
||||||
|
## Production Deployment
|
||||||
|
|
||||||
|
실제 AWS 프로덕션 환경에서는:
|
||||||
|
1. MongoDB → Amazon DocumentDB
|
||||||
|
2. Redis → Amazon ElastiCache
|
||||||
|
3. Kafka → Amazon MSK
|
||||||
|
4. Local Registry → Amazon ECR
|
||||||
|
5. K8s → Amazon EKS
|
||||||
|
|
||||||
|
ConfigMap에서 연결 정보만 변경하면 됩니다.
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
1. **이미지 버전 관리**: latest 대신 구체적인 버전 태그 사용
|
||||||
|
2. **리소스 제한**: 적절한 requests/limits 설정
|
||||||
|
3. **모니터링**: Prometheus/Grafana 등 모니터링 도구 설치
|
||||||
|
4. **로그 관리**: 중앙 로그 수집 시스템 구축
|
||||||
|
5. **백업**: MongoDB 정기 백업 설정
|
||||||
188
k8s/KIND-AUTOSCALING.md
Normal file
188
k8s/KIND-AUTOSCALING.md
Normal file
@ -0,0 +1,188 @@
|
|||||||
|
# KIND-AUTOSCALING
|
||||||
|
|
||||||
|
## Kind 환경에서 Cluster Autoscaler 시뮬레이션
|
||||||
|
|
||||||
|
### 문제점
|
||||||
|
- Kind는 Docker 컨테이너 기반이라 실제 클라우드 리소스가 없음
|
||||||
|
- 진짜 Cluster Autoscaler는 AWS/GCP/Azure API가 필요
|
||||||
|
|
||||||
|
### 해결책
|
||||||
|
|
||||||
|
#### 1. **수동 노드 스케일링 스크립트** (실용적)
|
||||||
|
```bash
|
||||||
|
# 스크립트 실행
|
||||||
|
chmod +x kind-autoscaler.sh
|
||||||
|
./kind-autoscaler.sh
|
||||||
|
|
||||||
|
# 기능:
|
||||||
|
- CPU 사용률 모니터링
|
||||||
|
- Pending Pod 감지
|
||||||
|
- 자동 노드 추가/제거
|
||||||
|
- Min: 3, Max: 10 노드
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 2. **Kwok (Kubernetes WithOut Kubelet)** - 가상 노드
|
||||||
|
```bash
|
||||||
|
# Kwok 설치
|
||||||
|
kubectl apply -f https://github.com/kubernetes-sigs/kwok/releases/download/v0.4.0/kwok.yaml
|
||||||
|
|
||||||
|
# 가상 노드 생성
|
||||||
|
kubectl apply -f - <<EOF
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Node
|
||||||
|
metadata:
|
||||||
|
name: fake-node-1
|
||||||
|
annotations:
|
||||||
|
kwok.x-k8s.io/node: fake
|
||||||
|
labels:
|
||||||
|
type: virtual
|
||||||
|
node.kubernetes.io/instance-type: m5.large
|
||||||
|
spec:
|
||||||
|
taints:
|
||||||
|
- key: kwok.x-k8s.io/node
|
||||||
|
effect: NoSchedule
|
||||||
|
EOF
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 3. **Cluster API + Docker (CAPD)**
|
||||||
|
```bash
|
||||||
|
# Cluster API 설치
|
||||||
|
clusterctl init --infrastructure docker
|
||||||
|
|
||||||
|
# MachineDeployment로 노드 관리
|
||||||
|
kubectl apply -f - <<EOF
|
||||||
|
apiVersion: cluster.x-k8s.io/v1beta1
|
||||||
|
kind: MachineDeployment
|
||||||
|
metadata:
|
||||||
|
name: worker-md
|
||||||
|
spec:
|
||||||
|
replicas: 3 # 동적 조정 가능
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
cluster.x-k8s.io/deployment-name: worker-md
|
||||||
|
template:
|
||||||
|
spec:
|
||||||
|
clusterName: docker-desktop
|
||||||
|
version: v1.27.3
|
||||||
|
EOF
|
||||||
|
```
|
||||||
|
|
||||||
|
### 실습: Kind 노드 수동 추가/제거
|
||||||
|
|
||||||
|
#### 노드 추가
|
||||||
|
```bash
|
||||||
|
# 새 워커 노드 추가
|
||||||
|
docker run -d \
|
||||||
|
--name desktop-worker7 \
|
||||||
|
--network kind \
|
||||||
|
--label io.x-k8s.kind.cluster=docker-desktop \
|
||||||
|
--label io.x-k8s.kind.role=worker \
|
||||||
|
--privileged \
|
||||||
|
--security-opt seccomp=unconfined \
|
||||||
|
--security-opt apparmor=unconfined \
|
||||||
|
--tmpfs /tmp \
|
||||||
|
--tmpfs /run \
|
||||||
|
--volume /var \
|
||||||
|
--volume /lib/modules:/lib/modules:ro \
|
||||||
|
kindest/node:v1.27.3
|
||||||
|
|
||||||
|
# 노드 합류 대기
|
||||||
|
sleep 20
|
||||||
|
|
||||||
|
# 노드 확인
|
||||||
|
kubectl get nodes
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 노드 제거
|
||||||
|
```bash
|
||||||
|
# 노드 드레인
|
||||||
|
kubectl drain desktop-worker7 --ignore-daemonsets --force
|
||||||
|
|
||||||
|
# 노드 삭제
|
||||||
|
kubectl delete node desktop-worker7
|
||||||
|
|
||||||
|
# 컨테이너 정지 및 제거
|
||||||
|
docker stop desktop-worker7
|
||||||
|
docker rm desktop-worker7
|
||||||
|
```
|
||||||
|
|
||||||
|
### HPA와 함께 사용
|
||||||
|
|
||||||
|
#### 1. Metrics Server 확인
|
||||||
|
```bash
|
||||||
|
kubectl -n kube-system get deployment metrics-server
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 2. 부하 생성 및 Pod 스케일링
|
||||||
|
```bash
|
||||||
|
# 부하 생성
|
||||||
|
kubectl run -it --rm load-generator --image=busybox -- /bin/sh
|
||||||
|
# 내부에서: while true; do wget -q -O- http://console-backend.site11-pipeline:8000; done
|
||||||
|
|
||||||
|
# HPA 모니터링
|
||||||
|
kubectl -n site11-pipeline get hpa -w
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 3. 노드 부족 시뮬레이션
|
||||||
|
```bash
|
||||||
|
# 많은 Pod 생성
|
||||||
|
kubectl -n site11-pipeline scale deployment pipeline-translator --replicas=20
|
||||||
|
|
||||||
|
# Pending Pod 확인
|
||||||
|
kubectl get pods --all-namespaces --field-selector=status.phase=Pending
|
||||||
|
|
||||||
|
# 수동으로 노드 추가 (위 스크립트 사용)
|
||||||
|
./kind-autoscaler.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### 프로덕션 마이그레이션 준비
|
||||||
|
|
||||||
|
#### AWS EKS에서 실제 Cluster Autoscaler
|
||||||
|
```yaml
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: cluster-autoscaler
|
||||||
|
namespace: kube-system
|
||||||
|
spec:
|
||||||
|
template:
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.27.0
|
||||||
|
name: cluster-autoscaler
|
||||||
|
command:
|
||||||
|
- ./cluster-autoscaler
|
||||||
|
- --v=4
|
||||||
|
- --cloud-provider=aws
|
||||||
|
- --skip-nodes-with-local-storage=false
|
||||||
|
- --expander=least-waste
|
||||||
|
- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled
|
||||||
|
env:
|
||||||
|
- name: AWS_REGION
|
||||||
|
value: us-west-2
|
||||||
|
```
|
||||||
|
|
||||||
|
### 권장사항
|
||||||
|
|
||||||
|
1. **로컬 테스트**:
|
||||||
|
- HPA로 Pod 자동 스케일링 ✅
|
||||||
|
- 수동 스크립트로 노드 추가/제거 시뮬레이션 ✅
|
||||||
|
|
||||||
|
2. **스테이징 환경**:
|
||||||
|
- 실제 클라우드에 작은 클러스터
|
||||||
|
- 진짜 Cluster Autoscaler 테스트
|
||||||
|
|
||||||
|
3. **프로덕션**:
|
||||||
|
- AWS EKS + Cluster Autoscaler
|
||||||
|
- 또는 Karpenter (더 빠름)
|
||||||
|
|
||||||
|
### 모니터링 대시보드
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# K9s 설치 (TUI 대시보드)
|
||||||
|
brew install k9s
|
||||||
|
k9s
|
||||||
|
|
||||||
|
# 또는 Lens 사용 (GUI)
|
||||||
|
# https://k8slens.dev/
|
||||||
|
```
|
||||||
124
k8s/kind-autoscaler.sh
Executable file
124
k8s/kind-autoscaler.sh
Executable file
@ -0,0 +1,124 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# Kind Cluster Autoscaler Simulator
|
||||||
|
# ==================================
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
CLUSTER_NAME="${KIND_CLUSTER:-docker-desktop}"
|
||||||
|
MIN_NODES=3
|
||||||
|
MAX_NODES=10
|
||||||
|
SCALE_UP_THRESHOLD=80 # CPU usage %
|
||||||
|
SCALE_DOWN_THRESHOLD=30
|
||||||
|
CHECK_INTERVAL=30
|
||||||
|
|
||||||
|
# Colors
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
RED='\033[0;31m'
|
||||||
|
NC='\033[0m'
|
||||||
|
|
||||||
|
echo "🚀 Kind Cluster Autoscaler Simulator"
|
||||||
|
echo "====================================="
|
||||||
|
echo "Cluster: $CLUSTER_NAME"
|
||||||
|
echo "Min nodes: $MIN_NODES, Max nodes: $MAX_NODES"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Function to get current worker node count
|
||||||
|
get_node_count() {
|
||||||
|
kubectl get nodes --no-headers | grep -v control-plane | wc -l
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to get average CPU usage
|
||||||
|
get_cpu_usage() {
|
||||||
|
kubectl top nodes --no-headers | grep -v control-plane | \
|
||||||
|
awk '{sum+=$3; count++} END {if(count>0) print int(sum/count); else print 0}'
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to add a node
|
||||||
|
add_node() {
|
||||||
|
local current_count=$1
|
||||||
|
local new_node_num=$((current_count + 1))
|
||||||
|
local node_name="desktop-worker${new_node_num}"
|
||||||
|
|
||||||
|
echo -e "${GREEN}📈 Scaling up: Adding node $node_name${NC}"
|
||||||
|
|
||||||
|
# Create new Kind worker node container
|
||||||
|
docker run -d \
|
||||||
|
--name "$node_name" \
|
||||||
|
--hostname "$node_name" \
|
||||||
|
--network kind \
|
||||||
|
--restart on-failure:1 \
|
||||||
|
--label io.x-k8s.kind.cluster="$CLUSTER_NAME" \
|
||||||
|
--label io.x-k8s.kind.role=worker \
|
||||||
|
--privileged \
|
||||||
|
--security-opt seccomp=unconfined \
|
||||||
|
--security-opt apparmor=unconfined \
|
||||||
|
--tmpfs /tmp \
|
||||||
|
--tmpfs /run \
|
||||||
|
--volume /var \
|
||||||
|
--volume /lib/modules:/lib/modules:ro \
|
||||||
|
kindest/node:v1.27.3
|
||||||
|
|
||||||
|
# Wait for node to join
|
||||||
|
sleep 10
|
||||||
|
|
||||||
|
# Label the new node
|
||||||
|
kubectl label node "$node_name" node-role.kubernetes.io/worker=true --overwrite
|
||||||
|
|
||||||
|
echo -e "${GREEN}✅ Node $node_name added successfully${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to remove a node
|
||||||
|
remove_node() {
|
||||||
|
local node_to_remove=$(kubectl get nodes --no-headers | grep -v control-plane | tail -1 | awk '{print $1}')
|
||||||
|
|
||||||
|
if [ -z "$node_to_remove" ]; then
|
||||||
|
echo -e "${YELLOW}⚠️ No nodes to remove${NC}"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo -e "${YELLOW}📉 Scaling down: Removing node $node_to_remove${NC}"
|
||||||
|
|
||||||
|
# Drain the node
|
||||||
|
kubectl drain "$node_to_remove" --ignore-daemonsets --delete-emptydir-data --force
|
||||||
|
|
||||||
|
# Delete the node
|
||||||
|
kubectl delete node "$node_to_remove"
|
||||||
|
|
||||||
|
# Stop and remove the container
|
||||||
|
docker stop "$node_to_remove"
|
||||||
|
docker rm "$node_to_remove"
|
||||||
|
|
||||||
|
echo -e "${YELLOW}✅ Node $node_to_remove removed successfully${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Main monitoring loop
|
||||||
|
echo "Starting autoscaler loop (Ctrl+C to stop)..."
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
while true; do
|
||||||
|
NODE_COUNT=$(get_node_count)
|
||||||
|
CPU_USAGE=$(get_cpu_usage)
|
||||||
|
PENDING_PODS=$(kubectl get pods --all-namespaces --field-selector=status.phase=Pending --no-headers 2>/dev/null | wc -l)
|
||||||
|
|
||||||
|
echo "$(date '+%H:%M:%S') - Nodes: $NODE_COUNT | CPU: ${CPU_USAGE}% | Pending Pods: $PENDING_PODS"
|
||||||
|
|
||||||
|
# Scale up conditions
|
||||||
|
if [ "$PENDING_PODS" -gt 0 ] || [ "$CPU_USAGE" -gt "$SCALE_UP_THRESHOLD" ]; then
|
||||||
|
if [ "$NODE_COUNT" -lt "$MAX_NODES" ]; then
|
||||||
|
echo -e "${GREEN}🔺 Scale up triggered (CPU: ${CPU_USAGE}%, Pending: ${PENDING_PODS})${NC}"
|
||||||
|
add_node "$NODE_COUNT"
|
||||||
|
else
|
||||||
|
echo -e "${YELLOW}⚠️ Already at max nodes ($MAX_NODES)${NC}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Scale down conditions
|
||||||
|
elif [ "$CPU_USAGE" -lt "$SCALE_DOWN_THRESHOLD" ] && [ "$NODE_COUNT" -gt "$MIN_NODES" ]; then
|
||||||
|
echo -e "${YELLOW}🔻 Scale down triggered (CPU: ${CPU_USAGE}%)${NC}"
|
||||||
|
remove_node
|
||||||
|
fi
|
||||||
|
|
||||||
|
sleep "$CHECK_INTERVAL"
|
||||||
|
done
|
||||||
71
k8s/kind-dev-cluster.yaml
Normal file
71
k8s/kind-dev-cluster.yaml
Normal file
@ -0,0 +1,71 @@
|
|||||||
|
kind: Cluster
|
||||||
|
apiVersion: kind.x-k8s.io/v1alpha4
|
||||||
|
name: site11-dev
|
||||||
|
|
||||||
|
# 노드 구성 (1 Control Plane + 4 Workers = 5 Nodes)
|
||||||
|
nodes:
|
||||||
|
# Control Plane (마스터 노드)
|
||||||
|
- role: control-plane
|
||||||
|
kubeadmConfigPatches:
|
||||||
|
- |
|
||||||
|
kind: InitConfiguration
|
||||||
|
nodeRegistration:
|
||||||
|
kubeletExtraArgs:
|
||||||
|
node-labels: "node-type=control-plane"
|
||||||
|
extraPortMappings:
|
||||||
|
# Console Frontend
|
||||||
|
- containerPort: 30080
|
||||||
|
hostPort: 3000
|
||||||
|
protocol: TCP
|
||||||
|
# Console Backend
|
||||||
|
- containerPort: 30081
|
||||||
|
hostPort: 8000
|
||||||
|
protocol: TCP
|
||||||
|
|
||||||
|
# Worker Node 1 (Console 서비스용)
|
||||||
|
- role: worker
|
||||||
|
labels:
|
||||||
|
workload: console
|
||||||
|
node-type: worker
|
||||||
|
kubeadmConfigPatches:
|
||||||
|
- |
|
||||||
|
kind: JoinConfiguration
|
||||||
|
nodeRegistration:
|
||||||
|
kubeletExtraArgs:
|
||||||
|
node-labels: "workload=console"
|
||||||
|
|
||||||
|
# Worker Node 2 (Pipeline 서비스용 - 수집)
|
||||||
|
- role: worker
|
||||||
|
labels:
|
||||||
|
workload: pipeline-collector
|
||||||
|
node-type: worker
|
||||||
|
kubeadmConfigPatches:
|
||||||
|
- |
|
||||||
|
kind: JoinConfiguration
|
||||||
|
nodeRegistration:
|
||||||
|
kubeletExtraArgs:
|
||||||
|
node-labels: "workload=pipeline-collector"
|
||||||
|
|
||||||
|
# Worker Node 3 (Pipeline 서비스용 - 처리)
|
||||||
|
- role: worker
|
||||||
|
labels:
|
||||||
|
workload: pipeline-processor
|
||||||
|
node-type: worker
|
||||||
|
kubeadmConfigPatches:
|
||||||
|
- |
|
||||||
|
kind: JoinConfiguration
|
||||||
|
nodeRegistration:
|
||||||
|
kubeletExtraArgs:
|
||||||
|
node-labels: "workload=pipeline-processor"
|
||||||
|
|
||||||
|
# Worker Node 4 (Pipeline 서비스용 - 생성)
|
||||||
|
- role: worker
|
||||||
|
labels:
|
||||||
|
workload: pipeline-generator
|
||||||
|
node-type: worker
|
||||||
|
kubeadmConfigPatches:
|
||||||
|
- |
|
||||||
|
kind: JoinConfiguration
|
||||||
|
nodeRegistration:
|
||||||
|
kubeletExtraArgs:
|
||||||
|
node-labels: "workload=pipeline-generator"
|
||||||
23
k8s/kind-multi-node.yaml
Normal file
23
k8s/kind-multi-node.yaml
Normal file
@ -0,0 +1,23 @@
|
|||||||
|
kind: Cluster
|
||||||
|
apiVersion: kind.x-k8s.io/v1alpha4
|
||||||
|
name: site11-autoscale
|
||||||
|
nodes:
|
||||||
|
# Control plane
|
||||||
|
- role: control-plane
|
||||||
|
extraPortMappings:
|
||||||
|
- containerPort: 30000
|
||||||
|
hostPort: 30000
|
||||||
|
protocol: TCP
|
||||||
|
- containerPort: 30001
|
||||||
|
hostPort: 30001
|
||||||
|
protocol: TCP
|
||||||
|
# Initial worker nodes
|
||||||
|
- role: worker
|
||||||
|
labels:
|
||||||
|
node-role.kubernetes.io/worker: "true"
|
||||||
|
- role: worker
|
||||||
|
labels:
|
||||||
|
node-role.kubernetes.io/worker: "true"
|
||||||
|
- role: worker
|
||||||
|
labels:
|
||||||
|
node-role.kubernetes.io/worker: "true"
|
||||||
79
k8s/kind/console-backend.yaml
Normal file
79
k8s/kind/console-backend.yaml
Normal file
@ -0,0 +1,79 @@
|
|||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: console-backend
|
||||||
|
namespace: site11-console
|
||||||
|
labels:
|
||||||
|
app: console-backend
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: console-backend
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: console-backend
|
||||||
|
spec:
|
||||||
|
nodeSelector:
|
||||||
|
workload: console
|
||||||
|
containers:
|
||||||
|
- name: console-backend
|
||||||
|
image: yakenator/site11-console-backend:latest
|
||||||
|
imagePullPolicy: IfNotPresent
|
||||||
|
ports:
|
||||||
|
- containerPort: 8000
|
||||||
|
protocol: TCP
|
||||||
|
env:
|
||||||
|
- name: ENV
|
||||||
|
value: "development"
|
||||||
|
- name: DEBUG
|
||||||
|
value: "true"
|
||||||
|
- name: MONGODB_URL
|
||||||
|
value: "mongodb://site11-mongodb:27017"
|
||||||
|
- name: DB_NAME
|
||||||
|
value: "console_db"
|
||||||
|
- name: REDIS_URL
|
||||||
|
value: "redis://site11-redis:6379"
|
||||||
|
- name: JWT_SECRET_KEY
|
||||||
|
value: "dev-secret-key-please-change-in-production"
|
||||||
|
- name: JWT_ALGORITHM
|
||||||
|
value: "HS256"
|
||||||
|
- name: ACCESS_TOKEN_EXPIRE_MINUTES
|
||||||
|
value: "30"
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
memory: "256Mi"
|
||||||
|
cpu: "100m"
|
||||||
|
limits:
|
||||||
|
memory: "512Mi"
|
||||||
|
cpu: "500m"
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 8000
|
||||||
|
initialDelaySeconds: 10
|
||||||
|
periodSeconds: 5
|
||||||
|
livenessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 8000
|
||||||
|
initialDelaySeconds: 30
|
||||||
|
periodSeconds: 10
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: console-backend
|
||||||
|
namespace: site11-console
|
||||||
|
labels:
|
||||||
|
app: console-backend
|
||||||
|
spec:
|
||||||
|
type: NodePort
|
||||||
|
selector:
|
||||||
|
app: console-backend
|
||||||
|
ports:
|
||||||
|
- port: 8000
|
||||||
|
targetPort: 8000
|
||||||
|
nodePort: 30081
|
||||||
|
protocol: TCP
|
||||||
65
k8s/kind/console-frontend.yaml
Normal file
65
k8s/kind/console-frontend.yaml
Normal file
@ -0,0 +1,65 @@
|
|||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: console-frontend
|
||||||
|
namespace: site11-console
|
||||||
|
labels:
|
||||||
|
app: console-frontend
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: console-frontend
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: console-frontend
|
||||||
|
spec:
|
||||||
|
nodeSelector:
|
||||||
|
workload: console
|
||||||
|
containers:
|
||||||
|
- name: console-frontend
|
||||||
|
image: yakenator/site11-console-frontend:latest
|
||||||
|
imagePullPolicy: IfNotPresent
|
||||||
|
ports:
|
||||||
|
- containerPort: 80
|
||||||
|
protocol: TCP
|
||||||
|
env:
|
||||||
|
- name: VITE_API_URL
|
||||||
|
value: "http://localhost:8000"
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
memory: "128Mi"
|
||||||
|
cpu: "50m"
|
||||||
|
limits:
|
||||||
|
memory: "256Mi"
|
||||||
|
cpu: "200m"
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /
|
||||||
|
port: 80
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 5
|
||||||
|
livenessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /
|
||||||
|
port: 80
|
||||||
|
initialDelaySeconds: 15
|
||||||
|
periodSeconds: 10
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: console-frontend
|
||||||
|
namespace: site11-console
|
||||||
|
labels:
|
||||||
|
app: console-frontend
|
||||||
|
spec:
|
||||||
|
type: NodePort
|
||||||
|
selector:
|
||||||
|
app: console-frontend
|
||||||
|
ports:
|
||||||
|
- port: 3000
|
||||||
|
targetPort: 80
|
||||||
|
nodePort: 30080
|
||||||
|
protocol: TCP
|
||||||
108
k8s/kind/console-mongodb-redis.yaml
Normal file
108
k8s/kind/console-mongodb-redis.yaml
Normal file
@ -0,0 +1,108 @@
|
|||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: mongodb
|
||||||
|
namespace: site11-console
|
||||||
|
labels:
|
||||||
|
app: mongodb
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: mongodb
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: mongodb
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: mongodb
|
||||||
|
image: mongo:7.0
|
||||||
|
ports:
|
||||||
|
- containerPort: 27017
|
||||||
|
protocol: TCP
|
||||||
|
env:
|
||||||
|
- name: MONGO_INITDB_DATABASE
|
||||||
|
value: "console_db"
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
memory: "512Mi"
|
||||||
|
cpu: "250m"
|
||||||
|
limits:
|
||||||
|
memory: "1Gi"
|
||||||
|
cpu: "500m"
|
||||||
|
volumeMounts:
|
||||||
|
- name: mongodb-data
|
||||||
|
mountPath: /data/db
|
||||||
|
volumes:
|
||||||
|
- name: mongodb-data
|
||||||
|
emptyDir: {}
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: mongodb
|
||||||
|
namespace: site11-console
|
||||||
|
labels:
|
||||||
|
app: mongodb
|
||||||
|
spec:
|
||||||
|
type: ClusterIP
|
||||||
|
selector:
|
||||||
|
app: mongodb
|
||||||
|
ports:
|
||||||
|
- port: 27017
|
||||||
|
targetPort: 27017
|
||||||
|
protocol: TCP
|
||||||
|
---
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: redis
|
||||||
|
namespace: site11-console
|
||||||
|
labels:
|
||||||
|
app: redis
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: redis
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: redis
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: redis
|
||||||
|
image: redis:7-alpine
|
||||||
|
ports:
|
||||||
|
- containerPort: 6379
|
||||||
|
protocol: TCP
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
memory: "128Mi"
|
||||||
|
cpu: "100m"
|
||||||
|
limits:
|
||||||
|
memory: "256Mi"
|
||||||
|
cpu: "200m"
|
||||||
|
volumeMounts:
|
||||||
|
- name: redis-data
|
||||||
|
mountPath: /data
|
||||||
|
volumes:
|
||||||
|
- name: redis-data
|
||||||
|
emptyDir: {}
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: redis
|
||||||
|
namespace: site11-console
|
||||||
|
labels:
|
||||||
|
app: redis
|
||||||
|
spec:
|
||||||
|
type: ClusterIP
|
||||||
|
selector:
|
||||||
|
app: redis
|
||||||
|
ports:
|
||||||
|
- port: 6379
|
||||||
|
targetPort: 6379
|
||||||
|
protocol: TCP
|
||||||
21
k8s/load-test.yaml
Normal file
21
k8s/load-test.yaml
Normal file
@ -0,0 +1,21 @@
|
|||||||
|
apiVersion: v1
|
||||||
|
kind: Pod
|
||||||
|
metadata:
|
||||||
|
name: load-generator
|
||||||
|
namespace: site11-pipeline
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: busybox
|
||||||
|
image: busybox
|
||||||
|
command:
|
||||||
|
- /bin/sh
|
||||||
|
- -c
|
||||||
|
- |
|
||||||
|
echo "Starting load test on console-backend..."
|
||||||
|
while true; do
|
||||||
|
for i in $(seq 1 100); do
|
||||||
|
wget -q -O- http://console-backend:8000/health &
|
||||||
|
done
|
||||||
|
wait
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
78
k8s/mock-cluster-autoscaler.yaml
Normal file
78
k8s/mock-cluster-autoscaler.yaml
Normal file
@ -0,0 +1,78 @@
|
|||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: cluster-autoscaler-status
|
||||||
|
namespace: kube-system
|
||||||
|
data:
|
||||||
|
nodes.max: "10"
|
||||||
|
nodes.min: "3"
|
||||||
|
---
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: cluster-autoscaler
|
||||||
|
namespace: kube-system
|
||||||
|
labels:
|
||||||
|
app: cluster-autoscaler
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: cluster-autoscaler
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: cluster-autoscaler
|
||||||
|
spec:
|
||||||
|
serviceAccountName: cluster-autoscaler
|
||||||
|
containers:
|
||||||
|
- image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.27.0
|
||||||
|
name: cluster-autoscaler
|
||||||
|
command:
|
||||||
|
- ./cluster-autoscaler
|
||||||
|
- --v=4
|
||||||
|
- --stderrthreshold=info
|
||||||
|
- --cloud-provider=clusterapi
|
||||||
|
- --namespace=kube-system
|
||||||
|
- --nodes=3:10:kind-worker
|
||||||
|
- --scale-down-delay-after-add=1m
|
||||||
|
- --scale-down-unneeded-time=1m
|
||||||
|
- --skip-nodes-with-local-storage=false
|
||||||
|
- --skip-nodes-with-system-pods=false
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ServiceAccount
|
||||||
|
metadata:
|
||||||
|
name: cluster-autoscaler
|
||||||
|
namespace: kube-system
|
||||||
|
---
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: ClusterRole
|
||||||
|
metadata:
|
||||||
|
name: cluster-autoscaler
|
||||||
|
rules:
|
||||||
|
- apiGroups: [""]
|
||||||
|
resources: ["events", "endpoints"]
|
||||||
|
verbs: ["create", "patch"]
|
||||||
|
- apiGroups: [""]
|
||||||
|
resources: ["pods/eviction"]
|
||||||
|
verbs: ["create"]
|
||||||
|
- apiGroups: [""]
|
||||||
|
resources: ["pods/status"]
|
||||||
|
verbs: ["update"]
|
||||||
|
- apiGroups: [""]
|
||||||
|
resources: ["nodes"]
|
||||||
|
verbs: ["watch", "list", "get", "update"]
|
||||||
|
---
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: ClusterRoleBinding
|
||||||
|
metadata:
|
||||||
|
name: cluster-autoscaler
|
||||||
|
roleRef:
|
||||||
|
apiGroup: rbac.authorization.k8s.io
|
||||||
|
kind: ClusterRole
|
||||||
|
name: cluster-autoscaler
|
||||||
|
subjects:
|
||||||
|
- kind: ServiceAccount
|
||||||
|
name: cluster-autoscaler
|
||||||
|
namespace: kube-system
|
||||||
157
k8s/news-api/README.md
Normal file
157
k8s/news-api/README.md
Normal file
@ -0,0 +1,157 @@
|
|||||||
|
# News API Kubernetes Deployment
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
Multi-language news articles REST API service for Kubernetes deployment.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
- **9 Language Support**: ko, en, zh_cn, zh_tw, ja, fr, de, es, it
|
||||||
|
- **REST API**: FastAPI with async MongoDB
|
||||||
|
- **Auto-scaling**: HPA based on CPU/Memory
|
||||||
|
- **Health Checks**: Liveness and readiness probes
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
### Option 1: Local Kubernetes
|
||||||
|
```bash
|
||||||
|
# Build Docker image
|
||||||
|
docker build -t site11/news-api:latest services/news-api/backend/
|
||||||
|
|
||||||
|
# Deploy to K8s
|
||||||
|
kubectl apply -f k8s/news-api/news-api-deployment.yaml
|
||||||
|
|
||||||
|
# Check status
|
||||||
|
kubectl -n site11-news get pods
|
||||||
|
```
|
||||||
|
|
||||||
|
### Option 2: Docker Hub
|
||||||
|
```bash
|
||||||
|
# Set Docker Hub user
|
||||||
|
export DOCKER_HUB_USER=your-username
|
||||||
|
|
||||||
|
# Build and push
|
||||||
|
docker build -t ${DOCKER_HUB_USER}/news-api:latest services/news-api/backend/
|
||||||
|
docker push ${DOCKER_HUB_USER}/news-api:latest
|
||||||
|
|
||||||
|
# Deploy
|
||||||
|
envsubst < k8s/news-api/news-api-dockerhub.yaml | kubectl apply -f -
|
||||||
|
```
|
||||||
|
|
||||||
|
### Option 3: Kind Cluster
|
||||||
|
```bash
|
||||||
|
# Build image
|
||||||
|
docker build -t site11/news-api:latest services/news-api/backend/
|
||||||
|
|
||||||
|
# Load to Kind
|
||||||
|
kind load docker-image site11/news-api:latest --name site11-cluster
|
||||||
|
|
||||||
|
# Deploy
|
||||||
|
kubectl apply -f k8s/news-api/news-api-deployment.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
### Get Articles List
|
||||||
|
```bash
|
||||||
|
GET /api/v1/{language}/articles?page=1&page_size=20&category=tech
|
||||||
|
```
|
||||||
|
|
||||||
|
### Get Latest Articles
|
||||||
|
```bash
|
||||||
|
GET /api/v1/{language}/articles/latest?limit=10
|
||||||
|
```
|
||||||
|
|
||||||
|
### Search Articles
|
||||||
|
```bash
|
||||||
|
GET /api/v1/{language}/articles/search?q=keyword&page=1
|
||||||
|
```
|
||||||
|
|
||||||
|
### Get Article by ID
|
||||||
|
```bash
|
||||||
|
GET /api/v1/{language}/articles/{article_id}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Get Categories
|
||||||
|
```bash
|
||||||
|
GET /api/v1/{language}/categories
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
### Port Forward
|
||||||
|
```bash
|
||||||
|
kubectl -n site11-news port-forward svc/news-api-service 8050:8000
|
||||||
|
```
|
||||||
|
|
||||||
|
### Test API
|
||||||
|
```bash
|
||||||
|
# Health check
|
||||||
|
curl http://localhost:8050/health
|
||||||
|
|
||||||
|
# Get Korean articles
|
||||||
|
curl http://localhost:8050/api/v1/ko/articles
|
||||||
|
|
||||||
|
# Get latest English articles
|
||||||
|
curl http://localhost:8050/api/v1/en/articles/latest?limit=5
|
||||||
|
|
||||||
|
# Search Japanese articles
|
||||||
|
curl "http://localhost:8050/api/v1/ja/articles/search?q=AI"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Monitoring
|
||||||
|
|
||||||
|
### View Pods
|
||||||
|
```bash
|
||||||
|
kubectl -n site11-news get pods -w
|
||||||
|
```
|
||||||
|
|
||||||
|
### View Logs
|
||||||
|
```bash
|
||||||
|
kubectl -n site11-news logs -f deployment/news-api
|
||||||
|
```
|
||||||
|
|
||||||
|
### Check HPA
|
||||||
|
```bash
|
||||||
|
kubectl -n site11-news get hpa
|
||||||
|
```
|
||||||
|
|
||||||
|
### Describe Service
|
||||||
|
```bash
|
||||||
|
kubectl -n site11-news describe svc news-api-service
|
||||||
|
```
|
||||||
|
|
||||||
|
## Scaling
|
||||||
|
|
||||||
|
### Manual Scaling
|
||||||
|
```bash
|
||||||
|
# Scale up
|
||||||
|
kubectl -n site11-news scale deployment news-api --replicas=5
|
||||||
|
|
||||||
|
# Scale down
|
||||||
|
kubectl -n site11-news scale deployment news-api --replicas=2
|
||||||
|
```
|
||||||
|
|
||||||
|
### Auto-scaling
|
||||||
|
HPA automatically scales between 2-10 replicas based on:
|
||||||
|
- CPU usage: 70% threshold
|
||||||
|
- Memory usage: 80% threshold
|
||||||
|
|
||||||
|
## Cleanup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Delete all resources
|
||||||
|
kubectl delete namespace site11-news
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Issue: ImagePullBackOff
|
||||||
|
**Solution**: Use Docker Hub deployment or load image to Kind
|
||||||
|
|
||||||
|
### Issue: MongoDB Connection Failed
|
||||||
|
**Solution**: Ensure MongoDB is running at `host.docker.internal:27017`
|
||||||
|
|
||||||
|
### Issue: No Articles Returned
|
||||||
|
**Solution**: Check if articles exist in MongoDB collections
|
||||||
|
|
||||||
|
### Issue: 404 on all endpoints
|
||||||
|
**Solution**: Verify correct namespace and service name in port-forward
|
||||||
113
k8s/news-api/news-api-deployment.yaml
Normal file
113
k8s/news-api/news-api-deployment.yaml
Normal file
@ -0,0 +1,113 @@
|
|||||||
|
apiVersion: v1
|
||||||
|
kind: Namespace
|
||||||
|
metadata:
|
||||||
|
name: site11-news
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: news-api-config
|
||||||
|
namespace: site11-news
|
||||||
|
data:
|
||||||
|
MONGODB_URL: "mongodb://host.docker.internal:27017"
|
||||||
|
DB_NAME: "ai_writer_db"
|
||||||
|
SERVICE_NAME: "news-api"
|
||||||
|
API_V1_STR: "/api/v1"
|
||||||
|
DEFAULT_PAGE_SIZE: "20"
|
||||||
|
MAX_PAGE_SIZE: "100"
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: news-api
|
||||||
|
namespace: site11-news
|
||||||
|
labels:
|
||||||
|
app: news-api
|
||||||
|
tier: backend
|
||||||
|
spec:
|
||||||
|
replicas: 3
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: news-api
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: news-api
|
||||||
|
tier: backend
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: news-api
|
||||||
|
image: site11/news-api:latest
|
||||||
|
imagePullPolicy: IfNotPresent
|
||||||
|
ports:
|
||||||
|
- containerPort: 8000
|
||||||
|
name: http
|
||||||
|
envFrom:
|
||||||
|
- configMapRef:
|
||||||
|
name: news-api-config
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 128Mi
|
||||||
|
limits:
|
||||||
|
cpu: 500m
|
||||||
|
memory: 512Mi
|
||||||
|
livenessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 8000
|
||||||
|
initialDelaySeconds: 10
|
||||||
|
periodSeconds: 30
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 8000
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 10
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: news-api-service
|
||||||
|
namespace: site11-news
|
||||||
|
labels:
|
||||||
|
app: news-api
|
||||||
|
spec:
|
||||||
|
type: ClusterIP
|
||||||
|
ports:
|
||||||
|
- port: 8000
|
||||||
|
targetPort: 8000
|
||||||
|
protocol: TCP
|
||||||
|
name: http
|
||||||
|
selector:
|
||||||
|
app: news-api
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: autoscaling/v2
|
||||||
|
kind: HorizontalPodAutoscaler
|
||||||
|
metadata:
|
||||||
|
name: news-api-hpa
|
||||||
|
namespace: site11-news
|
||||||
|
spec:
|
||||||
|
scaleTargetRef:
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
name: news-api
|
||||||
|
minReplicas: 3
|
||||||
|
maxReplicas: 10
|
||||||
|
metrics:
|
||||||
|
- type: Resource
|
||||||
|
resource:
|
||||||
|
name: cpu
|
||||||
|
target:
|
||||||
|
type: Utilization
|
||||||
|
averageUtilization: 70
|
||||||
|
- type: Resource
|
||||||
|
resource:
|
||||||
|
name: memory
|
||||||
|
target:
|
||||||
|
type: Utilization
|
||||||
|
averageUtilization: 80
|
||||||
113
k8s/news-api/news-api-dockerhub.yaml
Normal file
113
k8s/news-api/news-api-dockerhub.yaml
Normal file
@ -0,0 +1,113 @@
|
|||||||
|
apiVersion: v1
|
||||||
|
kind: Namespace
|
||||||
|
metadata:
|
||||||
|
name: site11-news
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: news-api-config
|
||||||
|
namespace: site11-news
|
||||||
|
data:
|
||||||
|
MONGODB_URL: "mongodb://host.docker.internal:27017"
|
||||||
|
DB_NAME: "ai_writer_db"
|
||||||
|
SERVICE_NAME: "news-api"
|
||||||
|
API_V1_STR: "/api/v1"
|
||||||
|
DEFAULT_PAGE_SIZE: "20"
|
||||||
|
MAX_PAGE_SIZE: "100"
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: news-api
|
||||||
|
namespace: site11-news
|
||||||
|
labels:
|
||||||
|
app: news-api
|
||||||
|
tier: backend
|
||||||
|
spec:
|
||||||
|
replicas: 3
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: news-api
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: news-api
|
||||||
|
tier: backend
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: news-api
|
||||||
|
image: ${DOCKER_HUB_USER}/news-api:latest
|
||||||
|
imagePullPolicy: Always
|
||||||
|
ports:
|
||||||
|
- containerPort: 8000
|
||||||
|
name: http
|
||||||
|
envFrom:
|
||||||
|
- configMapRef:
|
||||||
|
name: news-api-config
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 128Mi
|
||||||
|
limits:
|
||||||
|
cpu: 500m
|
||||||
|
memory: 512Mi
|
||||||
|
livenessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 8000
|
||||||
|
initialDelaySeconds: 10
|
||||||
|
periodSeconds: 30
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 8000
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 10
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: news-api-service
|
||||||
|
namespace: site11-news
|
||||||
|
labels:
|
||||||
|
app: news-api
|
||||||
|
spec:
|
||||||
|
type: ClusterIP
|
||||||
|
ports:
|
||||||
|
- port: 8000
|
||||||
|
targetPort: 8000
|
||||||
|
protocol: TCP
|
||||||
|
name: http
|
||||||
|
selector:
|
||||||
|
app: news-api
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: autoscaling/v2
|
||||||
|
kind: HorizontalPodAutoscaler
|
||||||
|
metadata:
|
||||||
|
name: news-api-hpa
|
||||||
|
namespace: site11-news
|
||||||
|
spec:
|
||||||
|
scaleTargetRef:
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
name: news-api
|
||||||
|
minReplicas: 3
|
||||||
|
maxReplicas: 10
|
||||||
|
metrics:
|
||||||
|
- type: Resource
|
||||||
|
resource:
|
||||||
|
name: cpu
|
||||||
|
target:
|
||||||
|
type: Utilization
|
||||||
|
averageUtilization: 70
|
||||||
|
- type: Resource
|
||||||
|
resource:
|
||||||
|
name: memory
|
||||||
|
target:
|
||||||
|
type: Utilization
|
||||||
|
averageUtilization: 80
|
||||||
@ -5,7 +5,6 @@ metadata:
|
|||||||
namespace: site11-pipeline
|
namespace: site11-pipeline
|
||||||
labels:
|
labels:
|
||||||
app: pipeline-ai-article-generator
|
app: pipeline-ai-article-generator
|
||||||
component: processor
|
|
||||||
spec:
|
spec:
|
||||||
replicas: 2
|
replicas: 2
|
||||||
selector:
|
selector:
|
||||||
@ -15,12 +14,11 @@ spec:
|
|||||||
metadata:
|
metadata:
|
||||||
labels:
|
labels:
|
||||||
app: pipeline-ai-article-generator
|
app: pipeline-ai-article-generator
|
||||||
component: processor
|
|
||||||
spec:
|
spec:
|
||||||
containers:
|
containers:
|
||||||
- name: ai-article-generator
|
- name: ai-article-generator
|
||||||
image: site11/pipeline-ai-article-generator:latest
|
image: yakenator/site11-pipeline-ai-article-generator:latest
|
||||||
imagePullPolicy: Always
|
imagePullPolicy: Always # Always pull from Docker Hub
|
||||||
envFrom:
|
envFrom:
|
||||||
- configMapRef:
|
- configMapRef:
|
||||||
name: pipeline-config
|
name: pipeline-config
|
||||||
@ -28,28 +26,27 @@ spec:
|
|||||||
name: pipeline-secrets
|
name: pipeline-secrets
|
||||||
resources:
|
resources:
|
||||||
requests:
|
requests:
|
||||||
memory: "512Mi"
|
memory: "256Mi"
|
||||||
cpu: "200m"
|
cpu: "100m"
|
||||||
limits:
|
limits:
|
||||||
memory: "1Gi"
|
memory: "512Mi"
|
||||||
cpu: "1000m"
|
cpu: "500m"
|
||||||
livenessProbe:
|
|
||||||
exec:
|
|
||||||
command:
|
|
||||||
- python
|
|
||||||
- -c
|
|
||||||
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
|
|
||||||
initialDelaySeconds: 30
|
|
||||||
periodSeconds: 30
|
|
||||||
readinessProbe:
|
readinessProbe:
|
||||||
exec:
|
exec:
|
||||||
command:
|
command:
|
||||||
- python
|
- python
|
||||||
- -c
|
- -c
|
||||||
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
|
- "import sys; sys.exit(0)"
|
||||||
initialDelaySeconds: 10
|
initialDelaySeconds: 10
|
||||||
|
periodSeconds: 5
|
||||||
|
livenessProbe:
|
||||||
|
exec:
|
||||||
|
command:
|
||||||
|
- python
|
||||||
|
- -c
|
||||||
|
- "import sys; sys.exit(0)"
|
||||||
|
initialDelaySeconds: 30
|
||||||
periodSeconds: 10
|
periodSeconds: 10
|
||||||
|
|
||||||
---
|
---
|
||||||
apiVersion: autoscaling/v2
|
apiVersion: autoscaling/v2
|
||||||
kind: HorizontalPodAutoscaler
|
kind: HorizontalPodAutoscaler
|
||||||
@ -61,8 +58,8 @@ spec:
|
|||||||
apiVersion: apps/v1
|
apiVersion: apps/v1
|
||||||
kind: Deployment
|
kind: Deployment
|
||||||
name: pipeline-ai-article-generator
|
name: pipeline-ai-article-generator
|
||||||
minReplicas: 1
|
minReplicas: 2
|
||||||
maxReplicas: 8
|
maxReplicas: 10
|
||||||
metrics:
|
metrics:
|
||||||
- type: Resource
|
- type: Resource
|
||||||
resource:
|
resource:
|
||||||
@ -75,4 +72,4 @@ spec:
|
|||||||
name: memory
|
name: memory
|
||||||
target:
|
target:
|
||||||
type: Utilization
|
type: Utilization
|
||||||
averageUtilization: 80
|
averageUtilization: 80
|
||||||
37
k8s/pipeline/configmap-dockerhub.yaml
Normal file
37
k8s/pipeline/configmap-dockerhub.yaml
Normal file
@ -0,0 +1,37 @@
|
|||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: pipeline-config
|
||||||
|
namespace: site11-pipeline
|
||||||
|
data:
|
||||||
|
# External Redis - AWS ElastiCache simulation
|
||||||
|
REDIS_URL: "redis://host.docker.internal:6379"
|
||||||
|
|
||||||
|
# External MongoDB - AWS DocumentDB simulation
|
||||||
|
MONGODB_URL: "mongodb://host.docker.internal:27017"
|
||||||
|
DB_NAME: "ai_writer_db"
|
||||||
|
|
||||||
|
# Logging
|
||||||
|
LOG_LEVEL: "INFO"
|
||||||
|
|
||||||
|
# Worker settings
|
||||||
|
WORKER_COUNT: "2"
|
||||||
|
BATCH_SIZE: "10"
|
||||||
|
|
||||||
|
# Queue delays
|
||||||
|
RSS_ENQUEUE_DELAY: "1.0"
|
||||||
|
GOOGLE_SEARCH_DELAY: "2.0"
|
||||||
|
TRANSLATION_DELAY: "1.0"
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: pipeline-secrets
|
||||||
|
namespace: site11-pipeline
|
||||||
|
type: Opaque
|
||||||
|
stringData:
|
||||||
|
DEEPL_API_KEY: "3abbc796-2515-44a8-972d-22dcf27ab54a"
|
||||||
|
CLAUDE_API_KEY: "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"
|
||||||
|
OPENAI_API_KEY: "sk-openai-api-key-here" # Replace with actual key
|
||||||
|
SERP_API_KEY: "serp-api-key-here" # Replace with actual key
|
||||||
94
k8s/pipeline/console-backend-dockerhub.yaml
Normal file
94
k8s/pipeline/console-backend-dockerhub.yaml
Normal file
@ -0,0 +1,94 @@
|
|||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: console-backend
|
||||||
|
namespace: site11-pipeline
|
||||||
|
labels:
|
||||||
|
app: console-backend
|
||||||
|
spec:
|
||||||
|
replicas: 2
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: console-backend
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: console-backend
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: console-backend
|
||||||
|
image: yakenator/site11-console-backend:latest
|
||||||
|
imagePullPolicy: Always
|
||||||
|
ports:
|
||||||
|
- containerPort: 8000
|
||||||
|
protocol: TCP
|
||||||
|
env:
|
||||||
|
- name: ENV
|
||||||
|
value: "production"
|
||||||
|
- name: MONGODB_URL
|
||||||
|
value: "mongodb://host.docker.internal:27017"
|
||||||
|
- name: REDIS_URL
|
||||||
|
value: "redis://host.docker.internal:6379"
|
||||||
|
- name: USERS_SERVICE_URL
|
||||||
|
value: "http://users-backend:8000"
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
memory: "256Mi"
|
||||||
|
cpu: "100m"
|
||||||
|
limits:
|
||||||
|
memory: "512Mi"
|
||||||
|
cpu: "500m"
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 8000
|
||||||
|
initialDelaySeconds: 10
|
||||||
|
periodSeconds: 5
|
||||||
|
livenessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /health
|
||||||
|
port: 8000
|
||||||
|
initialDelaySeconds: 30
|
||||||
|
periodSeconds: 10
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: console-backend
|
||||||
|
namespace: site11-pipeline
|
||||||
|
labels:
|
||||||
|
app: console-backend
|
||||||
|
spec:
|
||||||
|
type: ClusterIP
|
||||||
|
selector:
|
||||||
|
app: console-backend
|
||||||
|
ports:
|
||||||
|
- port: 8000
|
||||||
|
targetPort: 8000
|
||||||
|
protocol: TCP
|
||||||
|
---
|
||||||
|
apiVersion: autoscaling/v2
|
||||||
|
kind: HorizontalPodAutoscaler
|
||||||
|
metadata:
|
||||||
|
name: console-backend-hpa
|
||||||
|
namespace: site11-pipeline
|
||||||
|
spec:
|
||||||
|
scaleTargetRef:
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
name: console-backend
|
||||||
|
minReplicas: 2
|
||||||
|
maxReplicas: 10
|
||||||
|
metrics:
|
||||||
|
- type: Resource
|
||||||
|
resource:
|
||||||
|
name: cpu
|
||||||
|
target:
|
||||||
|
type: Utilization
|
||||||
|
averageUtilization: 70
|
||||||
|
- type: Resource
|
||||||
|
resource:
|
||||||
|
name: memory
|
||||||
|
target:
|
||||||
|
type: Utilization
|
||||||
|
averageUtilization: 80
|
||||||
89
k8s/pipeline/console-frontend-dockerhub.yaml
Normal file
89
k8s/pipeline/console-frontend-dockerhub.yaml
Normal file
@ -0,0 +1,89 @@
|
|||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: console-frontend
|
||||||
|
namespace: site11-pipeline
|
||||||
|
labels:
|
||||||
|
app: console-frontend
|
||||||
|
spec:
|
||||||
|
replicas: 2
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: console-frontend
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: console-frontend
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: console-frontend
|
||||||
|
image: yakenator/site11-console-frontend:latest
|
||||||
|
imagePullPolicy: Always
|
||||||
|
ports:
|
||||||
|
- containerPort: 80
|
||||||
|
protocol: TCP
|
||||||
|
env:
|
||||||
|
- name: VITE_API_URL
|
||||||
|
value: "http://console-backend:8000"
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
memory: "128Mi"
|
||||||
|
cpu: "50m"
|
||||||
|
limits:
|
||||||
|
memory: "256Mi"
|
||||||
|
cpu: "200m"
|
||||||
|
readinessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /
|
||||||
|
port: 80
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 5
|
||||||
|
livenessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /
|
||||||
|
port: 80
|
||||||
|
initialDelaySeconds: 15
|
||||||
|
periodSeconds: 10
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: console-frontend
|
||||||
|
namespace: site11-pipeline
|
||||||
|
labels:
|
||||||
|
app: console-frontend
|
||||||
|
spec:
|
||||||
|
type: LoadBalancer
|
||||||
|
selector:
|
||||||
|
app: console-frontend
|
||||||
|
ports:
|
||||||
|
- port: 3000
|
||||||
|
targetPort: 80
|
||||||
|
protocol: TCP
|
||||||
|
name: http
|
||||||
|
---
|
||||||
|
apiVersion: autoscaling/v2
|
||||||
|
kind: HorizontalPodAutoscaler
|
||||||
|
metadata:
|
||||||
|
name: console-frontend-hpa
|
||||||
|
namespace: site11-pipeline
|
||||||
|
spec:
|
||||||
|
scaleTargetRef:
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
name: console-frontend
|
||||||
|
minReplicas: 2
|
||||||
|
maxReplicas: 10
|
||||||
|
metrics:
|
||||||
|
- type: Resource
|
||||||
|
resource:
|
||||||
|
name: cpu
|
||||||
|
target:
|
||||||
|
type: Utilization
|
||||||
|
averageUtilization: 70
|
||||||
|
- type: Resource
|
||||||
|
resource:
|
||||||
|
name: memory
|
||||||
|
target:
|
||||||
|
type: Utilization
|
||||||
|
averageUtilization: 80
|
||||||
226
k8s/pipeline/deploy-docker-desktop.sh
Executable file
226
k8s/pipeline/deploy-docker-desktop.sh
Executable file
@ -0,0 +1,226 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# Site11 Pipeline K8s Docker Desktop Deployment Script
|
||||||
|
# =====================================================
|
||||||
|
# Deploys pipeline workers to Docker Desktop K8s with external infrastructure
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
echo "🚀 Site11 Pipeline K8s Docker Desktop Deployment"
|
||||||
|
echo "================================================"
|
||||||
|
echo ""
|
||||||
|
echo "Architecture:"
|
||||||
|
echo " - Infrastructure: External (Docker Compose)"
|
||||||
|
echo " - Workers: K8s (Docker Desktop)"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
# Check prerequisites
|
||||||
|
echo -e "${BLUE}Checking prerequisites...${NC}"
|
||||||
|
|
||||||
|
# Check if kubectl is available
|
||||||
|
if ! command -v kubectl &> /dev/null; then
|
||||||
|
echo -e "${RED}❌ kubectl is not installed${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check K8s cluster connection
|
||||||
|
echo -n " K8s cluster connection... "
|
||||||
|
if kubectl cluster-info &> /dev/null; then
|
||||||
|
echo -e "${GREEN}✓${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${RED}✗ Cannot connect to K8s cluster${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check if Docker infrastructure is running
|
||||||
|
echo -n " Docker infrastructure services... "
|
||||||
|
if docker ps | grep -q "site11_mongodb" && docker ps | grep -q "site11_redis"; then
|
||||||
|
echo -e "${GREEN}✓${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${YELLOW}⚠️ Infrastructure not running. Start with: docker-compose -f docker-compose-hybrid.yml up -d${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Step 1: Create namespace
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}1. Creating K8s namespace...${NC}"
|
||||||
|
kubectl apply -f namespace.yaml
|
||||||
|
|
||||||
|
# Step 2: Create ConfigMap and Secrets for external services
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}2. Configuring external service connections...${NC}"
|
||||||
|
cat > configmap-docker-desktop.yaml << 'EOF'
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: pipeline-config
|
||||||
|
namespace: site11-pipeline
|
||||||
|
data:
|
||||||
|
# External Redis (Docker host)
|
||||||
|
REDIS_URL: "redis://host.docker.internal:6379"
|
||||||
|
|
||||||
|
# External MongoDB (Docker host)
|
||||||
|
MONGODB_URL: "mongodb://host.docker.internal:27017"
|
||||||
|
DB_NAME: "ai_writer_db"
|
||||||
|
|
||||||
|
# Logging
|
||||||
|
LOG_LEVEL: "INFO"
|
||||||
|
|
||||||
|
# Worker settings
|
||||||
|
WORKER_COUNT: "2"
|
||||||
|
BATCH_SIZE: "10"
|
||||||
|
|
||||||
|
# Queue delays
|
||||||
|
RSS_ENQUEUE_DELAY: "1.0"
|
||||||
|
GOOGLE_SEARCH_DELAY: "2.0"
|
||||||
|
TRANSLATION_DELAY: "1.0"
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: pipeline-secrets
|
||||||
|
namespace: site11-pipeline
|
||||||
|
type: Opaque
|
||||||
|
stringData:
|
||||||
|
DEEPL_API_KEY: "3abbc796-2515-44a8-972d-22dcf27ab54a"
|
||||||
|
CLAUDE_API_KEY: "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"
|
||||||
|
OPENAI_API_KEY: "sk-openai-api-key-here" # Replace with actual key
|
||||||
|
SERP_API_KEY: "serp-api-key-here" # Replace with actual key
|
||||||
|
EOF
|
||||||
|
|
||||||
|
kubectl apply -f configmap-docker-desktop.yaml
|
||||||
|
|
||||||
|
# Step 3: Update deployment YAMLs to use Docker images directly
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}3. Creating deployments for Docker Desktop...${NC}"
|
||||||
|
services=("rss-collector" "google-search" "translator" "ai-article-generator" "image-generator")
|
||||||
|
|
||||||
|
for service in "${services[@]}"; do
|
||||||
|
cat > ${service}-docker-desktop.yaml << EOF
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: pipeline-$service
|
||||||
|
namespace: site11-pipeline
|
||||||
|
labels:
|
||||||
|
app: pipeline-$service
|
||||||
|
spec:
|
||||||
|
replicas: $([ "$service" = "translator" ] && echo "3" || echo "2")
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: pipeline-$service
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: pipeline-$service
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: $service
|
||||||
|
image: site11-pipeline-$service:latest
|
||||||
|
imagePullPolicy: Never # Use local Docker image
|
||||||
|
envFrom:
|
||||||
|
- configMapRef:
|
||||||
|
name: pipeline-config
|
||||||
|
- secretRef:
|
||||||
|
name: pipeline-secrets
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
memory: "256Mi"
|
||||||
|
cpu: "100m"
|
||||||
|
limits:
|
||||||
|
memory: "512Mi"
|
||||||
|
cpu: "500m"
|
||||||
|
readinessProbe:
|
||||||
|
exec:
|
||||||
|
command:
|
||||||
|
- python
|
||||||
|
- -c
|
||||||
|
- "import sys; sys.exit(0)"
|
||||||
|
initialDelaySeconds: 10
|
||||||
|
periodSeconds: 5
|
||||||
|
livenessProbe:
|
||||||
|
exec:
|
||||||
|
command:
|
||||||
|
- python
|
||||||
|
- -c
|
||||||
|
- "import sys; sys.exit(0)"
|
||||||
|
initialDelaySeconds: 30
|
||||||
|
periodSeconds: 10
|
||||||
|
---
|
||||||
|
apiVersion: autoscaling/v2
|
||||||
|
kind: HorizontalPodAutoscaler
|
||||||
|
metadata:
|
||||||
|
name: pipeline-$service-hpa
|
||||||
|
namespace: site11-pipeline
|
||||||
|
spec:
|
||||||
|
scaleTargetRef:
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
name: pipeline-$service
|
||||||
|
minReplicas: $([ "$service" = "translator" ] && echo "3" || echo "2")
|
||||||
|
maxReplicas: 10
|
||||||
|
metrics:
|
||||||
|
- type: Resource
|
||||||
|
resource:
|
||||||
|
name: cpu
|
||||||
|
target:
|
||||||
|
type: Utilization
|
||||||
|
averageUtilization: 70
|
||||||
|
- type: Resource
|
||||||
|
resource:
|
||||||
|
name: memory
|
||||||
|
target:
|
||||||
|
type: Utilization
|
||||||
|
averageUtilization: 80
|
||||||
|
EOF
|
||||||
|
done
|
||||||
|
|
||||||
|
# Step 4: Deploy services to K8s
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}4. Deploying workers to K8s...${NC}"
|
||||||
|
|
||||||
|
for service in "${services[@]}"; do
|
||||||
|
echo -n " Deploying $service... "
|
||||||
|
kubectl apply -f ${service}-docker-desktop.yaml && echo -e "${GREEN}✓${NC}"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Step 5: Check deployment status
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}5. Verifying deployments...${NC}"
|
||||||
|
kubectl -n site11-pipeline get deployments
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}6. Waiting for pods to be ready...${NC}"
|
||||||
|
kubectl -n site11-pipeline wait --for=condition=Ready pods --all --timeout=60s 2>/dev/null || {
|
||||||
|
echo -e "${YELLOW}⚠️ Some pods are still initializing...${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Step 6: Show final status
|
||||||
|
echo ""
|
||||||
|
echo -e "${GREEN}✅ Deployment Complete!${NC}"
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}Current pod status:${NC}"
|
||||||
|
kubectl -n site11-pipeline get pods
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}External infrastructure status:${NC}"
|
||||||
|
docker ps --format "table {{.Names}}\t{{.Status}}" | grep -E "site11_(mongodb|redis|kafka|zookeeper)" || echo "No infrastructure services found"
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}Useful commands:${NC}"
|
||||||
|
echo " View logs: kubectl -n site11-pipeline logs -f deployment/pipeline-translator"
|
||||||
|
echo " Scale workers: kubectl -n site11-pipeline scale deployment pipeline-translator --replicas=5"
|
||||||
|
echo " Check HPA: kubectl -n site11-pipeline get hpa"
|
||||||
|
echo " Monitor queues: docker-compose -f docker-compose-hybrid.yml logs -f pipeline-monitor"
|
||||||
|
echo " Delete K8s: kubectl delete namespace site11-pipeline"
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}Architecture Overview:${NC}"
|
||||||
|
echo " 📦 Infrastructure (Docker): MongoDB, Redis, Kafka"
|
||||||
|
echo " ☸️ Workers (K8s): RSS, Search, Translation, AI Generation, Image Generation"
|
||||||
|
echo " 🎛️ Control (Docker): Scheduler, Monitor, Language Sync"
|
||||||
246
k8s/pipeline/deploy-dockerhub.sh
Executable file
246
k8s/pipeline/deploy-dockerhub.sh
Executable file
@ -0,0 +1,246 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# Site11 Pipeline Docker Hub Deployment Script
|
||||||
|
# =============================================
|
||||||
|
# Push images to Docker Hub and deploy to K8s
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
echo "🚀 Site11 Pipeline Docker Hub Deployment"
|
||||||
|
echo "========================================"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
DOCKER_HUB_USER="${DOCKER_HUB_USER:-your-dockerhub-username}" # Set your Docker Hub username
|
||||||
|
IMAGE_TAG="${IMAGE_TAG:-latest}"
|
||||||
|
|
||||||
|
if [ "$DOCKER_HUB_USER" = "your-dockerhub-username" ]; then
|
||||||
|
echo -e "${RED}❌ Please set DOCKER_HUB_USER environment variable${NC}"
|
||||||
|
echo "Example: export DOCKER_HUB_USER=myusername"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check prerequisites
|
||||||
|
echo -e "${BLUE}Checking prerequisites...${NC}"
|
||||||
|
|
||||||
|
# Check if docker is logged in
|
||||||
|
echo -n " Docker Hub login... "
|
||||||
|
if docker info 2>/dev/null | grep -q "Username: $DOCKER_HUB_USER"; then
|
||||||
|
echo -e "${GREEN}✓${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${YELLOW}Please login${NC}"
|
||||||
|
docker login
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check if kubectl is available
|
||||||
|
if ! command -v kubectl &> /dev/null; then
|
||||||
|
echo -e "${RED}❌ kubectl is not installed${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check K8s cluster connection
|
||||||
|
echo -n " K8s cluster connection... "
|
||||||
|
if kubectl cluster-info &> /dev/null; then
|
||||||
|
echo -e "${GREEN}✓${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${RED}✗ Cannot connect to K8s cluster${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Services to deploy
|
||||||
|
services=("rss-collector" "google-search" "translator" "ai-article-generator" "image-generator")
|
||||||
|
|
||||||
|
# Step 1: Tag and push images to Docker Hub
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}1. Pushing images to Docker Hub...${NC}"
|
||||||
|
|
||||||
|
for service in "${services[@]}"; do
|
||||||
|
echo -n " Pushing pipeline-$service... "
|
||||||
|
docker tag site11-pipeline-$service:latest $DOCKER_HUB_USER/site11-pipeline-$service:$IMAGE_TAG
|
||||||
|
docker push $DOCKER_HUB_USER/site11-pipeline-$service:$IMAGE_TAG && echo -e "${GREEN}✓${NC}"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Step 2: Create namespace
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}2. Creating K8s namespace...${NC}"
|
||||||
|
kubectl apply -f namespace.yaml
|
||||||
|
|
||||||
|
# Step 3: Create ConfigMap and Secrets
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}3. Configuring external service connections...${NC}"
|
||||||
|
cat > configmap-dockerhub.yaml << 'EOF'
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: pipeline-config
|
||||||
|
namespace: site11-pipeline
|
||||||
|
data:
|
||||||
|
# External Redis - AWS ElastiCache simulation
|
||||||
|
REDIS_URL: "redis://host.docker.internal:6379"
|
||||||
|
|
||||||
|
# External MongoDB - AWS DocumentDB simulation
|
||||||
|
MONGODB_URL: "mongodb://host.docker.internal:27017"
|
||||||
|
DB_NAME: "ai_writer_db"
|
||||||
|
|
||||||
|
# Logging
|
||||||
|
LOG_LEVEL: "INFO"
|
||||||
|
|
||||||
|
# Worker settings
|
||||||
|
WORKER_COUNT: "2"
|
||||||
|
BATCH_SIZE: "10"
|
||||||
|
|
||||||
|
# Queue delays
|
||||||
|
RSS_ENQUEUE_DELAY: "1.0"
|
||||||
|
GOOGLE_SEARCH_DELAY: "2.0"
|
||||||
|
TRANSLATION_DELAY: "1.0"
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: pipeline-secrets
|
||||||
|
namespace: site11-pipeline
|
||||||
|
type: Opaque
|
||||||
|
stringData:
|
||||||
|
DEEPL_API_KEY: "3abbc796-2515-44a8-972d-22dcf27ab54a"
|
||||||
|
CLAUDE_API_KEY: "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"
|
||||||
|
OPENAI_API_KEY: "sk-openai-api-key-here" # Replace with actual key
|
||||||
|
SERP_API_KEY: "serp-api-key-here" # Replace with actual key
|
||||||
|
EOF
|
||||||
|
|
||||||
|
kubectl apply -f configmap-dockerhub.yaml
|
||||||
|
|
||||||
|
# Step 4: Create deployments using Docker Hub images
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}4. Creating K8s deployments...${NC}"
|
||||||
|
|
||||||
|
for service in "${services[@]}"; do
|
||||||
|
cat > ${service}-dockerhub.yaml << EOF
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: pipeline-$service
|
||||||
|
namespace: site11-pipeline
|
||||||
|
labels:
|
||||||
|
app: pipeline-$service
|
||||||
|
spec:
|
||||||
|
replicas: $([ "$service" = "translator" ] && echo "3" || echo "2")
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: pipeline-$service
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: pipeline-$service
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: $service
|
||||||
|
image: $DOCKER_HUB_USER/site11-pipeline-$service:$IMAGE_TAG
|
||||||
|
imagePullPolicy: Always # Always pull from Docker Hub
|
||||||
|
envFrom:
|
||||||
|
- configMapRef:
|
||||||
|
name: pipeline-config
|
||||||
|
- secretRef:
|
||||||
|
name: pipeline-secrets
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
memory: "256Mi"
|
||||||
|
cpu: "100m"
|
||||||
|
limits:
|
||||||
|
memory: "512Mi"
|
||||||
|
cpu: "500m"
|
||||||
|
readinessProbe:
|
||||||
|
exec:
|
||||||
|
command:
|
||||||
|
- python
|
||||||
|
- -c
|
||||||
|
- "import sys; sys.exit(0)"
|
||||||
|
initialDelaySeconds: 10
|
||||||
|
periodSeconds: 5
|
||||||
|
livenessProbe:
|
||||||
|
exec:
|
||||||
|
command:
|
||||||
|
- python
|
||||||
|
- -c
|
||||||
|
- "import sys; sys.exit(0)"
|
||||||
|
initialDelaySeconds: 30
|
||||||
|
periodSeconds: 10
|
||||||
|
---
|
||||||
|
apiVersion: autoscaling/v2
|
||||||
|
kind: HorizontalPodAutoscaler
|
||||||
|
metadata:
|
||||||
|
name: pipeline-$service-hpa
|
||||||
|
namespace: site11-pipeline
|
||||||
|
spec:
|
||||||
|
scaleTargetRef:
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
name: pipeline-$service
|
||||||
|
minReplicas: $([ "$service" = "translator" ] && echo "3" || echo "2")
|
||||||
|
maxReplicas: 10
|
||||||
|
metrics:
|
||||||
|
- type: Resource
|
||||||
|
resource:
|
||||||
|
name: cpu
|
||||||
|
target:
|
||||||
|
type: Utilization
|
||||||
|
averageUtilization: 70
|
||||||
|
- type: Resource
|
||||||
|
resource:
|
||||||
|
name: memory
|
||||||
|
target:
|
||||||
|
type: Utilization
|
||||||
|
averageUtilization: 80
|
||||||
|
EOF
|
||||||
|
done
|
||||||
|
|
||||||
|
# Step 5: Deploy services to K8s
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}5. Deploying workers to K8s...${NC}"
|
||||||
|
|
||||||
|
for service in "${services[@]}"; do
|
||||||
|
echo -n " Deploying $service... "
|
||||||
|
kubectl apply -f ${service}-dockerhub.yaml && echo -e "${GREEN}✓${NC}"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Step 6: Wait for deployments
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}6. Waiting for pods to be ready...${NC}"
|
||||||
|
kubectl -n site11-pipeline wait --for=condition=Ready pods --all --timeout=180s 2>/dev/null || {
|
||||||
|
echo -e "${YELLOW}⚠️ Some pods are still initializing...${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Step 7: Show status
|
||||||
|
echo ""
|
||||||
|
echo -e "${GREEN}✅ Deployment Complete!${NC}"
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}Deployment status:${NC}"
|
||||||
|
kubectl -n site11-pipeline get deployments
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}Pod status:${NC}"
|
||||||
|
kubectl -n site11-pipeline get pods
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}Images deployed:${NC}"
|
||||||
|
for service in "${services[@]}"; do
|
||||||
|
echo " $DOCKER_HUB_USER/site11-pipeline-$service:$IMAGE_TAG"
|
||||||
|
done
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}Useful commands:${NC}"
|
||||||
|
echo " View logs: kubectl -n site11-pipeline logs -f deployment/pipeline-translator"
|
||||||
|
echo " Scale: kubectl -n site11-pipeline scale deployment pipeline-translator --replicas=5"
|
||||||
|
echo " Check HPA: kubectl -n site11-pipeline get hpa"
|
||||||
|
echo " Update image: kubectl -n site11-pipeline set image deployment/pipeline-translator translator=$DOCKER_HUB_USER/site11-pipeline-translator:new-tag"
|
||||||
|
echo " Delete: kubectl delete namespace site11-pipeline"
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}Architecture:${NC}"
|
||||||
|
echo " 🌐 Images: Docker Hub ($DOCKER_HUB_USER/*)"
|
||||||
|
echo " 📦 Infrastructure: External (Docker Compose)"
|
||||||
|
echo " ☸️ Workers: K8s cluster"
|
||||||
|
echo " 🎛️ Control: Docker Compose (Scheduler, Monitor)"
|
||||||
240
k8s/pipeline/deploy-kind.sh
Executable file
240
k8s/pipeline/deploy-kind.sh
Executable file
@ -0,0 +1,240 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# Site11 Pipeline Kind Deployment Script
|
||||||
|
# =======================================
|
||||||
|
# Deploys pipeline workers to Kind cluster with external infrastructure
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
echo "🚀 Site11 Pipeline Kind Deployment"
|
||||||
|
echo "==================================="
|
||||||
|
echo ""
|
||||||
|
echo "This deployment uses:"
|
||||||
|
echo " - Infrastructure: External (Docker Compose)"
|
||||||
|
echo " - Workers: Kind K8s cluster"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
# Check prerequisites
|
||||||
|
echo -e "${BLUE}Checking prerequisites...${NC}"
|
||||||
|
|
||||||
|
# Check if kind is available
|
||||||
|
if ! command -v kind &> /dev/null; then
|
||||||
|
echo -e "${RED}❌ kind is not installed${NC}"
|
||||||
|
echo "Install with: brew install kind"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check if Docker infrastructure is running
|
||||||
|
echo -n " Docker infrastructure services... "
|
||||||
|
if docker ps | grep -q "site11_mongodb" && docker ps | grep -q "site11_redis"; then
|
||||||
|
echo -e "${GREEN}✓${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${YELLOW}⚠️ Infrastructure not running. Start with: docker-compose -f docker-compose-hybrid.yml up -d${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Step 1: Create or use existing Kind cluster
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}1. Setting up Kind cluster...${NC}"
|
||||||
|
if kind get clusters | grep -q "site11-cluster"; then
|
||||||
|
echo " Using existing site11-cluster"
|
||||||
|
kubectl config use-context kind-site11-cluster
|
||||||
|
else
|
||||||
|
echo " Creating new Kind cluster..."
|
||||||
|
kind create cluster --config kind-config.yaml
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Step 2: Load Docker images to Kind
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}2. Loading Docker images to Kind cluster...${NC}"
|
||||||
|
services=("rss-collector" "google-search" "translator" "ai-article-generator" "image-generator")
|
||||||
|
|
||||||
|
for service in "${services[@]}"; do
|
||||||
|
echo -n " Loading pipeline-$service... "
|
||||||
|
kind load docker-image site11-pipeline-$service:latest --name site11-cluster && echo -e "${GREEN}✓${NC}"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Step 3: Create namespace
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}3. Creating K8s namespace...${NC}"
|
||||||
|
kubectl apply -f namespace.yaml
|
||||||
|
|
||||||
|
# Step 4: Create ConfigMap and Secrets for external services
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}4. Configuring external service connections...${NC}"
|
||||||
|
cat > configmap-kind.yaml << 'EOF'
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: pipeline-config
|
||||||
|
namespace: site11-pipeline
|
||||||
|
data:
|
||||||
|
# External Redis (host network) - Docker services
|
||||||
|
REDIS_URL: "redis://host.docker.internal:6379"
|
||||||
|
|
||||||
|
# External MongoDB (host network) - Docker services
|
||||||
|
MONGODB_URL: "mongodb://host.docker.internal:27017"
|
||||||
|
DB_NAME: "ai_writer_db"
|
||||||
|
|
||||||
|
# Logging
|
||||||
|
LOG_LEVEL: "INFO"
|
||||||
|
|
||||||
|
# Worker settings
|
||||||
|
WORKER_COUNT: "2"
|
||||||
|
BATCH_SIZE: "10"
|
||||||
|
|
||||||
|
# Queue delays
|
||||||
|
RSS_ENQUEUE_DELAY: "1.0"
|
||||||
|
GOOGLE_SEARCH_DELAY: "2.0"
|
||||||
|
TRANSLATION_DELAY: "1.0"
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: pipeline-secrets
|
||||||
|
namespace: site11-pipeline
|
||||||
|
type: Opaque
|
||||||
|
stringData:
|
||||||
|
DEEPL_API_KEY: "3abbc796-2515-44a8-972d-22dcf27ab54a"
|
||||||
|
CLAUDE_API_KEY: "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"
|
||||||
|
OPENAI_API_KEY: "sk-openai-api-key-here" # Replace with actual key
|
||||||
|
SERP_API_KEY: "serp-api-key-here" # Replace with actual key
|
||||||
|
EOF
|
||||||
|
|
||||||
|
kubectl apply -f configmap-kind.yaml
|
||||||
|
|
||||||
|
# Step 5: Create deployments for Kind
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}5. Creating deployments for Kind...${NC}"
|
||||||
|
|
||||||
|
for service in "${services[@]}"; do
|
||||||
|
cat > ${service}-kind.yaml << EOF
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: pipeline-$service
|
||||||
|
namespace: site11-pipeline
|
||||||
|
labels:
|
||||||
|
app: pipeline-$service
|
||||||
|
spec:
|
||||||
|
replicas: $([ "$service" = "translator" ] && echo "3" || echo "2")
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: pipeline-$service
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: pipeline-$service
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: $service
|
||||||
|
image: site11-pipeline-$service:latest
|
||||||
|
imagePullPolicy: Never # Use loaded image
|
||||||
|
envFrom:
|
||||||
|
- configMapRef:
|
||||||
|
name: pipeline-config
|
||||||
|
- secretRef:
|
||||||
|
name: pipeline-secrets
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
memory: "256Mi"
|
||||||
|
cpu: "100m"
|
||||||
|
limits:
|
||||||
|
memory: "512Mi"
|
||||||
|
cpu: "500m"
|
||||||
|
readinessProbe:
|
||||||
|
exec:
|
||||||
|
command:
|
||||||
|
- python
|
||||||
|
- -c
|
||||||
|
- "import sys; sys.exit(0)"
|
||||||
|
initialDelaySeconds: 10
|
||||||
|
periodSeconds: 5
|
||||||
|
livenessProbe:
|
||||||
|
exec:
|
||||||
|
command:
|
||||||
|
- python
|
||||||
|
- -c
|
||||||
|
- "import sys; sys.exit(0)"
|
||||||
|
initialDelaySeconds: 30
|
||||||
|
periodSeconds: 10
|
||||||
|
---
|
||||||
|
apiVersion: autoscaling/v2
|
||||||
|
kind: HorizontalPodAutoscaler
|
||||||
|
metadata:
|
||||||
|
name: pipeline-$service-hpa
|
||||||
|
namespace: site11-pipeline
|
||||||
|
spec:
|
||||||
|
scaleTargetRef:
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
name: pipeline-$service
|
||||||
|
minReplicas: $([ "$service" = "translator" ] && echo "3" || echo "2")
|
||||||
|
maxReplicas: 10
|
||||||
|
metrics:
|
||||||
|
- type: Resource
|
||||||
|
resource:
|
||||||
|
name: cpu
|
||||||
|
target:
|
||||||
|
type: Utilization
|
||||||
|
averageUtilization: 70
|
||||||
|
- type: Resource
|
||||||
|
resource:
|
||||||
|
name: memory
|
||||||
|
target:
|
||||||
|
type: Utilization
|
||||||
|
averageUtilization: 80
|
||||||
|
EOF
|
||||||
|
done
|
||||||
|
|
||||||
|
# Step 6: Deploy services to K8s
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}6. Deploying workers to Kind cluster...${NC}"
|
||||||
|
|
||||||
|
for service in "${services[@]}"; do
|
||||||
|
echo -n " Deploying $service... "
|
||||||
|
kubectl apply -f ${service}-kind.yaml && echo -e "${GREEN}✓${NC}"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Step 7: Check deployment status
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}7. Verifying deployments...${NC}"
|
||||||
|
kubectl -n site11-pipeline get deployments
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}8. Waiting for pods to be ready...${NC}"
|
||||||
|
kubectl -n site11-pipeline wait --for=condition=Ready pods --all --timeout=120s 2>/dev/null || {
|
||||||
|
echo -e "${YELLOW}⚠️ Some pods are still initializing...${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Step 8: Show final status
|
||||||
|
echo ""
|
||||||
|
echo -e "${GREEN}✅ Deployment Complete!${NC}"
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}Current pod status:${NC}"
|
||||||
|
kubectl -n site11-pipeline get pods
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}External infrastructure status:${NC}"
|
||||||
|
docker ps --format "table {{.Names}}\t{{.Status}}" | grep -E "site11_(mongodb|redis|kafka|zookeeper)" || echo "No infrastructure services found"
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}Useful commands:${NC}"
|
||||||
|
echo " View logs: kubectl -n site11-pipeline logs -f deployment/pipeline-translator"
|
||||||
|
echo " Scale workers: kubectl -n site11-pipeline scale deployment pipeline-translator --replicas=5"
|
||||||
|
echo " Check HPA: kubectl -n site11-pipeline get hpa"
|
||||||
|
echo " Monitor queues: docker-compose -f docker-compose-hybrid.yml logs -f pipeline-monitor"
|
||||||
|
echo " Delete cluster: kind delete cluster --name site11-cluster"
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}Architecture Overview:${NC}"
|
||||||
|
echo " 📦 Infrastructure (Docker): MongoDB, Redis, Kafka"
|
||||||
|
echo " ☸️ Workers (Kind K8s): RSS, Search, Translation, AI Generation, Image Generation"
|
||||||
|
echo " 🎛️ Control (Docker): Scheduler, Monitor, Language Sync"
|
||||||
|
echo ""
|
||||||
|
echo -e "${YELLOW}Note: Kind uses 'host.docker.internal' to access host services${NC}"
|
||||||
170
k8s/pipeline/deploy-local.sh
Executable file
170
k8s/pipeline/deploy-local.sh
Executable file
@ -0,0 +1,170 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# Site11 Pipeline K8s Local Deployment Script
|
||||||
|
# ===========================================
|
||||||
|
# Deploys pipeline workers to K8s with external infrastructure (Docker Compose)
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
echo "🚀 Site11 Pipeline K8s Local Deployment (AWS-like Environment)"
|
||||||
|
echo "=============================================================="
|
||||||
|
echo ""
|
||||||
|
echo "This deployment simulates AWS architecture:"
|
||||||
|
echo " - Infrastructure: External (Docker Compose) - simulates AWS managed services"
|
||||||
|
echo " - Workers: K8s (local cluster) - simulates EKS workloads"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
# Check prerequisites
|
||||||
|
echo -e "${BLUE}Checking prerequisites...${NC}"
|
||||||
|
|
||||||
|
# Check if kubectl is available
|
||||||
|
if ! command -v kubectl &> /dev/null; then
|
||||||
|
echo -e "${RED}❌ kubectl is not installed${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check K8s cluster connection
|
||||||
|
echo -n " K8s cluster connection... "
|
||||||
|
if kubectl cluster-info &> /dev/null; then
|
||||||
|
echo -e "${GREEN}✓${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${RED}✗ Cannot connect to K8s cluster${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check if Docker infrastructure is running
|
||||||
|
echo -n " Docker infrastructure services... "
|
||||||
|
if docker ps | grep -q "site11_mongodb" && docker ps | grep -q "site11_redis"; then
|
||||||
|
echo -e "${GREEN}✓${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${YELLOW}⚠️ Infrastructure not running. Start with: docker-compose -f docker-compose-hybrid.yml up -d${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check local registry
|
||||||
|
echo -n " Local registry (port 5555)... "
|
||||||
|
if docker ps | grep -q "site11_registry"; then
|
||||||
|
echo -e "${GREEN}✓${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${YELLOW}⚠️ Registry not running. Start with: docker-compose -f docker-compose-hybrid.yml up -d registry${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Step 1: Create namespace
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}1. Creating K8s namespace...${NC}"
|
||||||
|
kubectl apply -f namespace.yaml
|
||||||
|
|
||||||
|
# Step 2: Create ConfigMap and Secrets for external services
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}2. Configuring external service connections...${NC}"
|
||||||
|
cat > configmap-local.yaml << 'EOF'
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: pipeline-config
|
||||||
|
namespace: site11-pipeline
|
||||||
|
data:
|
||||||
|
# External Redis (Docker host) - simulates AWS ElastiCache
|
||||||
|
REDIS_URL: "redis://host.docker.internal:6379"
|
||||||
|
|
||||||
|
# External MongoDB (Docker host) - simulates AWS DocumentDB
|
||||||
|
MONGODB_URL: "mongodb://host.docker.internal:27017"
|
||||||
|
DB_NAME: "ai_writer_db"
|
||||||
|
|
||||||
|
# Logging
|
||||||
|
LOG_LEVEL: "INFO"
|
||||||
|
|
||||||
|
# Worker settings
|
||||||
|
WORKER_COUNT: "2"
|
||||||
|
BATCH_SIZE: "10"
|
||||||
|
|
||||||
|
# Queue delays
|
||||||
|
RSS_ENQUEUE_DELAY: "1.0"
|
||||||
|
GOOGLE_SEARCH_DELAY: "2.0"
|
||||||
|
TRANSLATION_DELAY: "1.0"
|
||||||
|
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: pipeline-secrets
|
||||||
|
namespace: site11-pipeline
|
||||||
|
type: Opaque
|
||||||
|
stringData:
|
||||||
|
DEEPL_API_KEY: "3abbc796-2515-44a8-972d-22dcf27ab54a"
|
||||||
|
CLAUDE_API_KEY: "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"
|
||||||
|
OPENAI_API_KEY: "sk-openai-api-key-here" # Replace with actual key
|
||||||
|
SERP_API_KEY: "serp-api-key-here" # Replace with actual key
|
||||||
|
EOF
|
||||||
|
|
||||||
|
kubectl apply -f configmap-local.yaml
|
||||||
|
|
||||||
|
# Step 3: Update deployment YAMLs to use local registry
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}3. Updating deployments for local registry...${NC}"
|
||||||
|
services=("rss-collector" "google-search" "translator" "ai-article-generator" "image-generator")
|
||||||
|
|
||||||
|
for service in "${services[@]}"; do
|
||||||
|
# Update image references in deployment files
|
||||||
|
sed -i.bak "s|image: site11/pipeline-$service:latest|image: localhost:5555/pipeline-$service:latest|g" $service.yaml 2>/dev/null || \
|
||||||
|
sed -i '' "s|image: site11/pipeline-$service:latest|image: localhost:5555/pipeline-$service:latest|g" $service.yaml
|
||||||
|
done
|
||||||
|
|
||||||
|
# Step 4: Push images to local registry
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}4. Pushing images to local registry...${NC}"
|
||||||
|
for service in "${services[@]}"; do
|
||||||
|
echo -n " Pushing pipeline-$service... "
|
||||||
|
docker tag site11-pipeline-$service:latest localhost:5555/pipeline-$service:latest 2>/dev/null
|
||||||
|
docker push localhost:5555/pipeline-$service:latest 2>/dev/null && echo -e "${GREEN}✓${NC}" || echo -e "${YELLOW}already exists${NC}"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Step 5: Deploy services to K8s
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}5. Deploying workers to K8s...${NC}"
|
||||||
|
|
||||||
|
for service in "${services[@]}"; do
|
||||||
|
echo -n " Deploying $service... "
|
||||||
|
kubectl apply -f $service.yaml && echo -e "${GREEN}✓${NC}"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Step 6: Check deployment status
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}6. Verifying deployments...${NC}"
|
||||||
|
kubectl -n site11-pipeline get deployments
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}7. Waiting for pods to be ready...${NC}"
|
||||||
|
kubectl -n site11-pipeline wait --for=condition=Ready pods --all --timeout=60s 2>/dev/null || {
|
||||||
|
echo -e "${YELLOW}⚠️ Some pods are still initializing...${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Step 7: Show final status
|
||||||
|
echo ""
|
||||||
|
echo -e "${GREEN}✅ Deployment Complete!${NC}"
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}Current pod status:${NC}"
|
||||||
|
kubectl -n site11-pipeline get pods
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}External infrastructure status:${NC}"
|
||||||
|
docker ps --format "table {{.Names}}\t{{.Status}}" | grep -E "site11_(mongodb|redis|kafka|zookeeper|registry)" || echo "No infrastructure services found"
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}Useful commands:${NC}"
|
||||||
|
echo " View logs: kubectl -n site11-pipeline logs -f deployment/pipeline-translator"
|
||||||
|
echo " Scale workers: kubectl -n site11-pipeline scale deployment pipeline-translator --replicas=5"
|
||||||
|
echo " Check HPA: kubectl -n site11-pipeline get hpa"
|
||||||
|
echo " Monitor queues: docker-compose -f docker-compose-hybrid.yml logs -f pipeline-monitor"
|
||||||
|
echo " Delete K8s: kubectl delete namespace site11-pipeline"
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}Architecture Overview:${NC}"
|
||||||
|
echo " 📦 Infrastructure (Docker): MongoDB, Redis, Kafka, Registry"
|
||||||
|
echo " ☸️ Workers (K8s): RSS, Search, Translation, AI Generation, Image Generation"
|
||||||
|
echo " 🎛️ Control (Docker): Scheduler, Monitor, Language Sync"
|
||||||
@ -5,7 +5,6 @@ metadata:
|
|||||||
namespace: site11-pipeline
|
namespace: site11-pipeline
|
||||||
labels:
|
labels:
|
||||||
app: pipeline-google-search
|
app: pipeline-google-search
|
||||||
component: data-collector
|
|
||||||
spec:
|
spec:
|
||||||
replicas: 2
|
replicas: 2
|
||||||
selector:
|
selector:
|
||||||
@ -15,12 +14,11 @@ spec:
|
|||||||
metadata:
|
metadata:
|
||||||
labels:
|
labels:
|
||||||
app: pipeline-google-search
|
app: pipeline-google-search
|
||||||
component: data-collector
|
|
||||||
spec:
|
spec:
|
||||||
containers:
|
containers:
|
||||||
- name: google-search
|
- name: google-search
|
||||||
image: site11/pipeline-google-search:latest
|
image: yakenator/site11-pipeline-google-search:latest
|
||||||
imagePullPolicy: Always
|
imagePullPolicy: Always # Always pull from Docker Hub
|
||||||
envFrom:
|
envFrom:
|
||||||
- configMapRef:
|
- configMapRef:
|
||||||
name: pipeline-config
|
name: pipeline-config
|
||||||
@ -33,23 +31,22 @@ spec:
|
|||||||
limits:
|
limits:
|
||||||
memory: "512Mi"
|
memory: "512Mi"
|
||||||
cpu: "500m"
|
cpu: "500m"
|
||||||
livenessProbe:
|
|
||||||
exec:
|
|
||||||
command:
|
|
||||||
- python
|
|
||||||
- -c
|
|
||||||
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
|
|
||||||
initialDelaySeconds: 30
|
|
||||||
periodSeconds: 30
|
|
||||||
readinessProbe:
|
readinessProbe:
|
||||||
exec:
|
exec:
|
||||||
command:
|
command:
|
||||||
- python
|
- python
|
||||||
- -c
|
- -c
|
||||||
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
|
- "import sys; sys.exit(0)"
|
||||||
initialDelaySeconds: 10
|
initialDelaySeconds: 10
|
||||||
|
periodSeconds: 5
|
||||||
|
livenessProbe:
|
||||||
|
exec:
|
||||||
|
command:
|
||||||
|
- python
|
||||||
|
- -c
|
||||||
|
- "import sys; sys.exit(0)"
|
||||||
|
initialDelaySeconds: 30
|
||||||
periodSeconds: 10
|
periodSeconds: 10
|
||||||
|
|
||||||
---
|
---
|
||||||
apiVersion: autoscaling/v2
|
apiVersion: autoscaling/v2
|
||||||
kind: HorizontalPodAutoscaler
|
kind: HorizontalPodAutoscaler
|
||||||
@ -61,8 +58,8 @@ spec:
|
|||||||
apiVersion: apps/v1
|
apiVersion: apps/v1
|
||||||
kind: Deployment
|
kind: Deployment
|
||||||
name: pipeline-google-search
|
name: pipeline-google-search
|
||||||
minReplicas: 1
|
minReplicas: 2
|
||||||
maxReplicas: 5
|
maxReplicas: 10
|
||||||
metrics:
|
metrics:
|
||||||
- type: Resource
|
- type: Resource
|
||||||
resource:
|
resource:
|
||||||
@ -75,4 +72,4 @@ spec:
|
|||||||
name: memory
|
name: memory
|
||||||
target:
|
target:
|
||||||
type: Utilization
|
type: Utilization
|
||||||
averageUtilization: 80
|
averageUtilization: 80
|
||||||
@ -5,7 +5,6 @@ metadata:
|
|||||||
namespace: site11-pipeline
|
namespace: site11-pipeline
|
||||||
labels:
|
labels:
|
||||||
app: pipeline-image-generator
|
app: pipeline-image-generator
|
||||||
component: processor
|
|
||||||
spec:
|
spec:
|
||||||
replicas: 2
|
replicas: 2
|
||||||
selector:
|
selector:
|
||||||
@ -15,12 +14,11 @@ spec:
|
|||||||
metadata:
|
metadata:
|
||||||
labels:
|
labels:
|
||||||
app: pipeline-image-generator
|
app: pipeline-image-generator
|
||||||
component: processor
|
|
||||||
spec:
|
spec:
|
||||||
containers:
|
containers:
|
||||||
- name: image-generator
|
- name: image-generator
|
||||||
image: site11/pipeline-image-generator:latest
|
image: yakenator/site11-pipeline-image-generator:latest
|
||||||
imagePullPolicy: Always
|
imagePullPolicy: Always # Always pull from Docker Hub
|
||||||
envFrom:
|
envFrom:
|
||||||
- configMapRef:
|
- configMapRef:
|
||||||
name: pipeline-config
|
name: pipeline-config
|
||||||
@ -28,28 +26,27 @@ spec:
|
|||||||
name: pipeline-secrets
|
name: pipeline-secrets
|
||||||
resources:
|
resources:
|
||||||
requests:
|
requests:
|
||||||
memory: "512Mi"
|
memory: "256Mi"
|
||||||
cpu: "200m"
|
cpu: "100m"
|
||||||
limits:
|
limits:
|
||||||
memory: "1Gi"
|
memory: "512Mi"
|
||||||
cpu: "1000m"
|
cpu: "500m"
|
||||||
livenessProbe:
|
|
||||||
exec:
|
|
||||||
command:
|
|
||||||
- python
|
|
||||||
- -c
|
|
||||||
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
|
|
||||||
initialDelaySeconds: 30
|
|
||||||
periodSeconds: 30
|
|
||||||
readinessProbe:
|
readinessProbe:
|
||||||
exec:
|
exec:
|
||||||
command:
|
command:
|
||||||
- python
|
- python
|
||||||
- -c
|
- -c
|
||||||
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
|
- "import sys; sys.exit(0)"
|
||||||
initialDelaySeconds: 10
|
initialDelaySeconds: 10
|
||||||
|
periodSeconds: 5
|
||||||
|
livenessProbe:
|
||||||
|
exec:
|
||||||
|
command:
|
||||||
|
- python
|
||||||
|
- -c
|
||||||
|
- "import sys; sys.exit(0)"
|
||||||
|
initialDelaySeconds: 30
|
||||||
periodSeconds: 10
|
periodSeconds: 10
|
||||||
|
|
||||||
---
|
---
|
||||||
apiVersion: autoscaling/v2
|
apiVersion: autoscaling/v2
|
||||||
kind: HorizontalPodAutoscaler
|
kind: HorizontalPodAutoscaler
|
||||||
@ -61,8 +58,8 @@ spec:
|
|||||||
apiVersion: apps/v1
|
apiVersion: apps/v1
|
||||||
kind: Deployment
|
kind: Deployment
|
||||||
name: pipeline-image-generator
|
name: pipeline-image-generator
|
||||||
minReplicas: 1
|
minReplicas: 2
|
||||||
maxReplicas: 6
|
maxReplicas: 10
|
||||||
metrics:
|
metrics:
|
||||||
- type: Resource
|
- type: Resource
|
||||||
resource:
|
resource:
|
||||||
@ -75,4 +72,4 @@ spec:
|
|||||||
name: memory
|
name: memory
|
||||||
target:
|
target:
|
||||||
type: Utilization
|
type: Utilization
|
||||||
averageUtilization: 80
|
averageUtilization: 80
|
||||||
@ -5,7 +5,6 @@ metadata:
|
|||||||
namespace: site11-pipeline
|
namespace: site11-pipeline
|
||||||
labels:
|
labels:
|
||||||
app: pipeline-rss-collector
|
app: pipeline-rss-collector
|
||||||
component: data-collector
|
|
||||||
spec:
|
spec:
|
||||||
replicas: 2
|
replicas: 2
|
||||||
selector:
|
selector:
|
||||||
@ -15,12 +14,11 @@ spec:
|
|||||||
metadata:
|
metadata:
|
||||||
labels:
|
labels:
|
||||||
app: pipeline-rss-collector
|
app: pipeline-rss-collector
|
||||||
component: data-collector
|
|
||||||
spec:
|
spec:
|
||||||
containers:
|
containers:
|
||||||
- name: rss-collector
|
- name: rss-collector
|
||||||
image: site11/pipeline-rss-collector:latest
|
image: yakenator/site11-pipeline-rss-collector:latest
|
||||||
imagePullPolicy: Always
|
imagePullPolicy: Always # Always pull from Docker Hub
|
||||||
envFrom:
|
envFrom:
|
||||||
- configMapRef:
|
- configMapRef:
|
||||||
name: pipeline-config
|
name: pipeline-config
|
||||||
@ -33,23 +31,22 @@ spec:
|
|||||||
limits:
|
limits:
|
||||||
memory: "512Mi"
|
memory: "512Mi"
|
||||||
cpu: "500m"
|
cpu: "500m"
|
||||||
livenessProbe:
|
|
||||||
exec:
|
|
||||||
command:
|
|
||||||
- python
|
|
||||||
- -c
|
|
||||||
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
|
|
||||||
initialDelaySeconds: 30
|
|
||||||
periodSeconds: 30
|
|
||||||
readinessProbe:
|
readinessProbe:
|
||||||
exec:
|
exec:
|
||||||
command:
|
command:
|
||||||
- python
|
- python
|
||||||
- -c
|
- -c
|
||||||
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
|
- "import sys; sys.exit(0)"
|
||||||
initialDelaySeconds: 10
|
initialDelaySeconds: 10
|
||||||
|
periodSeconds: 5
|
||||||
|
livenessProbe:
|
||||||
|
exec:
|
||||||
|
command:
|
||||||
|
- python
|
||||||
|
- -c
|
||||||
|
- "import sys; sys.exit(0)"
|
||||||
|
initialDelaySeconds: 30
|
||||||
periodSeconds: 10
|
periodSeconds: 10
|
||||||
|
|
||||||
---
|
---
|
||||||
apiVersion: autoscaling/v2
|
apiVersion: autoscaling/v2
|
||||||
kind: HorizontalPodAutoscaler
|
kind: HorizontalPodAutoscaler
|
||||||
@ -61,8 +58,8 @@ spec:
|
|||||||
apiVersion: apps/v1
|
apiVersion: apps/v1
|
||||||
kind: Deployment
|
kind: Deployment
|
||||||
name: pipeline-rss-collector
|
name: pipeline-rss-collector
|
||||||
minReplicas: 1
|
minReplicas: 2
|
||||||
maxReplicas: 5
|
maxReplicas: 10
|
||||||
metrics:
|
metrics:
|
||||||
- type: Resource
|
- type: Resource
|
||||||
resource:
|
resource:
|
||||||
@ -75,4 +72,4 @@ spec:
|
|||||||
name: memory
|
name: memory
|
||||||
target:
|
target:
|
||||||
type: Utilization
|
type: Utilization
|
||||||
averageUtilization: 80
|
averageUtilization: 80
|
||||||
@ -5,7 +5,6 @@ metadata:
|
|||||||
namespace: site11-pipeline
|
namespace: site11-pipeline
|
||||||
labels:
|
labels:
|
||||||
app: pipeline-translator
|
app: pipeline-translator
|
||||||
component: processor
|
|
||||||
spec:
|
spec:
|
||||||
replicas: 3
|
replicas: 3
|
||||||
selector:
|
selector:
|
||||||
@ -15,12 +14,11 @@ spec:
|
|||||||
metadata:
|
metadata:
|
||||||
labels:
|
labels:
|
||||||
app: pipeline-translator
|
app: pipeline-translator
|
||||||
component: processor
|
|
||||||
spec:
|
spec:
|
||||||
containers:
|
containers:
|
||||||
- name: translator
|
- name: translator
|
||||||
image: site11/pipeline-translator:latest
|
image: yakenator/site11-pipeline-translator:latest
|
||||||
imagePullPolicy: Always
|
imagePullPolicy: Always # Always pull from Docker Hub
|
||||||
envFrom:
|
envFrom:
|
||||||
- configMapRef:
|
- configMapRef:
|
||||||
name: pipeline-config
|
name: pipeline-config
|
||||||
@ -28,28 +26,27 @@ spec:
|
|||||||
name: pipeline-secrets
|
name: pipeline-secrets
|
||||||
resources:
|
resources:
|
||||||
requests:
|
requests:
|
||||||
memory: "512Mi"
|
memory: "256Mi"
|
||||||
cpu: "200m"
|
cpu: "100m"
|
||||||
limits:
|
limits:
|
||||||
memory: "1Gi"
|
memory: "512Mi"
|
||||||
cpu: "1000m"
|
cpu: "500m"
|
||||||
livenessProbe:
|
|
||||||
exec:
|
|
||||||
command:
|
|
||||||
- python
|
|
||||||
- -c
|
|
||||||
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
|
|
||||||
initialDelaySeconds: 30
|
|
||||||
periodSeconds: 30
|
|
||||||
readinessProbe:
|
readinessProbe:
|
||||||
exec:
|
exec:
|
||||||
command:
|
command:
|
||||||
- python
|
- python
|
||||||
- -c
|
- -c
|
||||||
- "import redis; r=redis.from_url('redis://host.docker.internal:6379'); r.ping()"
|
- "import sys; sys.exit(0)"
|
||||||
initialDelaySeconds: 10
|
initialDelaySeconds: 10
|
||||||
|
periodSeconds: 5
|
||||||
|
livenessProbe:
|
||||||
|
exec:
|
||||||
|
command:
|
||||||
|
- python
|
||||||
|
- -c
|
||||||
|
- "import sys; sys.exit(0)"
|
||||||
|
initialDelaySeconds: 30
|
||||||
periodSeconds: 10
|
periodSeconds: 10
|
||||||
|
|
||||||
---
|
---
|
||||||
apiVersion: autoscaling/v2
|
apiVersion: autoscaling/v2
|
||||||
kind: HorizontalPodAutoscaler
|
kind: HorizontalPodAutoscaler
|
||||||
@ -61,7 +58,7 @@ spec:
|
|||||||
apiVersion: apps/v1
|
apiVersion: apps/v1
|
||||||
kind: Deployment
|
kind: Deployment
|
||||||
name: pipeline-translator
|
name: pipeline-translator
|
||||||
minReplicas: 2
|
minReplicas: 3
|
||||||
maxReplicas: 10
|
maxReplicas: 10
|
||||||
metrics:
|
metrics:
|
||||||
- type: Resource
|
- type: Resource
|
||||||
@ -75,4 +72,4 @@ spec:
|
|||||||
name: memory
|
name: memory
|
||||||
target:
|
target:
|
||||||
type: Utilization
|
type: Utilization
|
||||||
averageUtilization: 80
|
averageUtilization: 80
|
||||||
86
registry/config.yml
Normal file
86
registry/config.yml
Normal file
@ -0,0 +1,86 @@
|
|||||||
|
version: 0.1
|
||||||
|
log:
|
||||||
|
level: info
|
||||||
|
formatter: text
|
||||||
|
fields:
|
||||||
|
service: registry
|
||||||
|
|
||||||
|
storage:
|
||||||
|
filesystem:
|
||||||
|
rootdirectory: /var/lib/registry
|
||||||
|
maxthreads: 100
|
||||||
|
cache:
|
||||||
|
blobdescriptor: redis
|
||||||
|
maintenance:
|
||||||
|
uploadpurging:
|
||||||
|
enabled: true
|
||||||
|
age: 168h
|
||||||
|
interval: 24h
|
||||||
|
dryrun: false
|
||||||
|
delete:
|
||||||
|
enabled: true
|
||||||
|
|
||||||
|
redis:
|
||||||
|
addr: registry-redis:6379
|
||||||
|
pool:
|
||||||
|
maxidle: 16
|
||||||
|
maxactive: 64
|
||||||
|
idletimeout: 300s
|
||||||
|
|
||||||
|
http:
|
||||||
|
addr: :5000
|
||||||
|
headers:
|
||||||
|
X-Content-Type-Options: [nosniff]
|
||||||
|
http2:
|
||||||
|
disabled: false
|
||||||
|
|
||||||
|
# Proxy configuration for Docker Hub caching
|
||||||
|
proxy:
|
||||||
|
remoteurl: https://registry-1.docker.io
|
||||||
|
ttl: 168h # Cache for 7 days
|
||||||
|
|
||||||
|
# Health check
|
||||||
|
health:
|
||||||
|
storagedriver:
|
||||||
|
enabled: true
|
||||||
|
interval: 10s
|
||||||
|
threshold: 3
|
||||||
|
|
||||||
|
# Middleware for rate limiting and caching
|
||||||
|
middleware:
|
||||||
|
storage:
|
||||||
|
- name: cloudfront
|
||||||
|
options:
|
||||||
|
baseurl: https://registry-1.docker.io/
|
||||||
|
privatekey: /etc/docker/registry/pk.pem
|
||||||
|
keypairid: KEYPAIRID
|
||||||
|
duration: 3000s
|
||||||
|
ipfilteredby: aws
|
||||||
|
|
||||||
|
# Notifications (optional - for monitoring)
|
||||||
|
notifications:
|
||||||
|
endpoints:
|
||||||
|
- name: local-endpoint
|
||||||
|
url: http://pipeline-monitor:8100/webhook/registry
|
||||||
|
headers:
|
||||||
|
Authorization: [Bearer]
|
||||||
|
timeout: 1s
|
||||||
|
threshold: 10
|
||||||
|
backoff: 1s
|
||||||
|
disabled: false
|
||||||
|
|
||||||
|
# Garbage collection
|
||||||
|
gc:
|
||||||
|
enabled: true
|
||||||
|
interval: 12h
|
||||||
|
readonly:
|
||||||
|
enabled: false
|
||||||
|
|
||||||
|
# Validation
|
||||||
|
validation:
|
||||||
|
manifests:
|
||||||
|
urls:
|
||||||
|
allow:
|
||||||
|
- ^https?://
|
||||||
|
deny:
|
||||||
|
- ^http://localhost/
|
||||||
60
scripts/backup-mongodb.sh
Executable file
60
scripts/backup-mongodb.sh
Executable file
@ -0,0 +1,60 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# MongoDB Backup Script
|
||||||
|
# =====================
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# Colors
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
NC='\033[0m'
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
BACKUP_DIR="/Users/jungwoochoi/Desktop/prototype/site11/backups"
|
||||||
|
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||||
|
BACKUP_NAME="backup_$TIMESTAMP"
|
||||||
|
CONTAINER_NAME="site11_mongodb"
|
||||||
|
|
||||||
|
echo -e "${GREEN}MongoDB Backup Script${NC}"
|
||||||
|
echo "========================"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Create backup directory if it doesn't exist
|
||||||
|
mkdir -p "$BACKUP_DIR"
|
||||||
|
|
||||||
|
# Step 1: Create dump inside container
|
||||||
|
echo "1. Creating MongoDB dump..."
|
||||||
|
docker exec $CONTAINER_NAME mongodump --out /data/db/$BACKUP_NAME 2>/dev/null || {
|
||||||
|
echo -e "${YELLOW}Warning: Some collections might be empty${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Step 2: Copy backup to host
|
||||||
|
echo "2. Copying backup to host..."
|
||||||
|
docker cp $CONTAINER_NAME:/data/db/$BACKUP_NAME "$BACKUP_DIR/"
|
||||||
|
|
||||||
|
# Step 3: Compress backup
|
||||||
|
echo "3. Compressing backup..."
|
||||||
|
cd "$BACKUP_DIR"
|
||||||
|
tar -czf "$BACKUP_NAME.tar.gz" "$BACKUP_NAME"
|
||||||
|
rm -rf "$BACKUP_NAME"
|
||||||
|
|
||||||
|
# Step 4: Clean up old backups (keep only last 5)
|
||||||
|
echo "4. Cleaning up old backups..."
|
||||||
|
ls -t *.tar.gz 2>/dev/null | tail -n +6 | xargs rm -f 2>/dev/null || true
|
||||||
|
|
||||||
|
# Step 5: Show backup info
|
||||||
|
SIZE=$(ls -lh "$BACKUP_NAME.tar.gz" | awk '{print $5}')
|
||||||
|
echo ""
|
||||||
|
echo -e "${GREEN}✅ Backup completed successfully!${NC}"
|
||||||
|
echo " File: $BACKUP_DIR/$BACKUP_NAME.tar.gz"
|
||||||
|
echo " Size: $SIZE"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Optional: Clean up container backups older than 7 days
|
||||||
|
docker exec $CONTAINER_NAME find /data/db -name "backup_*" -type d -mtime +7 -exec rm -rf {} + 2>/dev/null || true
|
||||||
|
|
||||||
|
echo "To restore this backup, use:"
|
||||||
|
echo " tar -xzf $BACKUP_NAME.tar.gz"
|
||||||
|
echo " docker cp $BACKUP_NAME $CONTAINER_NAME:/data/db/"
|
||||||
|
echo " docker exec $CONTAINER_NAME mongorestore /data/db/$BACKUP_NAME"
|
||||||
103
scripts/deploy-news-api.sh
Executable file
103
scripts/deploy-news-api.sh
Executable file
@ -0,0 +1,103 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
echo "=================================================="
|
||||||
|
echo " News API Kubernetes Deployment"
|
||||||
|
echo "=================================================="
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Color codes
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
RED='\033[0;31m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
# Check if DOCKER_HUB_USER is set
|
||||||
|
if [ -z "$DOCKER_HUB_USER" ]; then
|
||||||
|
echo -e "${RED}Error: DOCKER_HUB_USER environment variable is not set${NC}"
|
||||||
|
echo "Please run: export DOCKER_HUB_USER=your-username"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Deployment option
|
||||||
|
DEPLOYMENT_TYPE=${1:-local}
|
||||||
|
|
||||||
|
echo -e "${BLUE}Deployment Type: ${DEPLOYMENT_TYPE}${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Step 1: Build Docker Image
|
||||||
|
echo -e "${YELLOW}[1/4] Building News API Docker image...${NC}"
|
||||||
|
docker build -t site11/news-api:latest services/news-api/backend/
|
||||||
|
echo -e "${GREEN}✓ Image built successfully${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Step 2: Push or Load Image
|
||||||
|
if [ "$DEPLOYMENT_TYPE" == "dockerhub" ]; then
|
||||||
|
echo -e "${YELLOW}[2/4] Tagging and pushing to Docker Hub...${NC}"
|
||||||
|
docker tag site11/news-api:latest ${DOCKER_HUB_USER}/news-api:latest
|
||||||
|
docker push ${DOCKER_HUB_USER}/news-api:latest
|
||||||
|
echo -e "${GREEN}✓ Image pushed to Docker Hub${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo -e "${YELLOW}[3/4] Deploying to Kubernetes with Docker Hub image...${NC}"
|
||||||
|
envsubst < k8s/news-api/news-api-dockerhub.yaml | kubectl apply -f -
|
||||||
|
elif [ "$DEPLOYMENT_TYPE" == "kind" ]; then
|
||||||
|
echo -e "${YELLOW}[2/4] Loading image to Kind cluster...${NC}"
|
||||||
|
kind load docker-image site11/news-api:latest --name site11-cluster
|
||||||
|
echo -e "${GREEN}✓ Image loaded to Kind${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo -e "${YELLOW}[3/4] Deploying to Kind Kubernetes...${NC}"
|
||||||
|
kubectl apply -f k8s/news-api/news-api-deployment.yaml
|
||||||
|
else
|
||||||
|
echo -e "${YELLOW}[2/4] Using local image...${NC}"
|
||||||
|
echo -e "${GREEN}✓ Image ready${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo -e "${YELLOW}[3/4] Deploying to Kubernetes...${NC}"
|
||||||
|
kubectl apply -f k8s/news-api/news-api-deployment.yaml
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo -e "${GREEN}✓ Deployment applied${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Step 4: Wait for Pods
|
||||||
|
echo -e "${YELLOW}[4/4] Waiting for pods to be ready...${NC}"
|
||||||
|
kubectl wait --for=condition=ready pod -l app=news-api -n site11-news --timeout=120s || true
|
||||||
|
echo -e "${GREEN}✓ Pods are ready${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Display Status
|
||||||
|
echo -e "${BLUE}=================================================="
|
||||||
|
echo " Deployment Status"
|
||||||
|
echo "==================================================${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo -e "${YELLOW}Pods:${NC}"
|
||||||
|
kubectl -n site11-news get pods
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e "${YELLOW}Service:${NC}"
|
||||||
|
kubectl -n site11-news get svc
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e "${YELLOW}HPA:${NC}"
|
||||||
|
kubectl -n site11-news get hpa
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e "${BLUE}=================================================="
|
||||||
|
echo " Access the API"
|
||||||
|
echo "==================================================${NC}"
|
||||||
|
echo ""
|
||||||
|
echo "Port forward to access locally:"
|
||||||
|
echo -e "${GREEN}kubectl -n site11-news port-forward svc/news-api-service 8050:8000${NC}"
|
||||||
|
echo ""
|
||||||
|
echo "Then visit:"
|
||||||
|
echo " - Health: http://localhost:8050/health"
|
||||||
|
echo " - Docs: http://localhost:8050/docs"
|
||||||
|
echo " - Korean Articles: http://localhost:8050/api/v1/ko/articles"
|
||||||
|
echo " - Latest: http://localhost:8050/api/v1/en/articles/latest"
|
||||||
|
echo ""
|
||||||
|
echo -e "${GREEN}✓ Deployment completed successfully!${NC}"
|
||||||
225
scripts/kind-setup.sh
Executable file
225
scripts/kind-setup.sh
Executable file
@ -0,0 +1,225 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# Site11 KIND Cluster Setup Script
|
||||||
|
# This script manages the KIND (Kubernetes IN Docker) development cluster
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
CLUSTER_NAME="site11-dev"
|
||||||
|
CONFIG_FILE="k8s/kind-dev-cluster.yaml"
|
||||||
|
K8S_DIR="k8s/kind"
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
echo -e "${GREEN}=====================================${NC}"
|
||||||
|
echo -e "${GREEN}Site11 KIND Cluster Manager${NC}"
|
||||||
|
echo -e "${GREEN}=====================================${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Check if KIND is installed
|
||||||
|
if ! command -v kind &> /dev/null; then
|
||||||
|
echo -e "${RED}ERROR: kind is not installed${NC}"
|
||||||
|
echo "Please install KIND: https://kind.sigs.k8s.io/docs/user/quick-start/#installation"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check if kubectl is installed
|
||||||
|
if ! command -v kubectl &> /dev/null; then
|
||||||
|
echo -e "${RED}ERROR: kubectl is not installed${NC}"
|
||||||
|
echo "Please install kubectl: https://kubernetes.io/docs/tasks/tools/"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Function to create cluster
|
||||||
|
create_cluster() {
|
||||||
|
echo -e "${YELLOW}Creating KIND cluster: $CLUSTER_NAME${NC}"
|
||||||
|
|
||||||
|
if kind get clusters | grep -q "^$CLUSTER_NAME$"; then
|
||||||
|
echo -e "${YELLOW}Cluster $CLUSTER_NAME already exists${NC}"
|
||||||
|
read -p "Do you want to delete and recreate? (y/N): " -n 1 -r
|
||||||
|
echo
|
||||||
|
if [[ $REPLY =~ ^[Yy]$ ]]; then
|
||||||
|
delete_cluster
|
||||||
|
else
|
||||||
|
echo "Skipping cluster creation"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
kind create cluster --config "$CONFIG_FILE"
|
||||||
|
echo -e "${GREEN}✅ Cluster created successfully${NC}"
|
||||||
|
|
||||||
|
# Wait for cluster to be ready
|
||||||
|
echo "Waiting for cluster to be ready..."
|
||||||
|
kubectl wait --for=condition=Ready nodes --all --timeout=120s
|
||||||
|
|
||||||
|
echo -e "${GREEN}✅ All nodes are ready${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to delete cluster
|
||||||
|
delete_cluster() {
|
||||||
|
echo -e "${YELLOW}Deleting KIND cluster: $CLUSTER_NAME${NC}"
|
||||||
|
kind delete cluster --name "$CLUSTER_NAME"
|
||||||
|
echo -e "${GREEN}✅ Cluster deleted${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to deploy namespaces
|
||||||
|
deploy_namespaces() {
|
||||||
|
echo -e "${YELLOW}Creating namespaces${NC}"
|
||||||
|
kubectl create namespace site11-console --dry-run=client -o yaml | kubectl apply -f -
|
||||||
|
kubectl create namespace site11-pipeline --dry-run=client -o yaml | kubectl apply -f -
|
||||||
|
echo -e "${GREEN}✅ Namespaces created${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to load images
|
||||||
|
load_images() {
|
||||||
|
echo -e "${YELLOW}Loading Docker images into KIND cluster${NC}"
|
||||||
|
|
||||||
|
images=(
|
||||||
|
"yakenator/site11-console-backend:latest"
|
||||||
|
"yakenator/site11-console-frontend:latest"
|
||||||
|
)
|
||||||
|
|
||||||
|
for image in "${images[@]}"; do
|
||||||
|
echo "Loading $image..."
|
||||||
|
if docker image inspect "$image" &> /dev/null; then
|
||||||
|
kind load docker-image "$image" --name "$CLUSTER_NAME"
|
||||||
|
else
|
||||||
|
echo -e "${YELLOW}Warning: Image $image not found locally, skipping${NC}"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
echo -e "${GREEN}✅ Images loaded${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to deploy console services
|
||||||
|
deploy_console() {
|
||||||
|
echo -e "${YELLOW}Deploying Console services${NC}"
|
||||||
|
|
||||||
|
# Deploy in order: databases first, then applications
|
||||||
|
kubectl apply -f "$K8S_DIR/console-mongodb-redis.yaml"
|
||||||
|
echo "Waiting for databases to be ready..."
|
||||||
|
sleep 5
|
||||||
|
|
||||||
|
kubectl apply -f "$K8S_DIR/console-backend.yaml"
|
||||||
|
kubectl apply -f "$K8S_DIR/console-frontend.yaml"
|
||||||
|
|
||||||
|
echo -e "${GREEN}✅ Console services deployed${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to check cluster status
|
||||||
|
status() {
|
||||||
|
echo -e "${YELLOW}Cluster Status${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
if ! kind get clusters | grep -q "^$CLUSTER_NAME$"; then
|
||||||
|
echo -e "${RED}Cluster $CLUSTER_NAME does not exist${NC}"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "=== Nodes ==="
|
||||||
|
kubectl get nodes
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo "=== Console Namespace Pods ==="
|
||||||
|
kubectl get pods -n site11-console -o wide
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo "=== Console Services ==="
|
||||||
|
kubectl get svc -n site11-console
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo "=== Pipeline Namespace Pods ==="
|
||||||
|
kubectl get pods -n site11-pipeline -o wide 2>/dev/null || echo "No pods found"
|
||||||
|
echo ""
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to show logs
|
||||||
|
logs() {
|
||||||
|
namespace=${1:-site11-console}
|
||||||
|
pod_name=${2:-}
|
||||||
|
|
||||||
|
if [ -z "$pod_name" ]; then
|
||||||
|
echo "Available pods in namespace $namespace:"
|
||||||
|
kubectl get pods -n "$namespace" --no-headers | awk '{print $1}'
|
||||||
|
echo ""
|
||||||
|
echo "Usage: $0 logs [namespace] [pod-name]"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
kubectl logs -n "$namespace" "$pod_name" -f
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to access services
|
||||||
|
access() {
|
||||||
|
echo -e "${GREEN}Console Services Access Information${NC}"
|
||||||
|
echo ""
|
||||||
|
echo "Frontend: http://localhost:3000 (NodePort 30080)"
|
||||||
|
echo "Backend: http://localhost:8000 (NodePort 30081)"
|
||||||
|
echo ""
|
||||||
|
echo "These services are accessible because they use NodePort type"
|
||||||
|
echo "and are mapped in the KIND cluster configuration."
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to setup everything
|
||||||
|
setup() {
|
||||||
|
echo -e "${GREEN}Setting up complete KIND development environment${NC}"
|
||||||
|
create_cluster
|
||||||
|
deploy_namespaces
|
||||||
|
load_images
|
||||||
|
deploy_console
|
||||||
|
status
|
||||||
|
access
|
||||||
|
echo -e "${GREEN}✅ Setup complete!${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Main script logic
|
||||||
|
case "${1:-}" in
|
||||||
|
create)
|
||||||
|
create_cluster
|
||||||
|
;;
|
||||||
|
delete)
|
||||||
|
delete_cluster
|
||||||
|
;;
|
||||||
|
deploy-namespaces)
|
||||||
|
deploy_namespaces
|
||||||
|
;;
|
||||||
|
load-images)
|
||||||
|
load_images
|
||||||
|
;;
|
||||||
|
deploy-console)
|
||||||
|
deploy_console
|
||||||
|
;;
|
||||||
|
status)
|
||||||
|
status
|
||||||
|
;;
|
||||||
|
logs)
|
||||||
|
logs "$2" "$3"
|
||||||
|
;;
|
||||||
|
access)
|
||||||
|
access
|
||||||
|
;;
|
||||||
|
setup)
|
||||||
|
setup
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Usage: $0 {create|delete|deploy-namespaces|load-images|deploy-console|status|logs|access|setup}"
|
||||||
|
echo ""
|
||||||
|
echo "Commands:"
|
||||||
|
echo " create - Create KIND cluster"
|
||||||
|
echo " delete - Delete KIND cluster"
|
||||||
|
echo " deploy-namespaces - Create namespaces"
|
||||||
|
echo " load-images - Load Docker images into cluster"
|
||||||
|
echo " deploy-console - Deploy console services"
|
||||||
|
echo " status - Show cluster status"
|
||||||
|
echo " logs [ns] [pod] - Show pod logs"
|
||||||
|
echo " access - Show service access information"
|
||||||
|
echo " setup - Complete setup (create + deploy everything)"
|
||||||
|
echo ""
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
268
scripts/setup-registry-cache.sh
Normal file
268
scripts/setup-registry-cache.sh
Normal file
@ -0,0 +1,268 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
#
|
||||||
|
# Docker Registry Cache Setup Script
|
||||||
|
# Sets up and configures Docker registry cache for faster builds and deployments
|
||||||
|
#
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
echo -e "${GREEN}========================================${NC}"
|
||||||
|
echo -e "${GREEN}Docker Registry Cache Setup${NC}"
|
||||||
|
echo -e "${GREEN}========================================${NC}"
|
||||||
|
|
||||||
|
# Function to check if service is running
|
||||||
|
check_service() {
|
||||||
|
local service=$1
|
||||||
|
if docker ps --format "table {{.Names}}" | grep -q "$service"; then
|
||||||
|
echo -e "${GREEN}✓${NC} $service is running"
|
||||||
|
return 0
|
||||||
|
else
|
||||||
|
echo -e "${RED}✗${NC} $service is not running"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to wait for service to be ready
|
||||||
|
wait_for_service() {
|
||||||
|
local service=$1
|
||||||
|
local url=$2
|
||||||
|
local max_attempts=30
|
||||||
|
local attempt=0
|
||||||
|
|
||||||
|
echo -n "Waiting for $service to be ready..."
|
||||||
|
while [ $attempt -lt $max_attempts ]; do
|
||||||
|
if curl -s -f "$url" > /dev/null 2>&1; then
|
||||||
|
echo -e " ${GREEN}Ready!${NC}"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
echo -n "."
|
||||||
|
sleep 2
|
||||||
|
attempt=$((attempt + 1))
|
||||||
|
done
|
||||||
|
echo -e " ${RED}Timeout!${NC}"
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# 1. Start Registry Cache
|
||||||
|
echo -e "\n${YELLOW}1. Starting Registry Cache Service...${NC}"
|
||||||
|
docker-compose -f docker-compose-registry-cache.yml up -d registry-cache
|
||||||
|
|
||||||
|
# 2. Wait for registry to be ready
|
||||||
|
wait_for_service "Registry Cache" "http://localhost:5000/v2/"
|
||||||
|
|
||||||
|
# 3. Configure Docker daemon to use registry cache
|
||||||
|
echo -e "\n${YELLOW}2. Configuring Docker daemon...${NC}"
|
||||||
|
|
||||||
|
# Create daemon.json configuration
|
||||||
|
cat > /tmp/daemon.json.tmp <<EOF
|
||||||
|
{
|
||||||
|
"registry-mirrors": ["http://localhost:5000"],
|
||||||
|
"insecure-registries": ["localhost:5000", "127.0.0.1:5000"],
|
||||||
|
"max-concurrent-downloads": 10,
|
||||||
|
"max-concurrent-uploads": 5,
|
||||||
|
"storage-driver": "overlay2",
|
||||||
|
"log-driver": "json-file",
|
||||||
|
"log-opts": {
|
||||||
|
"max-size": "10m",
|
||||||
|
"max-file": "3"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# Check OS and apply configuration
|
||||||
|
if [[ "$OSTYPE" == "darwin"* ]]; then
|
||||||
|
echo -e "${YELLOW}macOS detected - Please configure Docker Desktop:${NC}"
|
||||||
|
echo "1. Open Docker Desktop"
|
||||||
|
echo "2. Go to Preferences > Docker Engine"
|
||||||
|
echo "3. Add the following configuration:"
|
||||||
|
cat /tmp/daemon.json.tmp
|
||||||
|
echo -e "\n4. Click 'Apply & Restart'"
|
||||||
|
echo -e "\n${YELLOW}Press Enter when Docker Desktop has been configured...${NC}"
|
||||||
|
read
|
||||||
|
elif [[ "$OSTYPE" == "linux-gnu"* ]]; then
|
||||||
|
# Linux - direct configuration
|
||||||
|
echo "Configuring Docker daemon for Linux..."
|
||||||
|
|
||||||
|
# Backup existing configuration
|
||||||
|
if [ -f /etc/docker/daemon.json ]; then
|
||||||
|
sudo cp /etc/docker/daemon.json /etc/docker/daemon.json.backup
|
||||||
|
echo "Backed up existing daemon.json to daemon.json.backup"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Apply new configuration
|
||||||
|
sudo cp /tmp/daemon.json.tmp /etc/docker/daemon.json
|
||||||
|
|
||||||
|
# Restart Docker
|
||||||
|
echo "Restarting Docker daemon..."
|
||||||
|
sudo systemctl restart docker
|
||||||
|
|
||||||
|
echo -e "${GREEN}Docker daemon configured and restarted${NC}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 4. Test registry cache
|
||||||
|
echo -e "\n${YELLOW}3. Testing Registry Cache...${NC}"
|
||||||
|
|
||||||
|
# Pull a test image through cache
|
||||||
|
echo "Pulling test image (alpine) through cache..."
|
||||||
|
docker pull alpine:latest
|
||||||
|
|
||||||
|
# Check if image is cached
|
||||||
|
echo -e "\nChecking cached images..."
|
||||||
|
curl -s http://localhost:5000/v2/_catalog | python3 -m json.tool || echo "No cached images yet"
|
||||||
|
|
||||||
|
# 5. Configure buildx for multi-platform builds with cache
|
||||||
|
echo -e "\n${YELLOW}4. Configuring Docker Buildx with cache...${NC}"
|
||||||
|
|
||||||
|
# Create buildx builder with registry cache
|
||||||
|
docker buildx create \
|
||||||
|
--name site11-builder \
|
||||||
|
--driver docker-container \
|
||||||
|
--config /dev/stdin <<EOF
|
||||||
|
[registry."localhost:5000"]
|
||||||
|
mirrors = ["localhost:5000"]
|
||||||
|
insecure = true
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# Use the new builder
|
||||||
|
docker buildx use site11-builder
|
||||||
|
|
||||||
|
# Bootstrap the builder
|
||||||
|
docker buildx inspect --bootstrap
|
||||||
|
|
||||||
|
echo -e "${GREEN}✓ Buildx configured with registry cache${NC}"
|
||||||
|
|
||||||
|
# 6. Setup build script with cache
|
||||||
|
echo -e "\n${YELLOW}5. Creating optimized build script...${NC}"
|
||||||
|
|
||||||
|
cat > scripts/build-with-cache.sh <<'SCRIPT'
|
||||||
|
#!/bin/bash
|
||||||
|
#
|
||||||
|
# Build script optimized for registry cache
|
||||||
|
#
|
||||||
|
|
||||||
|
SERVICE=$1
|
||||||
|
if [ -z "$SERVICE" ]; then
|
||||||
|
echo "Usage: $0 <service-name>"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Building $SERVICE with cache optimization..."
|
||||||
|
|
||||||
|
# Build with cache mount and registry cache
|
||||||
|
docker buildx build \
|
||||||
|
--cache-from type=registry,ref=localhost:5000/site11-$SERVICE:cache \
|
||||||
|
--cache-to type=registry,ref=localhost:5000/site11-$SERVICE:cache,mode=max \
|
||||||
|
--platform linux/amd64 \
|
||||||
|
--tag site11-$SERVICE:latest \
|
||||||
|
--tag localhost:5000/site11-$SERVICE:latest \
|
||||||
|
--push \
|
||||||
|
-f services/$SERVICE/Dockerfile \
|
||||||
|
services/$SERVICE
|
||||||
|
|
||||||
|
echo "Build complete for $SERVICE"
|
||||||
|
SCRIPT
|
||||||
|
|
||||||
|
chmod +x scripts/build-with-cache.sh
|
||||||
|
|
||||||
|
# 7. Create cache warming script
|
||||||
|
echo -e "\n${YELLOW}6. Creating cache warming script...${NC}"
|
||||||
|
|
||||||
|
cat > scripts/warm-cache.sh <<'WARMSCRIPT'
|
||||||
|
#!/bin/bash
|
||||||
|
#
|
||||||
|
# Warm up registry cache with commonly used base images
|
||||||
|
#
|
||||||
|
|
||||||
|
echo "Warming up registry cache..."
|
||||||
|
|
||||||
|
# Base images used in the project
|
||||||
|
IMAGES=(
|
||||||
|
"python:3.11-slim"
|
||||||
|
"node:18-alpine"
|
||||||
|
"nginx:alpine"
|
||||||
|
"redis:7-alpine"
|
||||||
|
"mongo:7.0"
|
||||||
|
"zookeeper:3.9"
|
||||||
|
"bitnami/kafka:3.5"
|
||||||
|
)
|
||||||
|
|
||||||
|
for image in "${IMAGES[@]}"; do
|
||||||
|
echo "Caching $image..."
|
||||||
|
docker pull "$image"
|
||||||
|
docker tag "$image" "localhost:5000/$image"
|
||||||
|
docker push "localhost:5000/$image"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "Cache warming complete!"
|
||||||
|
WARMSCRIPT
|
||||||
|
|
||||||
|
chmod +x scripts/warm-cache.sh
|
||||||
|
|
||||||
|
# 8. Create registry management script
|
||||||
|
echo -e "\n${YELLOW}7. Creating registry management script...${NC}"
|
||||||
|
|
||||||
|
cat > scripts/manage-registry.sh <<'MANAGE'
|
||||||
|
#!/bin/bash
|
||||||
|
#
|
||||||
|
# Registry cache management utilities
|
||||||
|
#
|
||||||
|
|
||||||
|
case "$1" in
|
||||||
|
status)
|
||||||
|
echo "Registry Cache Status:"
|
||||||
|
curl -s http://localhost:5000/v2/_catalog | python3 -m json.tool
|
||||||
|
;;
|
||||||
|
size)
|
||||||
|
echo "Registry Cache Size:"
|
||||||
|
docker exec site11_registry_cache du -sh /var/lib/registry
|
||||||
|
;;
|
||||||
|
clean)
|
||||||
|
echo "Running garbage collection..."
|
||||||
|
docker exec site11_registry_cache registry garbage-collect /etc/docker/registry/config.yml
|
||||||
|
;;
|
||||||
|
logs)
|
||||||
|
docker logs -f site11_registry_cache
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Usage: $0 {status|size|clean|logs}"
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
MANAGE
|
||||||
|
|
||||||
|
chmod +x scripts/manage-registry.sh
|
||||||
|
|
||||||
|
# 9. Summary
|
||||||
|
echo -e "\n${GREEN}========================================${NC}"
|
||||||
|
echo -e "${GREEN}Registry Cache Setup Complete!${NC}"
|
||||||
|
echo -e "${GREEN}========================================${NC}"
|
||||||
|
|
||||||
|
echo -e "\n${YELLOW}Available commands:${NC}"
|
||||||
|
echo " - scripts/build-with-cache.sh <service> # Build with cache"
|
||||||
|
echo " - scripts/warm-cache.sh # Pre-cache base images"
|
||||||
|
echo " - scripts/manage-registry.sh status # Check cache status"
|
||||||
|
echo " - scripts/manage-registry.sh size # Check cache size"
|
||||||
|
echo " - scripts/manage-registry.sh clean # Clean cache"
|
||||||
|
|
||||||
|
echo -e "\n${YELLOW}Registry endpoints:${NC}"
|
||||||
|
echo " - Registry: http://localhost:5000"
|
||||||
|
echo " - Catalog: http://localhost:5000/v2/_catalog"
|
||||||
|
echo " - Health: http://localhost:5000/v2/"
|
||||||
|
|
||||||
|
echo -e "\n${YELLOW}Next steps:${NC}"
|
||||||
|
echo "1. Run './scripts/warm-cache.sh' to pre-cache base images"
|
||||||
|
echo "2. Use './scripts/build-with-cache.sh <service>' for faster builds"
|
||||||
|
echo "3. Monitor cache with './scripts/manage-registry.sh status'"
|
||||||
|
|
||||||
|
# Optional: Warm cache immediately
|
||||||
|
read -p "Would you like to warm the cache now? (y/n) " -n 1 -r
|
||||||
|
echo
|
||||||
|
if [[ $REPLY =~ ^[Yy]$ ]]; then
|
||||||
|
./scripts/warm-cache.sh
|
||||||
|
fi
|
||||||
91
scripts/start-k8s-port-forward.sh
Executable file
91
scripts/start-k8s-port-forward.sh
Executable file
@ -0,0 +1,91 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
#
|
||||||
|
# Kubernetes Port Forwarding Setup Script
|
||||||
|
# Sets up port forwarding for accessing K8s services locally
|
||||||
|
#
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
RED='\033[0;31m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
echo -e "${GREEN}========================================${NC}"
|
||||||
|
echo -e "${GREEN}Starting K8s Port Forwarding${NC}"
|
||||||
|
echo -e "${GREEN}========================================${NC}"
|
||||||
|
|
||||||
|
# Function to stop existing port forwards
|
||||||
|
stop_existing_forwards() {
|
||||||
|
echo -e "${YELLOW}Stopping existing port forwards...${NC}"
|
||||||
|
pkill -f "kubectl.*port-forward" 2>/dev/null || true
|
||||||
|
sleep 2
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to start port forward
|
||||||
|
start_port_forward() {
|
||||||
|
local service=$1
|
||||||
|
local local_port=$2
|
||||||
|
local service_port=$3
|
||||||
|
|
||||||
|
echo -e "Starting port forward: ${GREEN}$service${NC} (localhost:$local_port → service:$service_port)"
|
||||||
|
kubectl -n site11-pipeline port-forward service/$service $local_port:$service_port &
|
||||||
|
|
||||||
|
# Wait a moment for the port forward to establish
|
||||||
|
sleep 2
|
||||||
|
|
||||||
|
# Check if port forward is working
|
||||||
|
if lsof -i :$local_port | grep -q LISTEN; then
|
||||||
|
echo -e " ${GREEN}✓${NC} Port forward established on localhost:$local_port"
|
||||||
|
else
|
||||||
|
echo -e " ${RED}✗${NC} Failed to establish port forward on localhost:$local_port"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Stop existing forwards first
|
||||||
|
stop_existing_forwards
|
||||||
|
|
||||||
|
# Start port forwards
|
||||||
|
echo -e "\n${YELLOW}Starting port forwards...${NC}\n"
|
||||||
|
|
||||||
|
# Console Frontend
|
||||||
|
start_port_forward "console-frontend" 8080 3000
|
||||||
|
|
||||||
|
# Console Backend
|
||||||
|
start_port_forward "console-backend" 8000 8000
|
||||||
|
|
||||||
|
# Summary
|
||||||
|
echo -e "\n${GREEN}========================================${NC}"
|
||||||
|
echo -e "${GREEN}Port Forwarding Active!${NC}"
|
||||||
|
echo -e "${GREEN}========================================${NC}"
|
||||||
|
|
||||||
|
echo -e "\n${YELLOW}Available endpoints:${NC}"
|
||||||
|
echo -e " Console Frontend: ${GREEN}http://localhost:8080${NC}"
|
||||||
|
echo -e " Console Backend: ${GREEN}http://localhost:8000${NC}"
|
||||||
|
echo -e " Health Check: ${GREEN}http://localhost:8000/health${NC}"
|
||||||
|
echo -e " API Health: ${GREEN}http://localhost:8000/api/health${NC}"
|
||||||
|
|
||||||
|
echo -e "\n${YELLOW}To stop port forwarding:${NC}"
|
||||||
|
echo -e " pkill -f 'kubectl.*port-forward'"
|
||||||
|
|
||||||
|
echo -e "\n${YELLOW}To check status:${NC}"
|
||||||
|
echo -e " ps aux | grep 'kubectl.*port-forward'"
|
||||||
|
|
||||||
|
# Keep script running
|
||||||
|
echo -e "\n${YELLOW}Port forwarding is running in background.${NC}"
|
||||||
|
echo -e "Press Ctrl+C to stop all port forwards..."
|
||||||
|
|
||||||
|
# Trap to clean up on exit
|
||||||
|
trap "echo -e '\n${YELLOW}Stopping port forwards...${NC}'; pkill -f 'kubectl.*port-forward'; exit" INT TERM
|
||||||
|
|
||||||
|
# Keep the script running
|
||||||
|
while true; do
|
||||||
|
sleep 60
|
||||||
|
# Check if port forwards are still running
|
||||||
|
if ! pgrep -f "kubectl.*port-forward" > /dev/null; then
|
||||||
|
echo -e "${RED}Port forwards stopped unexpectedly. Restarting...${NC}"
|
||||||
|
start_port_forward "console-frontend" 8080 3000
|
||||||
|
start_port_forward "console-backend" 8000 8000
|
||||||
|
fi
|
||||||
|
done
|
||||||
247
scripts/status-check.sh
Executable file
247
scripts/status-check.sh
Executable file
@ -0,0 +1,247 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
#
|
||||||
|
# Site11 System Status Check Script
|
||||||
|
# Comprehensive status check for both Docker and Kubernetes services
|
||||||
|
#
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
RED='\033[0;31m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
echo -e "${BLUE}========================================${NC}"
|
||||||
|
echo -e "${BLUE}Site11 System Status Check${NC}"
|
||||||
|
echo -e "${BLUE}========================================${NC}"
|
||||||
|
|
||||||
|
# Function to check service status
|
||||||
|
check_url() {
|
||||||
|
local url=$1
|
||||||
|
local name=$2
|
||||||
|
local timeout=${3:-5}
|
||||||
|
|
||||||
|
if curl -s --max-time $timeout "$url" > /dev/null 2>&1; then
|
||||||
|
echo -e " ${GREEN}✓${NC} $name: $url"
|
||||||
|
return 0
|
||||||
|
else
|
||||||
|
echo -e " ${RED}✗${NC} $name: $url"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to check Docker service
|
||||||
|
check_docker_service() {
|
||||||
|
local service=$1
|
||||||
|
if docker ps --format "table {{.Names}}" | grep -q "$service"; then
|
||||||
|
echo -e " ${GREEN}✓${NC} $service"
|
||||||
|
return 0
|
||||||
|
else
|
||||||
|
echo -e " ${RED}✗${NC} $service"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to check Kubernetes deployment
|
||||||
|
check_k8s_deployment() {
|
||||||
|
local deployment=$1
|
||||||
|
local namespace=${2:-site11-pipeline}
|
||||||
|
|
||||||
|
if kubectl -n "$namespace" get deployment "$deployment" >/dev/null 2>&1; then
|
||||||
|
local ready=$(kubectl -n "$namespace" get deployment "$deployment" -o jsonpath='{.status.readyReplicas}')
|
||||||
|
local desired=$(kubectl -n "$namespace" get deployment "$deployment" -o jsonpath='{.spec.replicas}')
|
||||||
|
|
||||||
|
if [ "$ready" = "$desired" ] && [ "$ready" != "" ]; then
|
||||||
|
echo -e " ${GREEN}✓${NC} $deployment ($ready/$desired ready)"
|
||||||
|
return 0
|
||||||
|
else
|
||||||
|
echo -e " ${YELLOW}⚠${NC} $deployment ($ready/$desired ready)"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo -e " ${RED}✗${NC} $deployment (not found)"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# 1. Docker Infrastructure Services
|
||||||
|
echo -e "\n${YELLOW}1. Docker Infrastructure Services${NC}"
|
||||||
|
docker_services=(
|
||||||
|
"site11_mongodb"
|
||||||
|
"site11_redis"
|
||||||
|
"site11_kafka"
|
||||||
|
"site11_zookeeper"
|
||||||
|
"site11_pipeline_scheduler"
|
||||||
|
"site11_pipeline_monitor"
|
||||||
|
"site11_language_sync"
|
||||||
|
)
|
||||||
|
|
||||||
|
docker_healthy=0
|
||||||
|
for service in "${docker_services[@]}"; do
|
||||||
|
if check_docker_service "$service"; then
|
||||||
|
((docker_healthy++))
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
echo -e "Docker Services: ${GREEN}$docker_healthy${NC}/${#docker_services[@]} healthy"
|
||||||
|
|
||||||
|
# 2. Kubernetes Application Services
|
||||||
|
echo -e "\n${YELLOW}2. Kubernetes Application Services${NC}"
|
||||||
|
k8s_deployments=(
|
||||||
|
"console-backend"
|
||||||
|
"console-frontend"
|
||||||
|
"pipeline-rss-collector"
|
||||||
|
"pipeline-google-search"
|
||||||
|
"pipeline-translator"
|
||||||
|
"pipeline-ai-article-generator"
|
||||||
|
"pipeline-image-generator"
|
||||||
|
)
|
||||||
|
|
||||||
|
k8s_healthy=0
|
||||||
|
if kubectl cluster-info >/dev/null 2>&1; then
|
||||||
|
for deployment in "${k8s_deployments[@]}"; do
|
||||||
|
if check_k8s_deployment "$deployment"; then
|
||||||
|
((k8s_healthy++))
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
echo -e "Kubernetes Services: ${GREEN}$k8s_healthy${NC}/${#k8s_deployments[@]} healthy"
|
||||||
|
else
|
||||||
|
echo -e " ${RED}✗${NC} Kubernetes cluster not accessible"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 3. Health Check Endpoints
|
||||||
|
echo -e "\n${YELLOW}3. Health Check Endpoints${NC}"
|
||||||
|
health_endpoints=(
|
||||||
|
"http://localhost:8000/health|Console Backend"
|
||||||
|
"http://localhost:8000/api/health|Console API Health"
|
||||||
|
"http://localhost:8000/api/users/health|Users Service"
|
||||||
|
"http://localhost:8080/|Console Frontend"
|
||||||
|
"http://localhost:8100/health|Pipeline Monitor"
|
||||||
|
"http://localhost:8099/health|Pipeline Scheduler"
|
||||||
|
)
|
||||||
|
|
||||||
|
health_count=0
|
||||||
|
for endpoint in "${health_endpoints[@]}"; do
|
||||||
|
IFS='|' read -r url name <<< "$endpoint"
|
||||||
|
if check_url "$url" "$name"; then
|
||||||
|
((health_count++))
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
echo -e "Health Endpoints: ${GREEN}$health_count${NC}/${#health_endpoints[@]} accessible"
|
||||||
|
|
||||||
|
# 4. Port Forward Status
|
||||||
|
echo -e "\n${YELLOW}4. Port Forward Status${NC}"
|
||||||
|
port_forwards=()
|
||||||
|
while IFS= read -r line; do
|
||||||
|
if [[ $line == *"kubectl"* && $line == *"port-forward"* ]]; then
|
||||||
|
# Extract port from the command
|
||||||
|
if [[ $line =~ ([0-9]+):([0-9]+) ]]; then
|
||||||
|
local_port="${BASH_REMATCH[1]}"
|
||||||
|
service_port="${BASH_REMATCH[2]}"
|
||||||
|
service_name=$(echo "$line" | grep -o 'service/[^ ]*' | cut -d'/' -f2)
|
||||||
|
port_forwards+=("$local_port:$service_port|$service_name")
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
done < <(ps aux | grep "kubectl.*port-forward" | grep -v grep)
|
||||||
|
|
||||||
|
if [ ${#port_forwards[@]} -eq 0 ]; then
|
||||||
|
echo -e " ${RED}✗${NC} No port forwards running"
|
||||||
|
echo -e " ${YELLOW}ℹ${NC} Run: ./scripts/start-k8s-port-forward.sh"
|
||||||
|
else
|
||||||
|
for pf in "${port_forwards[@]}"; do
|
||||||
|
IFS='|' read -r ports service <<< "$pf"
|
||||||
|
echo -e " ${GREEN}✓${NC} $service: localhost:$ports"
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 5. Resource Usage
|
||||||
|
echo -e "\n${YELLOW}5. Resource Usage${NC}"
|
||||||
|
|
||||||
|
# Docker resource usage
|
||||||
|
if command -v docker &> /dev/null; then
|
||||||
|
docker_containers=$(docker ps --filter "name=site11_" --format "table {{.Names}}" | wc -l)
|
||||||
|
echo -e " Docker Containers: ${GREEN}$docker_containers${NC} running"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Kubernetes resource usage
|
||||||
|
if kubectl cluster-info >/dev/null 2>&1; then
|
||||||
|
k8s_pods=$(kubectl -n site11-pipeline get pods --no-headers 2>/dev/null | wc -l)
|
||||||
|
k8s_running=$(kubectl -n site11-pipeline get pods --no-headers 2>/dev/null | grep -c "Running" || echo "0")
|
||||||
|
echo -e " Kubernetes Pods: ${GREEN}$k8s_running${NC}/$k8s_pods running"
|
||||||
|
|
||||||
|
# HPA status
|
||||||
|
if kubectl -n site11-pipeline get hpa >/dev/null 2>&1; then
|
||||||
|
hpa_count=$(kubectl -n site11-pipeline get hpa --no-headers 2>/dev/null | wc -l)
|
||||||
|
echo -e " HPA Controllers: ${GREEN}$hpa_count${NC} active"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 6. Queue Status (Redis)
|
||||||
|
echo -e "\n${YELLOW}6. Queue Status${NC}"
|
||||||
|
if check_docker_service "site11_redis"; then
|
||||||
|
queues=(
|
||||||
|
"queue:rss_collection"
|
||||||
|
"queue:google_search"
|
||||||
|
"queue:ai_generation"
|
||||||
|
"queue:translation"
|
||||||
|
"queue:image_generation"
|
||||||
|
)
|
||||||
|
|
||||||
|
for queue in "${queues[@]}"; do
|
||||||
|
length=$(docker exec site11_redis redis-cli LLEN "$queue" 2>/dev/null || echo "0")
|
||||||
|
if [ "$length" -gt 0 ]; then
|
||||||
|
echo -e " ${YELLOW}⚠${NC} $queue: $length items"
|
||||||
|
else
|
||||||
|
echo -e " ${GREEN}✓${NC} $queue: empty"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
else
|
||||||
|
echo -e " ${RED}✗${NC} Redis not available"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 7. Database Status
|
||||||
|
echo -e "\n${YELLOW}7. Database Status${NC}"
|
||||||
|
if check_docker_service "site11_mongodb"; then
|
||||||
|
# Check MongoDB collections
|
||||||
|
collections=$(docker exec site11_mongodb mongosh ai_writer_db --quiet --eval "db.getCollectionNames()" 2>/dev/null | grep -o '"articles_[^"]*"' | wc -l || echo "0")
|
||||||
|
echo -e " ${GREEN}✓${NC} MongoDB: $collections collections"
|
||||||
|
|
||||||
|
# Check article counts
|
||||||
|
ko_count=$(docker exec site11_mongodb mongosh ai_writer_db --quiet --eval "db.articles_ko.countDocuments({})" 2>/dev/null || echo "0")
|
||||||
|
echo -e " ${GREEN}✓${NC} Korean articles: $ko_count"
|
||||||
|
else
|
||||||
|
echo -e " ${RED}✗${NC} MongoDB not available"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 8. Summary
|
||||||
|
echo -e "\n${BLUE}========================================${NC}"
|
||||||
|
echo -e "${BLUE}Summary${NC}"
|
||||||
|
echo -e "${BLUE}========================================${NC}"
|
||||||
|
|
||||||
|
total_services=$((${#docker_services[@]} + ${#k8s_deployments[@]}))
|
||||||
|
total_healthy=$((docker_healthy + k8s_healthy))
|
||||||
|
|
||||||
|
if [ $total_healthy -eq $total_services ] && [ $health_count -eq ${#health_endpoints[@]} ]; then
|
||||||
|
echo -e "${GREEN}✓ All systems operational${NC}"
|
||||||
|
echo -e " Services: $total_healthy/$total_services"
|
||||||
|
echo -e " Health checks: $health_count/${#health_endpoints[@]}"
|
||||||
|
exit 0
|
||||||
|
elif [ $total_healthy -gt $((total_services / 2)) ]; then
|
||||||
|
echo -e "${YELLOW}⚠ System partially operational${NC}"
|
||||||
|
echo -e " Services: $total_healthy/$total_services"
|
||||||
|
echo -e " Health checks: $health_count/${#health_endpoints[@]}"
|
||||||
|
exit 1
|
||||||
|
else
|
||||||
|
echo -e "${RED}✗ System issues detected${NC}"
|
||||||
|
echo -e " Services: $total_healthy/$total_services"
|
||||||
|
echo -e " Health checks: $health_count/${#health_endpoints[@]}"
|
||||||
|
echo -e "\n${YELLOW}Troubleshooting:${NC}"
|
||||||
|
echo -e " 1. Check Docker: docker-compose -f docker-compose-hybrid.yml ps"
|
||||||
|
echo -e " 2. Check Kubernetes: kubectl -n site11-pipeline get pods"
|
||||||
|
echo -e " 3. Check port forwards: ./scripts/start-k8s-port-forward.sh"
|
||||||
|
echo -e " 4. Check logs: docker-compose -f docker-compose-hybrid.yml logs"
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
276
services/console/PHASE1_COMPLETION.md
Normal file
276
services/console/PHASE1_COMPLETION.md
Normal file
@ -0,0 +1,276 @@
|
|||||||
|
# Phase 1: Authentication System - Completion Report
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
Phase 1 of the Site11 Console project has been successfully completed. This phase establishes a complete authentication system with JWT token-based security for both backend and frontend.
|
||||||
|
|
||||||
|
**Completion Date**: October 28, 2025
|
||||||
|
|
||||||
|
## What Was Built
|
||||||
|
|
||||||
|
### 1. Backend Authentication API (FastAPI + MongoDB)
|
||||||
|
|
||||||
|
#### Core Features
|
||||||
|
- **User Registration**: Create new users with email, username, and password
|
||||||
|
- **User Login**: Authenticate users and issue JWT tokens
|
||||||
|
- **Token Management**: Access tokens (30 min) and refresh tokens (7 days)
|
||||||
|
- **Protected Endpoints**: JWT middleware for secure routes
|
||||||
|
- **Password Security**: bcrypt hashing with proper salt handling
|
||||||
|
- **Role-Based Access Control (RBAC)**: User roles (admin, editor, viewer)
|
||||||
|
|
||||||
|
#### Technology Stack
|
||||||
|
- FastAPI 0.109.0
|
||||||
|
- MongoDB with Motor (async driver)
|
||||||
|
- Pydantic v2 for data validation
|
||||||
|
- python-jose for JWT
|
||||||
|
- bcrypt 4.1.2 for password hashing
|
||||||
|
|
||||||
|
#### API Endpoints
|
||||||
|
| Method | Endpoint | Description | Auth Required |
|
||||||
|
|--------|----------|-------------|---------------|
|
||||||
|
| POST | `/api/auth/register` | Register new user | No |
|
||||||
|
| POST | `/api/auth/login` | Login and get tokens | No |
|
||||||
|
| GET | `/api/auth/me` | Get current user info | Yes |
|
||||||
|
| POST | `/api/auth/refresh` | Refresh access token | Yes (refresh token) |
|
||||||
|
| POST | `/api/auth/logout` | Logout user | Yes |
|
||||||
|
|
||||||
|
#### File Structure
|
||||||
|
```
|
||||||
|
services/console/backend/
|
||||||
|
├── app/
|
||||||
|
│ ├── core/
|
||||||
|
│ │ ├── config.py # Application settings
|
||||||
|
│ │ └── security.py # JWT & password hashing
|
||||||
|
│ ├── db/
|
||||||
|
│ │ └── mongodb.py # MongoDB connection
|
||||||
|
│ ├── models/
|
||||||
|
│ │ └── user.py # User data model
|
||||||
|
│ ├── schemas/
|
||||||
|
│ │ └── auth.py # Request/response schemas
|
||||||
|
│ ├── services/
|
||||||
|
│ │ └── user_service.py # Business logic
|
||||||
|
│ ├── routes/
|
||||||
|
│ │ └── auth.py # API endpoints
|
||||||
|
│ └── main.py # Application entry point
|
||||||
|
├── Dockerfile
|
||||||
|
└── requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Frontend Authentication UI (React + TypeScript)
|
||||||
|
|
||||||
|
#### Core Features
|
||||||
|
- **Login Page**: Material-UI form with validation
|
||||||
|
- **Register Page**: User creation with password confirmation
|
||||||
|
- **Auth Context**: Global authentication state management
|
||||||
|
- **Protected Routes**: Redirect unauthenticated users to login
|
||||||
|
- **Automatic Token Refresh**: Intercept 401 and refresh tokens
|
||||||
|
- **User Profile Display**: Show username and role in navigation
|
||||||
|
- **Logout Functionality**: Clear tokens and redirect to login
|
||||||
|
|
||||||
|
#### Technology Stack
|
||||||
|
- React 18.2.0
|
||||||
|
- TypeScript 5.2.2
|
||||||
|
- Material-UI v5
|
||||||
|
- React Router v6
|
||||||
|
- Axios for HTTP requests
|
||||||
|
- Vite for building
|
||||||
|
|
||||||
|
#### Component Structure
|
||||||
|
```
|
||||||
|
services/console/frontend/src/
|
||||||
|
├── types/
|
||||||
|
│ └── auth.ts # TypeScript interfaces
|
||||||
|
├── api/
|
||||||
|
│ └── auth.ts # API client with interceptors
|
||||||
|
├── contexts/
|
||||||
|
│ └── AuthContext.tsx # Global auth state
|
||||||
|
├── components/
|
||||||
|
│ ├── Layout.tsx # Main layout with nav
|
||||||
|
│ └── ProtectedRoute.tsx # Route guard component
|
||||||
|
├── pages/
|
||||||
|
│ ├── Login.tsx # Login page
|
||||||
|
│ ├── Register.tsx # Registration page
|
||||||
|
│ ├── Dashboard.tsx # Main dashboard (protected)
|
||||||
|
│ ├── Services.tsx # Services page (protected)
|
||||||
|
│ └── Users.tsx # Users page (protected)
|
||||||
|
├── App.tsx # Router configuration
|
||||||
|
└── main.tsx # Application entry point
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Deployment Configuration
|
||||||
|
|
||||||
|
#### Docker Images
|
||||||
|
Both services are containerized and pushed to Docker Hub:
|
||||||
|
- **Backend**: `yakenator/site11-console-backend:latest`
|
||||||
|
- **Frontend**: `yakenator/site11-console-frontend:latest`
|
||||||
|
|
||||||
|
#### Kubernetes Deployment
|
||||||
|
Deployed to `site11-pipeline` namespace with:
|
||||||
|
- 2 replicas for each service (backend and frontend)
|
||||||
|
- Service discovery via Kubernetes Services
|
||||||
|
- Nginx reverse proxy for frontend API routing
|
||||||
|
|
||||||
|
## Technical Challenges & Solutions
|
||||||
|
|
||||||
|
### Challenge 1: Bcrypt Password Length Limit
|
||||||
|
**Problem**: `passlib` threw error "password cannot be longer than 72 bytes"
|
||||||
|
|
||||||
|
**Solution**: Replaced `passlib[bcrypt]` with native `bcrypt==4.1.2` library
|
||||||
|
```python
|
||||||
|
import bcrypt
|
||||||
|
|
||||||
|
def get_password_hash(password: str) -> str:
|
||||||
|
password_bytes = password.encode('utf-8')
|
||||||
|
salt = bcrypt.gensalt()
|
||||||
|
return bcrypt.hashpw(password_bytes, salt).decode('utf-8')
|
||||||
|
```
|
||||||
|
|
||||||
|
### Challenge 2: Pydantic v2 Compatibility
|
||||||
|
**Problem**: `__modify_schema__` method not supported in Pydantic v2
|
||||||
|
|
||||||
|
**Solution**: Updated to Pydantic v2 patterns:
|
||||||
|
- Changed `__modify_schema__` to `__get_pydantic_core_schema__`
|
||||||
|
- Replaced `class Config` with `model_config = ConfigDict(...)`
|
||||||
|
- Updated all models to use new Pydantic v2 syntax
|
||||||
|
|
||||||
|
### Challenge 3: TypeScript Import.meta.env Types
|
||||||
|
**Problem**: TypeScript couldn't recognize `import.meta.env.VITE_API_URL`
|
||||||
|
|
||||||
|
**Solution**: Created `vite-env.d.ts` with proper type declarations:
|
||||||
|
```typescript
|
||||||
|
interface ImportMetaEnv {
|
||||||
|
readonly VITE_API_URL?: string
|
||||||
|
}
|
||||||
|
interface ImportMeta {
|
||||||
|
readonly env: ImportMetaEnv
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing Results
|
||||||
|
|
||||||
|
### Backend API Tests (via curl)
|
||||||
|
All endpoints tested and working correctly:
|
||||||
|
|
||||||
|
✅ **User Registration**
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/auth/register \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"email":"test@site11.com","username":"testuser","password":"test123"}'
|
||||||
|
# Returns: User object with _id, email, username, role
|
||||||
|
```
|
||||||
|
|
||||||
|
✅ **User Login**
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/auth/login \
|
||||||
|
-d "username=testuser&password=test123"
|
||||||
|
# Returns: access_token, refresh_token, token_type
|
||||||
|
```
|
||||||
|
|
||||||
|
✅ **Protected Endpoint**
|
||||||
|
```bash
|
||||||
|
curl -X GET http://localhost:8000/api/auth/me \
|
||||||
|
-H "Authorization: Bearer <access_token>"
|
||||||
|
# Returns: Current user details with last_login_at
|
||||||
|
```
|
||||||
|
|
||||||
|
✅ **Token Refresh**
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/auth/refresh \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"refresh_token":"<refresh_token>"}'
|
||||||
|
# Returns: New access_token and same refresh_token
|
||||||
|
```
|
||||||
|
|
||||||
|
✅ **Security Validations**
|
||||||
|
- Wrong password → "Incorrect username/email or password"
|
||||||
|
- No token → "Not authenticated"
|
||||||
|
- Duplicate email → "Email already registered"
|
||||||
|
|
||||||
|
### Frontend Tests
|
||||||
|
✅ Login page renders correctly
|
||||||
|
✅ Registration form with validation
|
||||||
|
✅ Protected routes redirect to login
|
||||||
|
✅ User info displayed in navigation bar
|
||||||
|
✅ Logout clears session and redirects
|
||||||
|
|
||||||
|
## Deployment Instructions
|
||||||
|
|
||||||
|
### Build Docker Images
|
||||||
|
```bash
|
||||||
|
# Backend
|
||||||
|
cd services/console/backend
|
||||||
|
docker build -t yakenator/site11-console-backend:latest .
|
||||||
|
docker push yakenator/site11-console-backend:latest
|
||||||
|
|
||||||
|
# Frontend
|
||||||
|
cd services/console/frontend
|
||||||
|
docker build -t yakenator/site11-console-frontend:latest .
|
||||||
|
docker push yakenator/site11-console-frontend:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
### Deploy to Kubernetes
|
||||||
|
```bash
|
||||||
|
# Delete old pods to pull new images
|
||||||
|
kubectl -n site11-pipeline delete pod -l app=console-backend
|
||||||
|
kubectl -n site11-pipeline delete pod -l app=console-frontend
|
||||||
|
|
||||||
|
# Wait for new pods to start
|
||||||
|
kubectl -n site11-pipeline get pods -w
|
||||||
|
```
|
||||||
|
|
||||||
|
### Local Access (Port Forwarding)
|
||||||
|
```bash
|
||||||
|
# Backend
|
||||||
|
kubectl -n site11-pipeline port-forward svc/console-backend 8000:8000 &
|
||||||
|
|
||||||
|
# Frontend
|
||||||
|
kubectl -n site11-pipeline port-forward svc/console-frontend 3000:80 &
|
||||||
|
|
||||||
|
# Access
|
||||||
|
open http://localhost:3000
|
||||||
|
```
|
||||||
|
|
||||||
|
## Next Steps (Phase 2)
|
||||||
|
|
||||||
|
### Service Management CRUD
|
||||||
|
1. **Backend**:
|
||||||
|
- Service model (name, url, status, health_endpoint, last_check)
|
||||||
|
- CRUD API endpoints
|
||||||
|
- Health check scheduler
|
||||||
|
- Service registry
|
||||||
|
|
||||||
|
2. **Frontend**:
|
||||||
|
- Services list page with table
|
||||||
|
- Add/Edit service modal
|
||||||
|
- Service status indicators
|
||||||
|
- Health monitoring dashboard
|
||||||
|
|
||||||
|
3. **Features**:
|
||||||
|
- Auto-discovery of services
|
||||||
|
- Periodic health checks
|
||||||
|
- Service availability statistics
|
||||||
|
- Alert notifications
|
||||||
|
|
||||||
|
## Success Metrics
|
||||||
|
|
||||||
|
✅ All authentication endpoints functional
|
||||||
|
✅ JWT tokens working correctly
|
||||||
|
✅ Token refresh implemented
|
||||||
|
✅ Frontend login/register flows complete
|
||||||
|
✅ Protected routes working
|
||||||
|
✅ Docker images built and pushed
|
||||||
|
✅ Deployed to Kubernetes successfully
|
||||||
|
✅ All tests passing
|
||||||
|
✅ Documentation complete
|
||||||
|
|
||||||
|
## Team Notes
|
||||||
|
- Code follows FastAPI and React best practices
|
||||||
|
- All secrets managed via environment variables
|
||||||
|
- Proper error handling implemented
|
||||||
|
- API endpoints follow RESTful conventions
|
||||||
|
- Frontend components are reusable and well-structured
|
||||||
|
- TypeScript types ensure type safety
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Phase 1 Status**: ✅ **COMPLETE**
|
||||||
|
**Ready for**: Phase 2 - Service Management CRUD
|
||||||
33
services/console/backend/.env.example
Normal file
33
services/console/backend/.env.example
Normal file
@ -0,0 +1,33 @@
|
|||||||
|
# App Settings
|
||||||
|
APP_NAME=Site11 Console
|
||||||
|
APP_VERSION=1.0.0
|
||||||
|
DEBUG=True
|
||||||
|
|
||||||
|
# Security
|
||||||
|
SECRET_KEY=your-secret-key-change-in-production-use-openssl-rand-hex-32
|
||||||
|
ALGORITHM=HS256
|
||||||
|
ACCESS_TOKEN_EXPIRE_MINUTES=30
|
||||||
|
REFRESH_TOKEN_EXPIRE_DAYS=7
|
||||||
|
|
||||||
|
# Database
|
||||||
|
MONGODB_URL=mongodb://localhost:27017
|
||||||
|
DB_NAME=site11_console
|
||||||
|
|
||||||
|
# Redis
|
||||||
|
REDIS_URL=redis://localhost:6379
|
||||||
|
|
||||||
|
# CORS
|
||||||
|
CORS_ORIGINS=["http://localhost:3000","http://localhost:8000"]
|
||||||
|
|
||||||
|
# Services
|
||||||
|
USERS_SERVICE_URL=http://users-backend:8000
|
||||||
|
IMAGES_SERVICE_URL=http://images-backend:8000
|
||||||
|
|
||||||
|
# Kafka (optional)
|
||||||
|
KAFKA_BOOTSTRAP_SERVERS=kafka:9092
|
||||||
|
|
||||||
|
# OAuth (optional - for Phase 1.5)
|
||||||
|
GOOGLE_CLIENT_ID=
|
||||||
|
GOOGLE_CLIENT_SECRET=
|
||||||
|
GITHUB_CLIENT_ID=
|
||||||
|
GITHUB_CLIENT_SECRET=
|
||||||
@ -17,5 +17,9 @@ COPY . .
|
|||||||
# Expose port
|
# Expose port
|
||||||
EXPOSE 8000
|
EXPOSE 8000
|
||||||
|
|
||||||
|
# Environment variables
|
||||||
|
ENV PYTHONUNBUFFERED=1
|
||||||
|
ENV PYTHONPATH=/app
|
||||||
|
|
||||||
# Run the application
|
# Run the application
|
||||||
CMD ["python", "main.py"]
|
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||||
0
services/console/backend/app/__init__.py
Normal file
0
services/console/backend/app/__init__.py
Normal file
0
services/console/backend/app/core/__init__.py
Normal file
0
services/console/backend/app/core/__init__.py
Normal file
47
services/console/backend/app/core/config.py
Normal file
47
services/console/backend/app/core/config.py
Normal file
@ -0,0 +1,47 @@
|
|||||||
|
from pydantic_settings import BaseSettings
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
|
||||||
|
class Settings(BaseSettings):
|
||||||
|
"""Application settings"""
|
||||||
|
|
||||||
|
# App
|
||||||
|
APP_NAME: str = "Site11 Console"
|
||||||
|
APP_VERSION: str = "1.0.0"
|
||||||
|
DEBUG: bool = False
|
||||||
|
|
||||||
|
# Security
|
||||||
|
SECRET_KEY: str = "your-secret-key-change-in-production"
|
||||||
|
ALGORITHM: str = "HS256"
|
||||||
|
ACCESS_TOKEN_EXPIRE_MINUTES: int = 30
|
||||||
|
REFRESH_TOKEN_EXPIRE_DAYS: int = 7
|
||||||
|
|
||||||
|
# Database
|
||||||
|
MONGODB_URL: str = "mongodb://localhost:27017"
|
||||||
|
DB_NAME: str = "site11_console"
|
||||||
|
|
||||||
|
# Redis
|
||||||
|
REDIS_URL: str = "redis://localhost:6379"
|
||||||
|
|
||||||
|
# CORS
|
||||||
|
CORS_ORIGINS: list = ["http://localhost:3000", "http://localhost:8000"]
|
||||||
|
|
||||||
|
# OAuth (Google, GitHub, etc.)
|
||||||
|
GOOGLE_CLIENT_ID: Optional[str] = None
|
||||||
|
GOOGLE_CLIENT_SECRET: Optional[str] = None
|
||||||
|
GITHUB_CLIENT_ID: Optional[str] = None
|
||||||
|
GITHUB_CLIENT_SECRET: Optional[str] = None
|
||||||
|
|
||||||
|
# Services URLs
|
||||||
|
USERS_SERVICE_URL: str = "http://users-backend:8000"
|
||||||
|
IMAGES_SERVICE_URL: str = "http://images-backend:8000"
|
||||||
|
|
||||||
|
# Kafka (optional)
|
||||||
|
KAFKA_BOOTSTRAP_SERVERS: str = "kafka:9092"
|
||||||
|
|
||||||
|
class Config:
|
||||||
|
env_file = ".env"
|
||||||
|
case_sensitive = True
|
||||||
|
|
||||||
|
|
||||||
|
settings = Settings()
|
||||||
78
services/console/backend/app/core/security.py
Normal file
78
services/console/backend/app/core/security.py
Normal file
@ -0,0 +1,78 @@
|
|||||||
|
from datetime import datetime, timedelta
|
||||||
|
from typing import Optional
|
||||||
|
from jose import JWTError, jwt
|
||||||
|
import bcrypt
|
||||||
|
from fastapi import Depends, HTTPException, status
|
||||||
|
from fastapi.security import OAuth2PasswordBearer
|
||||||
|
from .config import settings
|
||||||
|
|
||||||
|
|
||||||
|
# OAuth2 scheme
|
||||||
|
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="/api/auth/login")
|
||||||
|
|
||||||
|
|
||||||
|
def verify_password(plain_password: str, hashed_password: str) -> bool:
|
||||||
|
"""Verify a password against a hash"""
|
||||||
|
try:
|
||||||
|
password_bytes = plain_password.encode('utf-8')
|
||||||
|
hashed_bytes = hashed_password.encode('utf-8')
|
||||||
|
return bcrypt.checkpw(password_bytes, hashed_bytes)
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Password verification error: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def get_password_hash(password: str) -> str:
|
||||||
|
"""Hash a password"""
|
||||||
|
password_bytes = password.encode('utf-8')
|
||||||
|
salt = bcrypt.gensalt()
|
||||||
|
hashed = bcrypt.hashpw(password_bytes, salt)
|
||||||
|
return hashed.decode('utf-8')
|
||||||
|
|
||||||
|
|
||||||
|
def create_access_token(data: dict, expires_delta: Optional[timedelta] = None) -> str:
|
||||||
|
"""Create JWT access token"""
|
||||||
|
to_encode = data.copy()
|
||||||
|
if expires_delta:
|
||||||
|
expire = datetime.utcnow() + expires_delta
|
||||||
|
else:
|
||||||
|
expire = datetime.utcnow() + timedelta(minutes=settings.ACCESS_TOKEN_EXPIRE_MINUTES)
|
||||||
|
|
||||||
|
to_encode.update({"exp": expire, "type": "access"})
|
||||||
|
encoded_jwt = jwt.encode(to_encode, settings.SECRET_KEY, algorithm=settings.ALGORITHM)
|
||||||
|
return encoded_jwt
|
||||||
|
|
||||||
|
|
||||||
|
def create_refresh_token(data: dict) -> str:
|
||||||
|
"""Create JWT refresh token"""
|
||||||
|
to_encode = data.copy()
|
||||||
|
expire = datetime.utcnow() + timedelta(days=settings.REFRESH_TOKEN_EXPIRE_DAYS)
|
||||||
|
to_encode.update({"exp": expire, "type": "refresh"})
|
||||||
|
encoded_jwt = jwt.encode(to_encode, settings.SECRET_KEY, algorithm=settings.ALGORITHM)
|
||||||
|
return encoded_jwt
|
||||||
|
|
||||||
|
|
||||||
|
def decode_token(token: str) -> dict:
|
||||||
|
"""Decode and validate JWT token"""
|
||||||
|
try:
|
||||||
|
payload = jwt.decode(token, settings.SECRET_KEY, algorithms=[settings.ALGORITHM])
|
||||||
|
return payload
|
||||||
|
except JWTError:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||||
|
detail="Could not validate credentials",
|
||||||
|
headers={"WWW-Authenticate": "Bearer"},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def get_current_user_id(token: str = Depends(oauth2_scheme)) -> str:
|
||||||
|
"""Extract user ID from token"""
|
||||||
|
payload = decode_token(token)
|
||||||
|
user_id: str = payload.get("sub")
|
||||||
|
if user_id is None:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||||
|
detail="Could not validate credentials",
|
||||||
|
headers={"WWW-Authenticate": "Bearer"},
|
||||||
|
)
|
||||||
|
return user_id
|
||||||
0
services/console/backend/app/db/__init__.py
Normal file
0
services/console/backend/app/db/__init__.py
Normal file
37
services/console/backend/app/db/mongodb.py
Normal file
37
services/console/backend/app/db/mongodb.py
Normal file
@ -0,0 +1,37 @@
|
|||||||
|
from motor.motor_asyncio import AsyncIOMotorClient, AsyncIOMotorDatabase
|
||||||
|
from typing import Optional
|
||||||
|
from ..core.config import settings
|
||||||
|
|
||||||
|
|
||||||
|
class MongoDB:
|
||||||
|
"""MongoDB connection manager"""
|
||||||
|
|
||||||
|
client: Optional[AsyncIOMotorClient] = None
|
||||||
|
db: Optional[AsyncIOMotorDatabase] = None
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
async def connect(cls):
|
||||||
|
"""Connect to MongoDB"""
|
||||||
|
cls.client = AsyncIOMotorClient(settings.MONGODB_URL)
|
||||||
|
cls.db = cls.client[settings.DB_NAME]
|
||||||
|
print(f"✅ Connected to MongoDB: {settings.DB_NAME}")
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
async def disconnect(cls):
|
||||||
|
"""Disconnect from MongoDB"""
|
||||||
|
if cls.client:
|
||||||
|
cls.client.close()
|
||||||
|
print("❌ Disconnected from MongoDB")
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def get_db(cls) -> AsyncIOMotorDatabase:
|
||||||
|
"""Get database instance"""
|
||||||
|
if cls.db is None:
|
||||||
|
raise Exception("Database not initialized. Call connect() first.")
|
||||||
|
return cls.db
|
||||||
|
|
||||||
|
|
||||||
|
# Convenience function
|
||||||
|
async def get_database() -> AsyncIOMotorDatabase:
|
||||||
|
"""Dependency to get database"""
|
||||||
|
return MongoDB.get_db()
|
||||||
100
services/console/backend/app/main.py
Normal file
100
services/console/backend/app/main.py
Normal file
@ -0,0 +1,100 @@
|
|||||||
|
from fastapi import FastAPI
|
||||||
|
from fastapi.middleware.cors import CORSMiddleware
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from .core.config import settings
|
||||||
|
from .db.mongodb import MongoDB
|
||||||
|
from .routes import auth, services
|
||||||
|
|
||||||
|
# Configure logging
|
||||||
|
logging.basicConfig(
|
||||||
|
level=logging.INFO,
|
||||||
|
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||||
|
)
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def lifespan(app: FastAPI):
|
||||||
|
"""Application lifespan manager"""
|
||||||
|
# Startup
|
||||||
|
logger.info("🚀 Starting Console Backend...")
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Connect to MongoDB
|
||||||
|
await MongoDB.connect()
|
||||||
|
logger.info("✅ MongoDB connected successfully")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"❌ Failed to connect to MongoDB: {e}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
yield
|
||||||
|
|
||||||
|
# Shutdown
|
||||||
|
logger.info("👋 Shutting down Console Backend...")
|
||||||
|
await MongoDB.disconnect()
|
||||||
|
|
||||||
|
|
||||||
|
# Create FastAPI app
|
||||||
|
app = FastAPI(
|
||||||
|
title=settings.APP_NAME,
|
||||||
|
version=settings.APP_VERSION,
|
||||||
|
description="Site11 Console - Central management system for news generation pipeline",
|
||||||
|
lifespan=lifespan
|
||||||
|
)
|
||||||
|
|
||||||
|
# CORS middleware
|
||||||
|
app.add_middleware(
|
||||||
|
CORSMiddleware,
|
||||||
|
allow_origins=settings.CORS_ORIGINS if not settings.DEBUG else ["*"],
|
||||||
|
allow_credentials=True,
|
||||||
|
allow_methods=["*"],
|
||||||
|
allow_headers=["*"],
|
||||||
|
)
|
||||||
|
|
||||||
|
# Include routers
|
||||||
|
app.include_router(auth.router)
|
||||||
|
app.include_router(services.router)
|
||||||
|
|
||||||
|
|
||||||
|
# Health check endpoints
|
||||||
|
@app.get("/")
|
||||||
|
async def root():
|
||||||
|
"""Root endpoint"""
|
||||||
|
return {
|
||||||
|
"message": f"Welcome to {settings.APP_NAME}",
|
||||||
|
"version": settings.APP_VERSION,
|
||||||
|
"status": "running"
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/health")
|
||||||
|
async def health_check():
|
||||||
|
"""Health check endpoint"""
|
||||||
|
return {
|
||||||
|
"status": "healthy",
|
||||||
|
"service": "console-backend",
|
||||||
|
"version": settings.APP_VERSION
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/api/health")
|
||||||
|
async def api_health_check():
|
||||||
|
"""API health check endpoint for frontend"""
|
||||||
|
return {
|
||||||
|
"status": "healthy",
|
||||||
|
"service": "console-backend-api",
|
||||||
|
"version": settings.APP_VERSION
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
import uvicorn
|
||||||
|
uvicorn.run(
|
||||||
|
"app.main:app",
|
||||||
|
host="0.0.0.0",
|
||||||
|
port=8000,
|
||||||
|
reload=settings.DEBUG
|
||||||
|
)
|
||||||
0
services/console/backend/app/models/__init__.py
Normal file
0
services/console/backend/app/models/__init__.py
Normal file
81
services/console/backend/app/models/service.py
Normal file
81
services/console/backend/app/models/service.py
Normal file
@ -0,0 +1,81 @@
|
|||||||
|
from datetime import datetime
|
||||||
|
from typing import Optional, Dict, Any
|
||||||
|
from pydantic import BaseModel, Field, ConfigDict
|
||||||
|
from bson import ObjectId
|
||||||
|
from pydantic_core import core_schema
|
||||||
|
|
||||||
|
|
||||||
|
class PyObjectId(str):
|
||||||
|
"""Custom ObjectId type for Pydantic v2"""
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def __get_pydantic_core_schema__(cls, source_type, handler):
|
||||||
|
return core_schema.union_schema([
|
||||||
|
core_schema.is_instance_schema(ObjectId),
|
||||||
|
core_schema.chain_schema([
|
||||||
|
core_schema.str_schema(),
|
||||||
|
core_schema.no_info_plain_validator_function(cls.validate),
|
||||||
|
])
|
||||||
|
],
|
||||||
|
serialization=core_schema.plain_serializer_function_ser_schema(
|
||||||
|
lambda x: str(x)
|
||||||
|
))
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def validate(cls, v):
|
||||||
|
if not ObjectId.is_valid(v):
|
||||||
|
raise ValueError("Invalid ObjectId")
|
||||||
|
return ObjectId(v)
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceStatus:
|
||||||
|
"""Service status constants"""
|
||||||
|
HEALTHY = "healthy"
|
||||||
|
UNHEALTHY = "unhealthy"
|
||||||
|
UNKNOWN = "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceType:
|
||||||
|
"""Service type constants"""
|
||||||
|
BACKEND = "backend"
|
||||||
|
FRONTEND = "frontend"
|
||||||
|
DATABASE = "database"
|
||||||
|
CACHE = "cache"
|
||||||
|
MESSAGE_QUEUE = "message_queue"
|
||||||
|
OTHER = "other"
|
||||||
|
|
||||||
|
|
||||||
|
class Service(BaseModel):
|
||||||
|
"""Service model for MongoDB"""
|
||||||
|
id: Optional[PyObjectId] = Field(alias="_id", default=None)
|
||||||
|
name: str = Field(..., min_length=1, max_length=100)
|
||||||
|
description: Optional[str] = Field(default=None, max_length=500)
|
||||||
|
service_type: str = Field(default=ServiceType.BACKEND)
|
||||||
|
url: str = Field(..., min_length=1)
|
||||||
|
health_endpoint: Optional[str] = Field(default="/health")
|
||||||
|
status: str = Field(default=ServiceStatus.UNKNOWN)
|
||||||
|
last_health_check: Optional[datetime] = None
|
||||||
|
response_time_ms: Optional[float] = None
|
||||||
|
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||||
|
updated_at: datetime = Field(default_factory=datetime.utcnow)
|
||||||
|
metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
model_config = ConfigDict(
|
||||||
|
populate_by_name=True,
|
||||||
|
arbitrary_types_allowed=True,
|
||||||
|
json_encoders={ObjectId: str},
|
||||||
|
json_schema_extra={
|
||||||
|
"example": {
|
||||||
|
"name": "News API",
|
||||||
|
"description": "News generation and management API",
|
||||||
|
"service_type": "backend",
|
||||||
|
"url": "http://news-api:8050",
|
||||||
|
"health_endpoint": "/health",
|
||||||
|
"status": "healthy",
|
||||||
|
"metadata": {
|
||||||
|
"version": "1.0.0",
|
||||||
|
"port": 8050
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
89
services/console/backend/app/models/user.py
Normal file
89
services/console/backend/app/models/user.py
Normal file
@ -0,0 +1,89 @@
|
|||||||
|
from datetime import datetime
|
||||||
|
from typing import Optional, List, Annotated
|
||||||
|
from pydantic import BaseModel, EmailStr, Field, field_validator, ConfigDict
|
||||||
|
from pydantic_core import core_schema
|
||||||
|
from bson import ObjectId
|
||||||
|
|
||||||
|
|
||||||
|
class PyObjectId(str):
|
||||||
|
"""Custom ObjectId type for Pydantic v2"""
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def __get_pydantic_core_schema__(cls, source_type, handler):
|
||||||
|
return core_schema.union_schema([
|
||||||
|
core_schema.is_instance_schema(ObjectId),
|
||||||
|
core_schema.chain_schema([
|
||||||
|
core_schema.str_schema(),
|
||||||
|
core_schema.no_info_plain_validator_function(cls.validate),
|
||||||
|
])
|
||||||
|
],
|
||||||
|
serialization=core_schema.plain_serializer_function_ser_schema(
|
||||||
|
lambda x: str(x)
|
||||||
|
))
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def validate(cls, v):
|
||||||
|
if isinstance(v, ObjectId):
|
||||||
|
return v
|
||||||
|
if isinstance(v, str) and ObjectId.is_valid(v):
|
||||||
|
return ObjectId(v)
|
||||||
|
raise ValueError("Invalid ObjectId")
|
||||||
|
|
||||||
|
|
||||||
|
class UserRole(str):
|
||||||
|
"""User roles"""
|
||||||
|
ADMIN = "admin"
|
||||||
|
EDITOR = "editor"
|
||||||
|
VIEWER = "viewer"
|
||||||
|
|
||||||
|
|
||||||
|
class OAuthProvider(BaseModel):
|
||||||
|
"""OAuth provider information"""
|
||||||
|
provider: str = Field(..., description="OAuth provider name (google, github, azure)")
|
||||||
|
provider_user_id: str = Field(..., description="User ID from the provider")
|
||||||
|
access_token: Optional[str] = Field(None, description="Access token (encrypted)")
|
||||||
|
refresh_token: Optional[str] = Field(None, description="Refresh token (encrypted)")
|
||||||
|
|
||||||
|
|
||||||
|
class UserProfile(BaseModel):
|
||||||
|
"""User profile information"""
|
||||||
|
avatar_url: Optional[str] = None
|
||||||
|
department: Optional[str] = None
|
||||||
|
timezone: str = "Asia/Seoul"
|
||||||
|
|
||||||
|
|
||||||
|
class User(BaseModel):
|
||||||
|
"""User model"""
|
||||||
|
id: Optional[PyObjectId] = Field(alias="_id", default=None)
|
||||||
|
email: EmailStr = Field(..., description="User email")
|
||||||
|
username: str = Field(..., min_length=3, max_length=50, description="Username")
|
||||||
|
hashed_password: str = Field(..., description="Hashed password")
|
||||||
|
full_name: Optional[str] = Field(None, description="Full name")
|
||||||
|
role: str = Field(default=UserRole.VIEWER, description="User role")
|
||||||
|
permissions: List[str] = Field(default_factory=list, description="User permissions")
|
||||||
|
oauth_providers: List[OAuthProvider] = Field(default_factory=list)
|
||||||
|
profile: UserProfile = Field(default_factory=UserProfile)
|
||||||
|
status: str = Field(default="active", description="User status")
|
||||||
|
is_active: bool = Field(default=True)
|
||||||
|
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||||
|
updated_at: datetime = Field(default_factory=datetime.utcnow)
|
||||||
|
last_login_at: Optional[datetime] = None
|
||||||
|
|
||||||
|
model_config = ConfigDict(
|
||||||
|
populate_by_name=True,
|
||||||
|
arbitrary_types_allowed=True,
|
||||||
|
json_encoders={ObjectId: str},
|
||||||
|
json_schema_extra={
|
||||||
|
"example": {
|
||||||
|
"email": "user@example.com",
|
||||||
|
"username": "johndoe",
|
||||||
|
"full_name": "John Doe",
|
||||||
|
"role": "viewer"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class UserInDB(User):
|
||||||
|
"""User model with password hash"""
|
||||||
|
pass
|
||||||
0
services/console/backend/app/routes/__init__.py
Normal file
0
services/console/backend/app/routes/__init__.py
Normal file
167
services/console/backend/app/routes/auth.py
Normal file
167
services/console/backend/app/routes/auth.py
Normal file
@ -0,0 +1,167 @@
|
|||||||
|
from datetime import timedelta
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException, status
|
||||||
|
from fastapi.security import OAuth2PasswordRequestForm
|
||||||
|
from motor.motor_asyncio import AsyncIOMotorDatabase
|
||||||
|
|
||||||
|
from ..schemas.auth import UserRegister, Token, TokenRefresh, UserResponse
|
||||||
|
from ..services.user_service import UserService
|
||||||
|
from ..db.mongodb import get_database
|
||||||
|
from ..core.security import (
|
||||||
|
create_access_token,
|
||||||
|
create_refresh_token,
|
||||||
|
decode_token,
|
||||||
|
get_current_user_id
|
||||||
|
)
|
||||||
|
from ..core.config import settings
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/api/auth", tags=["authentication"])
|
||||||
|
|
||||||
|
|
||||||
|
def get_user_service(db: AsyncIOMotorDatabase = Depends(get_database)) -> UserService:
|
||||||
|
"""Dependency to get user service"""
|
||||||
|
return UserService(db)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/register", response_model=UserResponse, status_code=status.HTTP_201_CREATED)
|
||||||
|
async def register(
|
||||||
|
user_data: UserRegister,
|
||||||
|
user_service: UserService = Depends(get_user_service)
|
||||||
|
):
|
||||||
|
"""Register a new user"""
|
||||||
|
user = await user_service.create_user(user_data)
|
||||||
|
|
||||||
|
return UserResponse(
|
||||||
|
_id=str(user.id),
|
||||||
|
email=user.email,
|
||||||
|
username=user.username,
|
||||||
|
full_name=user.full_name,
|
||||||
|
role=user.role,
|
||||||
|
permissions=user.permissions,
|
||||||
|
status=user.status,
|
||||||
|
is_active=user.is_active,
|
||||||
|
created_at=user.created_at.isoformat(),
|
||||||
|
last_login_at=user.last_login_at.isoformat() if user.last_login_at else None
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/login", response_model=Token)
|
||||||
|
async def login(
|
||||||
|
form_data: OAuth2PasswordRequestForm = Depends(),
|
||||||
|
user_service: UserService = Depends(get_user_service)
|
||||||
|
):
|
||||||
|
"""Login with username/email and password"""
|
||||||
|
user = await user_service.authenticate_user(form_data.username, form_data.password)
|
||||||
|
|
||||||
|
if not user:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||||
|
detail="Incorrect username/email or password",
|
||||||
|
headers={"WWW-Authenticate": "Bearer"},
|
||||||
|
)
|
||||||
|
|
||||||
|
# Update last login timestamp
|
||||||
|
await user_service.update_last_login(str(user.id))
|
||||||
|
|
||||||
|
# Create tokens
|
||||||
|
access_token_expires = timedelta(minutes=settings.ACCESS_TOKEN_EXPIRE_MINUTES)
|
||||||
|
access_token = create_access_token(
|
||||||
|
data={"sub": str(user.id), "username": user.username},
|
||||||
|
expires_delta=access_token_expires
|
||||||
|
)
|
||||||
|
refresh_token = create_refresh_token(data={"sub": str(user.id)})
|
||||||
|
|
||||||
|
return Token(
|
||||||
|
access_token=access_token,
|
||||||
|
refresh_token=refresh_token,
|
||||||
|
token_type="bearer"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/refresh", response_model=Token)
|
||||||
|
async def refresh_token(
|
||||||
|
token_data: TokenRefresh,
|
||||||
|
user_service: UserService = Depends(get_user_service)
|
||||||
|
):
|
||||||
|
"""Refresh access token using refresh token"""
|
||||||
|
try:
|
||||||
|
payload = decode_token(token_data.refresh_token)
|
||||||
|
|
||||||
|
# Verify it's a refresh token
|
||||||
|
if payload.get("type") != "refresh":
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||||
|
detail="Invalid token type"
|
||||||
|
)
|
||||||
|
|
||||||
|
user_id = payload.get("sub")
|
||||||
|
if not user_id:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||||
|
detail="Invalid token"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Verify user still exists and is active
|
||||||
|
user = await user_service.get_user_by_id(user_id)
|
||||||
|
if not user or not user.is_active:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||||
|
detail="User not found or inactive"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create new access token
|
||||||
|
access_token_expires = timedelta(minutes=settings.ACCESS_TOKEN_EXPIRE_MINUTES)
|
||||||
|
access_token = create_access_token(
|
||||||
|
data={"sub": user_id, "username": user.username},
|
||||||
|
expires_delta=access_token_expires
|
||||||
|
)
|
||||||
|
|
||||||
|
return Token(
|
||||||
|
access_token=access_token,
|
||||||
|
refresh_token=token_data.refresh_token,
|
||||||
|
token_type="bearer"
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||||
|
detail="Invalid or expired refresh token"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/me", response_model=UserResponse)
|
||||||
|
async def get_current_user(
|
||||||
|
user_id: str = Depends(get_current_user_id),
|
||||||
|
user_service: UserService = Depends(get_user_service)
|
||||||
|
):
|
||||||
|
"""Get current user information"""
|
||||||
|
user = await user_service.get_user_by_id(user_id)
|
||||||
|
|
||||||
|
if not user:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_404_NOT_FOUND,
|
||||||
|
detail="User not found"
|
||||||
|
)
|
||||||
|
|
||||||
|
return UserResponse(
|
||||||
|
_id=str(user.id),
|
||||||
|
email=user.email,
|
||||||
|
username=user.username,
|
||||||
|
full_name=user.full_name,
|
||||||
|
role=user.role,
|
||||||
|
permissions=user.permissions,
|
||||||
|
status=user.status,
|
||||||
|
is_active=user.is_active,
|
||||||
|
created_at=user.created_at.isoformat(),
|
||||||
|
last_login_at=user.last_login_at.isoformat() if user.last_login_at else None
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/logout")
|
||||||
|
async def logout(user_id: str = Depends(get_current_user_id)):
|
||||||
|
"""Logout endpoint (token should be removed on client side)"""
|
||||||
|
# In a more sophisticated system, you might want to:
|
||||||
|
# 1. Blacklist the token in Redis
|
||||||
|
# 2. Log the logout event
|
||||||
|
# 3. Clear any session data
|
||||||
|
|
||||||
|
return {"message": "Successfully logged out"}
|
||||||
113
services/console/backend/app/routes/services.py
Normal file
113
services/console/backend/app/routes/services.py
Normal file
@ -0,0 +1,113 @@
|
|||||||
|
from typing import List
|
||||||
|
from fastapi import APIRouter, Depends, HTTPException, status
|
||||||
|
|
||||||
|
from app.models.service import Service
|
||||||
|
from app.models.user import User
|
||||||
|
from app.schemas.service import (
|
||||||
|
ServiceCreate,
|
||||||
|
ServiceUpdate,
|
||||||
|
ServiceResponse,
|
||||||
|
ServiceHealthCheck
|
||||||
|
)
|
||||||
|
from app.services.service_service import ServiceService
|
||||||
|
from app.core.security import get_current_user
|
||||||
|
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/api/services", tags=["services"])
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("", response_model=ServiceResponse, status_code=status.HTTP_201_CREATED)
|
||||||
|
async def create_service(
|
||||||
|
service_data: ServiceCreate,
|
||||||
|
current_user: User = Depends(get_current_user)
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Create a new service
|
||||||
|
|
||||||
|
Requires authentication.
|
||||||
|
"""
|
||||||
|
service = await ServiceService.create_service(service_data)
|
||||||
|
return service.model_dump(by_alias=True)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("", response_model=List[ServiceResponse])
|
||||||
|
async def get_all_services(
|
||||||
|
current_user: User = Depends(get_current_user)
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Get all services
|
||||||
|
|
||||||
|
Requires authentication.
|
||||||
|
"""
|
||||||
|
services = await ServiceService.get_all_services()
|
||||||
|
return [service.model_dump(by_alias=True) for service in services]
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/{service_id}", response_model=ServiceResponse)
|
||||||
|
async def get_service(
|
||||||
|
service_id: str,
|
||||||
|
current_user: User = Depends(get_current_user)
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Get a service by ID
|
||||||
|
|
||||||
|
Requires authentication.
|
||||||
|
"""
|
||||||
|
service = await ServiceService.get_service(service_id)
|
||||||
|
return service.model_dump(by_alias=True)
|
||||||
|
|
||||||
|
|
||||||
|
@router.put("/{service_id}", response_model=ServiceResponse)
|
||||||
|
async def update_service(
|
||||||
|
service_id: str,
|
||||||
|
service_data: ServiceUpdate,
|
||||||
|
current_user: User = Depends(get_current_user)
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Update a service
|
||||||
|
|
||||||
|
Requires authentication.
|
||||||
|
"""
|
||||||
|
service = await ServiceService.update_service(service_id, service_data)
|
||||||
|
return service.model_dump(by_alias=True)
|
||||||
|
|
||||||
|
|
||||||
|
@router.delete("/{service_id}", status_code=status.HTTP_204_NO_CONTENT)
|
||||||
|
async def delete_service(
|
||||||
|
service_id: str,
|
||||||
|
current_user: User = Depends(get_current_user)
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Delete a service
|
||||||
|
|
||||||
|
Requires authentication.
|
||||||
|
"""
|
||||||
|
await ServiceService.delete_service(service_id)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/{service_id}/health-check", response_model=ServiceHealthCheck)
|
||||||
|
async def check_service_health(
|
||||||
|
service_id: str,
|
||||||
|
current_user: User = Depends(get_current_user)
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Check health of a specific service
|
||||||
|
|
||||||
|
Requires authentication.
|
||||||
|
"""
|
||||||
|
result = await ServiceService.check_service_health(service_id)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/health-check/all", response_model=List[ServiceHealthCheck])
|
||||||
|
async def check_all_services_health(
|
||||||
|
current_user: User = Depends(get_current_user)
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Check health of all services
|
||||||
|
|
||||||
|
Requires authentication.
|
||||||
|
"""
|
||||||
|
results = await ServiceService.check_all_services_health()
|
||||||
|
return results
|
||||||
0
services/console/backend/app/schemas/__init__.py
Normal file
0
services/console/backend/app/schemas/__init__.py
Normal file
89
services/console/backend/app/schemas/auth.py
Normal file
89
services/console/backend/app/schemas/auth.py
Normal file
@ -0,0 +1,89 @@
|
|||||||
|
from pydantic import BaseModel, EmailStr, Field, ConfigDict
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
|
||||||
|
class UserRegister(BaseModel):
|
||||||
|
"""User registration schema"""
|
||||||
|
email: EmailStr = Field(..., description="User email")
|
||||||
|
username: str = Field(..., min_length=3, max_length=50, description="Username")
|
||||||
|
password: str = Field(..., min_length=6, description="Password")
|
||||||
|
full_name: Optional[str] = Field(None, description="Full name")
|
||||||
|
|
||||||
|
model_config = ConfigDict(
|
||||||
|
json_schema_extra={
|
||||||
|
"example": {
|
||||||
|
"email": "user@example.com",
|
||||||
|
"username": "johndoe",
|
||||||
|
"password": "securepassword123",
|
||||||
|
"full_name": "John Doe"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class UserLogin(BaseModel):
|
||||||
|
"""User login schema"""
|
||||||
|
username: str = Field(..., description="Username or email")
|
||||||
|
password: str = Field(..., description="Password")
|
||||||
|
|
||||||
|
model_config = ConfigDict(
|
||||||
|
json_schema_extra={
|
||||||
|
"example": {
|
||||||
|
"username": "johndoe",
|
||||||
|
"password": "securepassword123"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class Token(BaseModel):
|
||||||
|
"""Token response schema"""
|
||||||
|
access_token: str = Field(..., description="JWT access token")
|
||||||
|
refresh_token: Optional[str] = Field(None, description="JWT refresh token")
|
||||||
|
token_type: str = Field(default="bearer", description="Token type")
|
||||||
|
|
||||||
|
model_config = ConfigDict(
|
||||||
|
json_schema_extra={
|
||||||
|
"example": {
|
||||||
|
"access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
|
||||||
|
"refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
|
||||||
|
"token_type": "bearer"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class TokenRefresh(BaseModel):
|
||||||
|
"""Token refresh schema"""
|
||||||
|
refresh_token: str = Field(..., description="Refresh token")
|
||||||
|
|
||||||
|
|
||||||
|
class UserResponse(BaseModel):
|
||||||
|
"""User response schema (without password)"""
|
||||||
|
id: str = Field(..., alias="_id", description="User ID")
|
||||||
|
email: EmailStr
|
||||||
|
username: str
|
||||||
|
full_name: Optional[str] = None
|
||||||
|
role: str
|
||||||
|
permissions: list = []
|
||||||
|
status: str
|
||||||
|
is_active: bool
|
||||||
|
created_at: str
|
||||||
|
last_login_at: Optional[str] = None
|
||||||
|
|
||||||
|
model_config = ConfigDict(
|
||||||
|
populate_by_name=True,
|
||||||
|
json_schema_extra={
|
||||||
|
"example": {
|
||||||
|
"_id": "507f1f77bcf86cd799439011",
|
||||||
|
"email": "user@example.com",
|
||||||
|
"username": "johndoe",
|
||||||
|
"full_name": "John Doe",
|
||||||
|
"role": "viewer",
|
||||||
|
"permissions": [],
|
||||||
|
"status": "active",
|
||||||
|
"is_active": True,
|
||||||
|
"created_at": "2024-01-01T00:00:00Z"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
93
services/console/backend/app/schemas/service.py
Normal file
93
services/console/backend/app/schemas/service.py
Normal file
@ -0,0 +1,93 @@
|
|||||||
|
from datetime import datetime
|
||||||
|
from typing import Optional, Dict, Any
|
||||||
|
from pydantic import BaseModel, Field, ConfigDict
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceCreate(BaseModel):
|
||||||
|
"""Schema for creating a new service"""
|
||||||
|
name: str = Field(..., min_length=1, max_length=100)
|
||||||
|
description: Optional[str] = Field(default=None, max_length=500)
|
||||||
|
service_type: str = Field(default="backend")
|
||||||
|
url: str = Field(..., min_length=1)
|
||||||
|
health_endpoint: Optional[str] = Field(default="/health")
|
||||||
|
metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
model_config = ConfigDict(
|
||||||
|
json_schema_extra={
|
||||||
|
"example": {
|
||||||
|
"name": "News API",
|
||||||
|
"description": "News generation and management API",
|
||||||
|
"service_type": "backend",
|
||||||
|
"url": "http://news-api:8050",
|
||||||
|
"health_endpoint": "/health",
|
||||||
|
"metadata": {
|
||||||
|
"version": "1.0.0",
|
||||||
|
"port": 8050
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceUpdate(BaseModel):
|
||||||
|
"""Schema for updating a service"""
|
||||||
|
name: Optional[str] = Field(default=None, min_length=1, max_length=100)
|
||||||
|
description: Optional[str] = Field(default=None, max_length=500)
|
||||||
|
service_type: Optional[str] = None
|
||||||
|
url: Optional[str] = Field(default=None, min_length=1)
|
||||||
|
health_endpoint: Optional[str] = None
|
||||||
|
metadata: Optional[Dict[str, Any]] = None
|
||||||
|
|
||||||
|
model_config = ConfigDict(
|
||||||
|
json_schema_extra={
|
||||||
|
"example": {
|
||||||
|
"description": "Updated description",
|
||||||
|
"metadata": {
|
||||||
|
"version": "1.1.0"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceResponse(BaseModel):
|
||||||
|
"""Schema for service response"""
|
||||||
|
id: str = Field(alias="_id")
|
||||||
|
name: str
|
||||||
|
description: Optional[str] = None
|
||||||
|
service_type: str
|
||||||
|
url: str
|
||||||
|
health_endpoint: Optional[str] = None
|
||||||
|
status: str
|
||||||
|
last_health_check: Optional[datetime] = None
|
||||||
|
response_time_ms: Optional[float] = None
|
||||||
|
created_at: datetime
|
||||||
|
updated_at: datetime
|
||||||
|
metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||||
|
|
||||||
|
model_config = ConfigDict(
|
||||||
|
populate_by_name=True,
|
||||||
|
from_attributes=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceHealthCheck(BaseModel):
|
||||||
|
"""Schema for health check result"""
|
||||||
|
service_id: str
|
||||||
|
service_name: str
|
||||||
|
status: str
|
||||||
|
response_time_ms: Optional[float] = None
|
||||||
|
checked_at: datetime
|
||||||
|
error_message: Optional[str] = None
|
||||||
|
|
||||||
|
model_config = ConfigDict(
|
||||||
|
json_schema_extra={
|
||||||
|
"example": {
|
||||||
|
"service_id": "507f1f77bcf86cd799439011",
|
||||||
|
"service_name": "News API",
|
||||||
|
"status": "healthy",
|
||||||
|
"response_time_ms": 45.2,
|
||||||
|
"checked_at": "2025-10-28T10:00:00Z"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
0
services/console/backend/app/services/__init__.py
Normal file
0
services/console/backend/app/services/__init__.py
Normal file
212
services/console/backend/app/services/service_service.py
Normal file
212
services/console/backend/app/services/service_service.py
Normal file
@ -0,0 +1,212 @@
|
|||||||
|
from datetime import datetime
|
||||||
|
from typing import List, Optional
|
||||||
|
import time
|
||||||
|
import httpx
|
||||||
|
from bson import ObjectId
|
||||||
|
from fastapi import HTTPException, status
|
||||||
|
|
||||||
|
from app.db.mongodb import MongoDB
|
||||||
|
from app.models.service import Service, ServiceStatus
|
||||||
|
from app.schemas.service import ServiceCreate, ServiceUpdate, ServiceHealthCheck
|
||||||
|
|
||||||
|
|
||||||
|
class ServiceService:
|
||||||
|
"""Service management business logic"""
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
async def create_service(service_data: ServiceCreate) -> Service:
|
||||||
|
"""Create a new service"""
|
||||||
|
db = MongoDB.db
|
||||||
|
|
||||||
|
# Check if service with same name already exists
|
||||||
|
existing = await db.services.find_one({"name": service_data.name})
|
||||||
|
if existing:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST,
|
||||||
|
detail="Service with this name already exists"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create service document
|
||||||
|
service = Service(
|
||||||
|
**service_data.model_dump(),
|
||||||
|
status=ServiceStatus.UNKNOWN,
|
||||||
|
created_at=datetime.utcnow(),
|
||||||
|
updated_at=datetime.utcnow()
|
||||||
|
)
|
||||||
|
|
||||||
|
# Insert into database
|
||||||
|
result = await db.services.insert_one(service.model_dump(by_alias=True, exclude={"id"}))
|
||||||
|
service.id = str(result.inserted_id)
|
||||||
|
|
||||||
|
return service
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
async def get_service(service_id: str) -> Service:
|
||||||
|
"""Get service by ID"""
|
||||||
|
db = MongoDB.db
|
||||||
|
|
||||||
|
if not ObjectId.is_valid(service_id):
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST,
|
||||||
|
detail="Invalid service ID"
|
||||||
|
)
|
||||||
|
|
||||||
|
service_doc = await db.services.find_one({"_id": ObjectId(service_id)})
|
||||||
|
if not service_doc:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_404_NOT_FOUND,
|
||||||
|
detail="Service not found"
|
||||||
|
)
|
||||||
|
|
||||||
|
return Service(**service_doc)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
async def get_all_services() -> List[Service]:
|
||||||
|
"""Get all services"""
|
||||||
|
db = MongoDB.db
|
||||||
|
|
||||||
|
cursor = db.services.find()
|
||||||
|
services = []
|
||||||
|
async for doc in cursor:
|
||||||
|
services.append(Service(**doc))
|
||||||
|
|
||||||
|
return services
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
async def update_service(service_id: str, service_data: ServiceUpdate) -> Service:
|
||||||
|
"""Update a service"""
|
||||||
|
db = MongoDB.db
|
||||||
|
|
||||||
|
if not ObjectId.is_valid(service_id):
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST,
|
||||||
|
detail="Invalid service ID"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Get existing service
|
||||||
|
existing = await db.services.find_one({"_id": ObjectId(service_id)})
|
||||||
|
if not existing:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_404_NOT_FOUND,
|
||||||
|
detail="Service not found"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Update only provided fields
|
||||||
|
update_data = service_data.model_dump(exclude_unset=True)
|
||||||
|
if update_data:
|
||||||
|
update_data["updated_at"] = datetime.utcnow()
|
||||||
|
|
||||||
|
# Check for name conflict if name is being updated
|
||||||
|
if "name" in update_data:
|
||||||
|
name_conflict = await db.services.find_one({
|
||||||
|
"name": update_data["name"],
|
||||||
|
"_id": {"$ne": ObjectId(service_id)}
|
||||||
|
})
|
||||||
|
if name_conflict:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST,
|
||||||
|
detail="Service with this name already exists"
|
||||||
|
)
|
||||||
|
|
||||||
|
await db.services.update_one(
|
||||||
|
{"_id": ObjectId(service_id)},
|
||||||
|
{"$set": update_data}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Return updated service
|
||||||
|
updated_doc = await db.services.find_one({"_id": ObjectId(service_id)})
|
||||||
|
return Service(**updated_doc)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
async def delete_service(service_id: str) -> bool:
|
||||||
|
"""Delete a service"""
|
||||||
|
db = MongoDB.db
|
||||||
|
|
||||||
|
if not ObjectId.is_valid(service_id):
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST,
|
||||||
|
detail="Invalid service ID"
|
||||||
|
)
|
||||||
|
|
||||||
|
result = await db.services.delete_one({"_id": ObjectId(service_id)})
|
||||||
|
if result.deleted_count == 0:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_404_NOT_FOUND,
|
||||||
|
detail="Service not found"
|
||||||
|
)
|
||||||
|
|
||||||
|
return True
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
async def check_service_health(service_id: str) -> ServiceHealthCheck:
|
||||||
|
"""Check health of a specific service"""
|
||||||
|
db = MongoDB.db
|
||||||
|
|
||||||
|
# Get service
|
||||||
|
service = await ServiceService.get_service(service_id)
|
||||||
|
|
||||||
|
# Perform health check
|
||||||
|
start_time = time.time()
|
||||||
|
status_result = ServiceStatus.UNKNOWN
|
||||||
|
error_message = None
|
||||||
|
|
||||||
|
try:
|
||||||
|
health_url = f"{service.url.rstrip('/')}{service.health_endpoint or '/health'}"
|
||||||
|
|
||||||
|
async with httpx.AsyncClient(timeout=5.0) as client:
|
||||||
|
response = await client.get(health_url)
|
||||||
|
|
||||||
|
if response.status_code == 200:
|
||||||
|
status_result = ServiceStatus.HEALTHY
|
||||||
|
else:
|
||||||
|
status_result = ServiceStatus.UNHEALTHY
|
||||||
|
error_message = f"HTTP {response.status_code}"
|
||||||
|
|
||||||
|
except httpx.TimeoutException:
|
||||||
|
status_result = ServiceStatus.UNHEALTHY
|
||||||
|
error_message = "Request timeout"
|
||||||
|
|
||||||
|
except httpx.RequestError as e:
|
||||||
|
status_result = ServiceStatus.UNHEALTHY
|
||||||
|
error_message = f"Connection error: {str(e)}"
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
status_result = ServiceStatus.UNHEALTHY
|
||||||
|
error_message = f"Error: {str(e)}"
|
||||||
|
|
||||||
|
response_time = (time.time() - start_time) * 1000 # Convert to ms
|
||||||
|
checked_at = datetime.utcnow()
|
||||||
|
|
||||||
|
# Update service status in database
|
||||||
|
await db.services.update_one(
|
||||||
|
{"_id": ObjectId(service_id)},
|
||||||
|
{
|
||||||
|
"$set": {
|
||||||
|
"status": status_result,
|
||||||
|
"last_health_check": checked_at,
|
||||||
|
"response_time_ms": response_time if status_result == ServiceStatus.HEALTHY else None,
|
||||||
|
"updated_at": checked_at
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
return ServiceHealthCheck(
|
||||||
|
service_id=service_id,
|
||||||
|
service_name=service.name,
|
||||||
|
status=status_result,
|
||||||
|
response_time_ms=response_time if status_result == ServiceStatus.HEALTHY else None,
|
||||||
|
checked_at=checked_at,
|
||||||
|
error_message=error_message
|
||||||
|
)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
async def check_all_services_health() -> List[ServiceHealthCheck]:
|
||||||
|
"""Check health of all services"""
|
||||||
|
services = await ServiceService.get_all_services()
|
||||||
|
results = []
|
||||||
|
|
||||||
|
for service in services:
|
||||||
|
result = await ServiceService.check_service_health(str(service.id))
|
||||||
|
results.append(result)
|
||||||
|
|
||||||
|
return results
|
||||||
143
services/console/backend/app/services/user_service.py
Normal file
143
services/console/backend/app/services/user_service.py
Normal file
@ -0,0 +1,143 @@
|
|||||||
|
from datetime import datetime
|
||||||
|
from typing import Optional
|
||||||
|
from motor.motor_asyncio import AsyncIOMotorDatabase
|
||||||
|
from bson import ObjectId
|
||||||
|
from fastapi import HTTPException, status
|
||||||
|
|
||||||
|
from ..models.user import User, UserInDB, UserRole
|
||||||
|
from ..schemas.auth import UserRegister
|
||||||
|
from ..core.security import get_password_hash, verify_password
|
||||||
|
|
||||||
|
|
||||||
|
class UserService:
|
||||||
|
"""User service for business logic"""
|
||||||
|
|
||||||
|
def __init__(self, db: AsyncIOMotorDatabase):
|
||||||
|
self.db = db
|
||||||
|
self.collection = db.users
|
||||||
|
|
||||||
|
async def create_user(self, user_data: UserRegister) -> UserInDB:
|
||||||
|
"""Create a new user"""
|
||||||
|
# Check if user already exists
|
||||||
|
existing_user = await self.collection.find_one({
|
||||||
|
"$or": [
|
||||||
|
{"email": user_data.email},
|
||||||
|
{"username": user_data.username}
|
||||||
|
]
|
||||||
|
})
|
||||||
|
|
||||||
|
if existing_user:
|
||||||
|
if existing_user["email"] == user_data.email:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST,
|
||||||
|
detail="Email already registered"
|
||||||
|
)
|
||||||
|
if existing_user["username"] == user_data.username:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_400_BAD_REQUEST,
|
||||||
|
detail="Username already taken"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create user document
|
||||||
|
user_dict = {
|
||||||
|
"email": user_data.email,
|
||||||
|
"username": user_data.username,
|
||||||
|
"hashed_password": get_password_hash(user_data.password),
|
||||||
|
"full_name": user_data.full_name,
|
||||||
|
"role": UserRole.VIEWER, # Default role
|
||||||
|
"permissions": [],
|
||||||
|
"oauth_providers": [],
|
||||||
|
"profile": {
|
||||||
|
"avatar_url": None,
|
||||||
|
"department": None,
|
||||||
|
"timezone": "Asia/Seoul"
|
||||||
|
},
|
||||||
|
"status": "active",
|
||||||
|
"is_active": True,
|
||||||
|
"created_at": datetime.utcnow(),
|
||||||
|
"updated_at": datetime.utcnow(),
|
||||||
|
"last_login_at": None
|
||||||
|
}
|
||||||
|
|
||||||
|
result = await self.collection.insert_one(user_dict)
|
||||||
|
user_dict["_id"] = result.inserted_id
|
||||||
|
|
||||||
|
return UserInDB(**user_dict)
|
||||||
|
|
||||||
|
async def get_user_by_username(self, username: str) -> Optional[UserInDB]:
|
||||||
|
"""Get user by username"""
|
||||||
|
user_dict = await self.collection.find_one({"username": username})
|
||||||
|
if user_dict:
|
||||||
|
return UserInDB(**user_dict)
|
||||||
|
return None
|
||||||
|
|
||||||
|
async def get_user_by_email(self, email: str) -> Optional[UserInDB]:
|
||||||
|
"""Get user by email"""
|
||||||
|
user_dict = await self.collection.find_one({"email": email})
|
||||||
|
if user_dict:
|
||||||
|
return UserInDB(**user_dict)
|
||||||
|
return None
|
||||||
|
|
||||||
|
async def get_user_by_id(self, user_id: str) -> Optional[UserInDB]:
|
||||||
|
"""Get user by ID"""
|
||||||
|
if not ObjectId.is_valid(user_id):
|
||||||
|
return None
|
||||||
|
|
||||||
|
user_dict = await self.collection.find_one({"_id": ObjectId(user_id)})
|
||||||
|
if user_dict:
|
||||||
|
return UserInDB(**user_dict)
|
||||||
|
return None
|
||||||
|
|
||||||
|
async def authenticate_user(self, username: str, password: str) -> Optional[UserInDB]:
|
||||||
|
"""Authenticate user with username/email and password"""
|
||||||
|
# Try to find by username or email
|
||||||
|
user = await self.get_user_by_username(username)
|
||||||
|
if not user:
|
||||||
|
user = await self.get_user_by_email(username)
|
||||||
|
|
||||||
|
if not user:
|
||||||
|
return None
|
||||||
|
|
||||||
|
if not verify_password(password, user.hashed_password):
|
||||||
|
return None
|
||||||
|
|
||||||
|
if not user.is_active:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_403_FORBIDDEN,
|
||||||
|
detail="User account is inactive"
|
||||||
|
)
|
||||||
|
|
||||||
|
return user
|
||||||
|
|
||||||
|
async def update_last_login(self, user_id: str):
|
||||||
|
"""Update user's last login timestamp"""
|
||||||
|
await self.collection.update_one(
|
||||||
|
{"_id": ObjectId(user_id)},
|
||||||
|
{"$set": {"last_login_at": datetime.utcnow()}}
|
||||||
|
)
|
||||||
|
|
||||||
|
async def update_user(self, user_id: str, update_data: dict) -> Optional[UserInDB]:
|
||||||
|
"""Update user data"""
|
||||||
|
if not ObjectId.is_valid(user_id):
|
||||||
|
return None
|
||||||
|
|
||||||
|
update_data["updated_at"] = datetime.utcnow()
|
||||||
|
|
||||||
|
await self.collection.update_one(
|
||||||
|
{"_id": ObjectId(user_id)},
|
||||||
|
{"$set": update_data}
|
||||||
|
)
|
||||||
|
|
||||||
|
return await self.get_user_by_id(user_id)
|
||||||
|
|
||||||
|
async def delete_user(self, user_id: str) -> bool:
|
||||||
|
"""Delete user (soft delete - set status to deleted)"""
|
||||||
|
if not ObjectId.is_valid(user_id):
|
||||||
|
return False
|
||||||
|
|
||||||
|
result = await self.collection.update_one(
|
||||||
|
{"_id": ObjectId(user_id)},
|
||||||
|
{"$set": {"status": "deleted", "is_active": False, "updated_at": datetime.utcnow()}}
|
||||||
|
)
|
||||||
|
|
||||||
|
return result.modified_count > 0
|
||||||
@ -93,6 +93,25 @@ async def health_check():
|
|||||||
"event_consumer": "running" if event_consumer else "not running"
|
"event_consumer": "running" if event_consumer else "not running"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@app.get("/api/health")
|
||||||
|
async def api_health_check():
|
||||||
|
"""API health check endpoint for frontend"""
|
||||||
|
return {
|
||||||
|
"status": "healthy",
|
||||||
|
"service": "console-backend",
|
||||||
|
"timestamp": datetime.now().isoformat()
|
||||||
|
}
|
||||||
|
|
||||||
|
@app.get("/api/users/health")
|
||||||
|
async def users_health_check():
|
||||||
|
"""Users service health check endpoint"""
|
||||||
|
# TODO: Replace with actual users service health check when implemented
|
||||||
|
return {
|
||||||
|
"status": "healthy",
|
||||||
|
"service": "users-service",
|
||||||
|
"timestamp": datetime.now().isoformat()
|
||||||
|
}
|
||||||
|
|
||||||
# Event Management Endpoints
|
# Event Management Endpoints
|
||||||
@app.get("/api/events/stats")
|
@app.get("/api/events/stats")
|
||||||
async def get_event_stats(current_user = Depends(get_current_user)):
|
async def get_event_stats(current_user = Depends(get_current_user)):
|
||||||
@ -2,9 +2,13 @@ fastapi==0.109.0
|
|||||||
uvicorn[standard]==0.27.0
|
uvicorn[standard]==0.27.0
|
||||||
python-dotenv==1.0.0
|
python-dotenv==1.0.0
|
||||||
pydantic==2.5.3
|
pydantic==2.5.3
|
||||||
|
pydantic-settings==2.1.0
|
||||||
httpx==0.26.0
|
httpx==0.26.0
|
||||||
python-jose[cryptography]==3.3.0
|
python-jose[cryptography]==3.3.0
|
||||||
passlib[bcrypt]==1.7.4
|
bcrypt==4.1.2
|
||||||
python-multipart==0.0.6
|
python-multipart==0.0.6
|
||||||
redis==5.0.1
|
redis==5.0.1
|
||||||
aiokafka==0.10.0
|
aiokafka==0.10.0
|
||||||
|
motor==3.3.2
|
||||||
|
pymongo==4.6.1
|
||||||
|
email-validator==2.1.0
|
||||||
35
services/console/frontend/src/App.tsx
Normal file
35
services/console/frontend/src/App.tsx
Normal file
@ -0,0 +1,35 @@
|
|||||||
|
import { Routes, Route, Navigate } from 'react-router-dom'
|
||||||
|
import { AuthProvider } from './contexts/AuthContext'
|
||||||
|
import ProtectedRoute from './components/ProtectedRoute'
|
||||||
|
import Layout from './components/Layout'
|
||||||
|
import Login from './pages/Login'
|
||||||
|
import Register from './pages/Register'
|
||||||
|
import Dashboard from './pages/Dashboard'
|
||||||
|
import Services from './pages/Services'
|
||||||
|
import Users from './pages/Users'
|
||||||
|
|
||||||
|
function App() {
|
||||||
|
return (
|
||||||
|
<AuthProvider>
|
||||||
|
<Routes>
|
||||||
|
<Route path="/login" element={<Login />} />
|
||||||
|
<Route path="/register" element={<Register />} />
|
||||||
|
<Route
|
||||||
|
path="/"
|
||||||
|
element={
|
||||||
|
<ProtectedRoute>
|
||||||
|
<Layout />
|
||||||
|
</ProtectedRoute>
|
||||||
|
}
|
||||||
|
>
|
||||||
|
<Route index element={<Dashboard />} />
|
||||||
|
<Route path="services" element={<Services />} />
|
||||||
|
<Route path="users" element={<Users />} />
|
||||||
|
</Route>
|
||||||
|
<Route path="*" element={<Navigate to="/" replace />} />
|
||||||
|
</Routes>
|
||||||
|
</AuthProvider>
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
export default App
|
||||||
100
services/console/frontend/src/api/auth.ts
Normal file
100
services/console/frontend/src/api/auth.ts
Normal file
@ -0,0 +1,100 @@
|
|||||||
|
import axios from 'axios';
|
||||||
|
import type { User, LoginRequest, RegisterRequest, AuthTokens } from '../types/auth';
|
||||||
|
|
||||||
|
const API_BASE_URL = import.meta.env.VITE_API_URL || 'http://localhost:8000';
|
||||||
|
|
||||||
|
const api = axios.create({
|
||||||
|
baseURL: API_BASE_URL,
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
// Add token to requests
|
||||||
|
api.interceptors.request.use((config) => {
|
||||||
|
const token = localStorage.getItem('access_token');
|
||||||
|
if (token) {
|
||||||
|
config.headers.Authorization = `Bearer ${token}`;
|
||||||
|
}
|
||||||
|
return config;
|
||||||
|
});
|
||||||
|
|
||||||
|
// Handle token refresh on 401
|
||||||
|
api.interceptors.response.use(
|
||||||
|
(response) => response,
|
||||||
|
async (error) => {
|
||||||
|
const originalRequest = error.config;
|
||||||
|
|
||||||
|
if (error.response?.status === 401 && !originalRequest._retry) {
|
||||||
|
originalRequest._retry = true;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const refreshToken = localStorage.getItem('refresh_token');
|
||||||
|
if (refreshToken) {
|
||||||
|
const { data } = await axios.post<AuthTokens>(
|
||||||
|
`${API_BASE_URL}/api/auth/refresh`,
|
||||||
|
{ refresh_token: refreshToken }
|
||||||
|
);
|
||||||
|
|
||||||
|
localStorage.setItem('access_token', data.access_token);
|
||||||
|
localStorage.setItem('refresh_token', data.refresh_token);
|
||||||
|
|
||||||
|
originalRequest.headers.Authorization = `Bearer ${data.access_token}`;
|
||||||
|
return api(originalRequest);
|
||||||
|
}
|
||||||
|
} catch (refreshError) {
|
||||||
|
localStorage.removeItem('access_token');
|
||||||
|
localStorage.removeItem('refresh_token');
|
||||||
|
window.location.href = '/login';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return Promise.reject(error);
|
||||||
|
}
|
||||||
|
);
|
||||||
|
|
||||||
|
export const authAPI = {
|
||||||
|
login: async (credentials: LoginRequest): Promise<AuthTokens> => {
|
||||||
|
const formData = new URLSearchParams();
|
||||||
|
formData.append('username', credentials.username);
|
||||||
|
formData.append('password', credentials.password);
|
||||||
|
|
||||||
|
const { data } = await axios.post<AuthTokens>(
|
||||||
|
`${API_BASE_URL}/api/auth/login`,
|
||||||
|
formData,
|
||||||
|
{
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/x-www-form-urlencoded',
|
||||||
|
},
|
||||||
|
}
|
||||||
|
);
|
||||||
|
return data;
|
||||||
|
},
|
||||||
|
|
||||||
|
register: async (userData: RegisterRequest): Promise<User> => {
|
||||||
|
const { data } = await axios.post<User>(
|
||||||
|
`${API_BASE_URL}/api/auth/register`,
|
||||||
|
userData
|
||||||
|
);
|
||||||
|
return data;
|
||||||
|
},
|
||||||
|
|
||||||
|
getCurrentUser: async (): Promise<User> => {
|
||||||
|
const { data } = await api.get<User>('/api/auth/me');
|
||||||
|
return data;
|
||||||
|
},
|
||||||
|
|
||||||
|
refreshToken: async (refreshToken: string): Promise<AuthTokens> => {
|
||||||
|
const { data } = await axios.post<AuthTokens>(
|
||||||
|
`${API_BASE_URL}/api/auth/refresh`,
|
||||||
|
{ refresh_token: refreshToken }
|
||||||
|
);
|
||||||
|
return data;
|
||||||
|
},
|
||||||
|
|
||||||
|
logout: async (): Promise<void> => {
|
||||||
|
await api.post('/api/auth/logout');
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
export default api;
|
||||||
45
services/console/frontend/src/api/service.ts
Normal file
45
services/console/frontend/src/api/service.ts
Normal file
@ -0,0 +1,45 @@
|
|||||||
|
import api from './auth';
|
||||||
|
import type { Service, ServiceCreate, ServiceUpdate, ServiceHealthCheck } from '../types/service';
|
||||||
|
|
||||||
|
export const serviceAPI = {
|
||||||
|
// Get all services
|
||||||
|
getAll: async (): Promise<Service[]> => {
|
||||||
|
const { data } = await api.get<Service[]>('/api/services');
|
||||||
|
return data;
|
||||||
|
},
|
||||||
|
|
||||||
|
// Get service by ID
|
||||||
|
getById: async (id: string): Promise<Service> => {
|
||||||
|
const { data } = await api.get<Service>(`/api/services/${id}`);
|
||||||
|
return data;
|
||||||
|
},
|
||||||
|
|
||||||
|
// Create new service
|
||||||
|
create: async (serviceData: ServiceCreate): Promise<Service> => {
|
||||||
|
const { data } = await api.post<Service>('/api/services', serviceData);
|
||||||
|
return data;
|
||||||
|
},
|
||||||
|
|
||||||
|
// Update service
|
||||||
|
update: async (id: string, serviceData: ServiceUpdate): Promise<Service> => {
|
||||||
|
const { data} = await api.put<Service>(`/api/services/${id}`, serviceData);
|
||||||
|
return data;
|
||||||
|
},
|
||||||
|
|
||||||
|
// Delete service
|
||||||
|
delete: async (id: string): Promise<void> => {
|
||||||
|
await api.delete(`/api/services/${id}`);
|
||||||
|
},
|
||||||
|
|
||||||
|
// Check service health
|
||||||
|
checkHealth: async (id: string): Promise<ServiceHealthCheck> => {
|
||||||
|
const { data } = await api.post<ServiceHealthCheck>(`/api/services/${id}/health-check`);
|
||||||
|
return data;
|
||||||
|
},
|
||||||
|
|
||||||
|
// Check all services health
|
||||||
|
checkAllHealth: async (): Promise<ServiceHealthCheck[]> => {
|
||||||
|
const { data } = await api.post<ServiceHealthCheck[]>('/api/services/health-check/all');
|
||||||
|
return data;
|
||||||
|
},
|
||||||
|
};
|
||||||
@ -1,5 +1,5 @@
|
|||||||
import { useState } from 'react'
|
import { useState } from 'react'
|
||||||
import { Outlet, Link as RouterLink } from 'react-router-dom'
|
import { Outlet, Link as RouterLink, useNavigate } from 'react-router-dom'
|
||||||
import {
|
import {
|
||||||
AppBar,
|
AppBar,
|
||||||
Box,
|
Box,
|
||||||
@ -12,13 +12,17 @@ import {
|
|||||||
ListItemText,
|
ListItemText,
|
||||||
Toolbar,
|
Toolbar,
|
||||||
Typography,
|
Typography,
|
||||||
|
Menu,
|
||||||
|
MenuItem,
|
||||||
} from '@mui/material'
|
} from '@mui/material'
|
||||||
import {
|
import {
|
||||||
Menu as MenuIcon,
|
Menu as MenuIcon,
|
||||||
Dashboard as DashboardIcon,
|
Dashboard as DashboardIcon,
|
||||||
Cloud as CloudIcon,
|
Cloud as CloudIcon,
|
||||||
People as PeopleIcon,
|
People as PeopleIcon,
|
||||||
|
AccountCircle,
|
||||||
} from '@mui/icons-material'
|
} from '@mui/icons-material'
|
||||||
|
import { useAuth } from '../contexts/AuthContext'
|
||||||
|
|
||||||
const drawerWidth = 240
|
const drawerWidth = 240
|
||||||
|
|
||||||
@ -30,11 +34,28 @@ const menuItems = [
|
|||||||
|
|
||||||
function Layout() {
|
function Layout() {
|
||||||
const [open, setOpen] = useState(true)
|
const [open, setOpen] = useState(true)
|
||||||
|
const [anchorEl, setAnchorEl] = useState<null | HTMLElement>(null)
|
||||||
|
const { user, logout } = useAuth()
|
||||||
|
const navigate = useNavigate()
|
||||||
|
|
||||||
const handleDrawerToggle = () => {
|
const handleDrawerToggle = () => {
|
||||||
setOpen(!open)
|
setOpen(!open)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const handleMenu = (event: React.MouseEvent<HTMLElement>) => {
|
||||||
|
setAnchorEl(event.currentTarget)
|
||||||
|
}
|
||||||
|
|
||||||
|
const handleClose = () => {
|
||||||
|
setAnchorEl(null)
|
||||||
|
}
|
||||||
|
|
||||||
|
const handleLogout = () => {
|
||||||
|
logout()
|
||||||
|
navigate('/login')
|
||||||
|
handleClose()
|
||||||
|
}
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<Box sx={{ display: 'flex' }}>
|
<Box sx={{ display: 'flex' }}>
|
||||||
<AppBar
|
<AppBar
|
||||||
@ -51,9 +72,41 @@ function Layout() {
|
|||||||
>
|
>
|
||||||
<MenuIcon />
|
<MenuIcon />
|
||||||
</IconButton>
|
</IconButton>
|
||||||
<Typography variant="h6" noWrap component="div">
|
<Typography variant="h6" noWrap component="div" sx={{ flexGrow: 1 }}>
|
||||||
Microservices Console
|
Site11 Console
|
||||||
</Typography>
|
</Typography>
|
||||||
|
<Box sx={{ display: 'flex', alignItems: 'center', gap: 2 }}>
|
||||||
|
<Typography variant="body2">
|
||||||
|
{user?.username} ({user?.role})
|
||||||
|
</Typography>
|
||||||
|
<IconButton
|
||||||
|
size="large"
|
||||||
|
aria-label="account of current user"
|
||||||
|
aria-controls="menu-appbar"
|
||||||
|
aria-haspopup="true"
|
||||||
|
onClick={handleMenu}
|
||||||
|
color="inherit"
|
||||||
|
>
|
||||||
|
<AccountCircle />
|
||||||
|
</IconButton>
|
||||||
|
<Menu
|
||||||
|
id="menu-appbar"
|
||||||
|
anchorEl={anchorEl}
|
||||||
|
anchorOrigin={{
|
||||||
|
vertical: 'top',
|
||||||
|
horizontal: 'right',
|
||||||
|
}}
|
||||||
|
keepMounted
|
||||||
|
transformOrigin={{
|
||||||
|
vertical: 'top',
|
||||||
|
horizontal: 'right',
|
||||||
|
}}
|
||||||
|
open={Boolean(anchorEl)}
|
||||||
|
onClose={handleClose}
|
||||||
|
>
|
||||||
|
<MenuItem onClick={handleLogout}>Logout</MenuItem>
|
||||||
|
</Menu>
|
||||||
|
</Box>
|
||||||
</Toolbar>
|
</Toolbar>
|
||||||
</AppBar>
|
</AppBar>
|
||||||
<Drawer
|
<Drawer
|
||||||
35
services/console/frontend/src/components/ProtectedRoute.tsx
Normal file
35
services/console/frontend/src/components/ProtectedRoute.tsx
Normal file
@ -0,0 +1,35 @@
|
|||||||
|
import React from 'react';
|
||||||
|
import { Navigate } from 'react-router-dom';
|
||||||
|
import { Box, CircularProgress } from '@mui/material';
|
||||||
|
import { useAuth } from '../contexts/AuthContext';
|
||||||
|
|
||||||
|
interface ProtectedRouteProps {
|
||||||
|
children: React.ReactNode;
|
||||||
|
}
|
||||||
|
|
||||||
|
const ProtectedRoute: React.FC<ProtectedRouteProps> = ({ children }) => {
|
||||||
|
const { isAuthenticated, isLoading } = useAuth();
|
||||||
|
|
||||||
|
if (isLoading) {
|
||||||
|
return (
|
||||||
|
<Box
|
||||||
|
sx={{
|
||||||
|
display: 'flex',
|
||||||
|
justifyContent: 'center',
|
||||||
|
alignItems: 'center',
|
||||||
|
minHeight: '100vh',
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
<CircularProgress />
|
||||||
|
</Box>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!isAuthenticated) {
|
||||||
|
return <Navigate to="/login" replace />;
|
||||||
|
}
|
||||||
|
|
||||||
|
return <>{children}</>;
|
||||||
|
};
|
||||||
|
|
||||||
|
export default ProtectedRoute;
|
||||||
96
services/console/frontend/src/contexts/AuthContext.tsx
Normal file
96
services/console/frontend/src/contexts/AuthContext.tsx
Normal file
@ -0,0 +1,96 @@
|
|||||||
|
import React, { createContext, useContext, useState, useEffect, ReactNode } from 'react';
|
||||||
|
import { authAPI } from '../api/auth';
|
||||||
|
import type { User, LoginRequest, RegisterRequest, AuthContextType } from '../types/auth';
|
||||||
|
|
||||||
|
const AuthContext = createContext<AuthContextType | undefined>(undefined);
|
||||||
|
|
||||||
|
export const useAuth = () => {
|
||||||
|
const context = useContext(AuthContext);
|
||||||
|
if (!context) {
|
||||||
|
throw new Error('useAuth must be used within an AuthProvider');
|
||||||
|
}
|
||||||
|
return context;
|
||||||
|
};
|
||||||
|
|
||||||
|
interface AuthProviderProps {
|
||||||
|
children: ReactNode;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const AuthProvider: React.FC<AuthProviderProps> = ({ children }) => {
|
||||||
|
const [user, setUser] = useState<User | null>(null);
|
||||||
|
const [isLoading, setIsLoading] = useState(true);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
// Check if user is already logged in
|
||||||
|
const initAuth = async () => {
|
||||||
|
const token = localStorage.getItem('access_token');
|
||||||
|
if (token) {
|
||||||
|
try {
|
||||||
|
const userData = await authAPI.getCurrentUser();
|
||||||
|
setUser(userData);
|
||||||
|
} catch (error) {
|
||||||
|
localStorage.removeItem('access_token');
|
||||||
|
localStorage.removeItem('refresh_token');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
setIsLoading(false);
|
||||||
|
};
|
||||||
|
|
||||||
|
initAuth();
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const login = async (credentials: LoginRequest) => {
|
||||||
|
const tokens = await authAPI.login(credentials);
|
||||||
|
localStorage.setItem('access_token', tokens.access_token);
|
||||||
|
localStorage.setItem('refresh_token', tokens.refresh_token);
|
||||||
|
|
||||||
|
const userData = await authAPI.getCurrentUser();
|
||||||
|
setUser(userData);
|
||||||
|
};
|
||||||
|
|
||||||
|
const register = async (data: RegisterRequest) => {
|
||||||
|
const newUser = await authAPI.register(data);
|
||||||
|
|
||||||
|
// Auto login after registration
|
||||||
|
const tokens = await authAPI.login({
|
||||||
|
username: data.username,
|
||||||
|
password: data.password,
|
||||||
|
});
|
||||||
|
localStorage.setItem('access_token', tokens.access_token);
|
||||||
|
localStorage.setItem('refresh_token', tokens.refresh_token);
|
||||||
|
|
||||||
|
setUser(newUser);
|
||||||
|
};
|
||||||
|
|
||||||
|
const logout = () => {
|
||||||
|
localStorage.removeItem('access_token');
|
||||||
|
localStorage.removeItem('refresh_token');
|
||||||
|
setUser(null);
|
||||||
|
|
||||||
|
// Optional: call backend logout endpoint
|
||||||
|
authAPI.logout().catch(() => {
|
||||||
|
// Ignore errors on logout
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
||||||
|
const refreshToken = async () => {
|
||||||
|
const token = localStorage.getItem('refresh_token');
|
||||||
|
if (token) {
|
||||||
|
const tokens = await authAPI.refreshToken(token);
|
||||||
|
localStorage.setItem('access_token', tokens.access_token);
|
||||||
|
localStorage.setItem('refresh_token', tokens.refresh_token);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const value: AuthContextType = {
|
||||||
|
user,
|
||||||
|
isAuthenticated: !!user,
|
||||||
|
isLoading,
|
||||||
|
login,
|
||||||
|
register,
|
||||||
|
logout,
|
||||||
|
refreshToken,
|
||||||
|
};
|
||||||
|
|
||||||
|
return <AuthContext.Provider value={value}>{children}</AuthContext.Provider>;
|
||||||
|
};
|
||||||
128
services/console/frontend/src/pages/Login.tsx
Normal file
128
services/console/frontend/src/pages/Login.tsx
Normal file
@ -0,0 +1,128 @@
|
|||||||
|
import React, { useState } from 'react';
|
||||||
|
import { useNavigate, Link as RouterLink } from 'react-router-dom';
|
||||||
|
import {
|
||||||
|
Container,
|
||||||
|
Box,
|
||||||
|
Paper,
|
||||||
|
TextField,
|
||||||
|
Button,
|
||||||
|
Typography,
|
||||||
|
Alert,
|
||||||
|
Link,
|
||||||
|
} from '@mui/material';
|
||||||
|
import { useAuth } from '../contexts/AuthContext';
|
||||||
|
|
||||||
|
const Login: React.FC = () => {
|
||||||
|
const navigate = useNavigate();
|
||||||
|
const { login } = useAuth();
|
||||||
|
const [formData, setFormData] = useState({
|
||||||
|
username: '',
|
||||||
|
password: '',
|
||||||
|
});
|
||||||
|
const [error, setError] = useState('');
|
||||||
|
const [loading, setLoading] = useState(false);
|
||||||
|
|
||||||
|
const handleChange = (e: React.ChangeEvent<HTMLInputElement>) => {
|
||||||
|
setFormData({
|
||||||
|
...formData,
|
||||||
|
[e.target.name]: e.target.value,
|
||||||
|
});
|
||||||
|
setError('');
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleSubmit = async (e: React.FormEvent) => {
|
||||||
|
e.preventDefault();
|
||||||
|
setError('');
|
||||||
|
setLoading(true);
|
||||||
|
|
||||||
|
try {
|
||||||
|
await login(formData);
|
||||||
|
navigate('/');
|
||||||
|
} catch (err: any) {
|
||||||
|
setError(err.response?.data?.detail || 'Login failed. Please try again.');
|
||||||
|
} finally {
|
||||||
|
setLoading(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
return (
|
||||||
|
<Container maxWidth="sm">
|
||||||
|
<Box
|
||||||
|
sx={{
|
||||||
|
minHeight: '100vh',
|
||||||
|
display: 'flex',
|
||||||
|
alignItems: 'center',
|
||||||
|
justifyContent: 'center',
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
<Paper
|
||||||
|
elevation={3}
|
||||||
|
sx={{
|
||||||
|
p: 4,
|
||||||
|
width: '100%',
|
||||||
|
maxWidth: 400,
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
<Typography variant="h4" component="h1" gutterBottom align="center">
|
||||||
|
Site11 Console
|
||||||
|
</Typography>
|
||||||
|
<Typography variant="h6" component="h2" gutterBottom align="center" color="text.secondary">
|
||||||
|
Sign In
|
||||||
|
</Typography>
|
||||||
|
|
||||||
|
{error && (
|
||||||
|
<Alert severity="error" sx={{ mb: 2 }}>
|
||||||
|
{error}
|
||||||
|
</Alert>
|
||||||
|
)}
|
||||||
|
|
||||||
|
<Box component="form" onSubmit={handleSubmit} sx={{ mt: 2 }}>
|
||||||
|
<TextField
|
||||||
|
fullWidth
|
||||||
|
label="Username"
|
||||||
|
name="username"
|
||||||
|
value={formData.username}
|
||||||
|
onChange={handleChange}
|
||||||
|
margin="normal"
|
||||||
|
required
|
||||||
|
autoFocus
|
||||||
|
disabled={loading}
|
||||||
|
/>
|
||||||
|
<TextField
|
||||||
|
fullWidth
|
||||||
|
label="Password"
|
||||||
|
name="password"
|
||||||
|
type="password"
|
||||||
|
value={formData.password}
|
||||||
|
onChange={handleChange}
|
||||||
|
margin="normal"
|
||||||
|
required
|
||||||
|
disabled={loading}
|
||||||
|
/>
|
||||||
|
<Button
|
||||||
|
type="submit"
|
||||||
|
fullWidth
|
||||||
|
variant="contained"
|
||||||
|
size="large"
|
||||||
|
sx={{ mt: 3, mb: 2 }}
|
||||||
|
disabled={loading}
|
||||||
|
>
|
||||||
|
{loading ? 'Signing in...' : 'Sign In'}
|
||||||
|
</Button>
|
||||||
|
|
||||||
|
<Box sx={{ textAlign: 'center' }}>
|
||||||
|
<Typography variant="body2">
|
||||||
|
Don't have an account?{' '}
|
||||||
|
<Link component={RouterLink} to="/register" underline="hover">
|
||||||
|
Sign Up
|
||||||
|
</Link>
|
||||||
|
</Typography>
|
||||||
|
</Box>
|
||||||
|
</Box>
|
||||||
|
</Paper>
|
||||||
|
</Box>
|
||||||
|
</Container>
|
||||||
|
);
|
||||||
|
};
|
||||||
|
|
||||||
|
export default Login;
|
||||||
182
services/console/frontend/src/pages/Register.tsx
Normal file
182
services/console/frontend/src/pages/Register.tsx
Normal file
@ -0,0 +1,182 @@
|
|||||||
|
import React, { useState } from 'react';
|
||||||
|
import { useNavigate, Link as RouterLink } from 'react-router-dom';
|
||||||
|
import {
|
||||||
|
Container,
|
||||||
|
Box,
|
||||||
|
Paper,
|
||||||
|
TextField,
|
||||||
|
Button,
|
||||||
|
Typography,
|
||||||
|
Alert,
|
||||||
|
Link,
|
||||||
|
} from '@mui/material';
|
||||||
|
import { useAuth } from '../contexts/AuthContext';
|
||||||
|
|
||||||
|
const Register: React.FC = () => {
|
||||||
|
const navigate = useNavigate();
|
||||||
|
const { register } = useAuth();
|
||||||
|
const [formData, setFormData] = useState({
|
||||||
|
email: '',
|
||||||
|
username: '',
|
||||||
|
password: '',
|
||||||
|
confirmPassword: '',
|
||||||
|
full_name: '',
|
||||||
|
});
|
||||||
|
const [error, setError] = useState('');
|
||||||
|
const [loading, setLoading] = useState(false);
|
||||||
|
|
||||||
|
const handleChange = (e: React.ChangeEvent<HTMLInputElement>) => {
|
||||||
|
setFormData({
|
||||||
|
...formData,
|
||||||
|
[e.target.name]: e.target.value,
|
||||||
|
});
|
||||||
|
setError('');
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleSubmit = async (e: React.FormEvent) => {
|
||||||
|
e.preventDefault();
|
||||||
|
setError('');
|
||||||
|
|
||||||
|
// Validate passwords match
|
||||||
|
if (formData.password !== formData.confirmPassword) {
|
||||||
|
setError('Passwords do not match');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate password length
|
||||||
|
if (formData.password.length < 6) {
|
||||||
|
setError('Password must be at least 6 characters');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
setLoading(true);
|
||||||
|
|
||||||
|
try {
|
||||||
|
await register({
|
||||||
|
email: formData.email,
|
||||||
|
username: formData.username,
|
||||||
|
password: formData.password,
|
||||||
|
full_name: formData.full_name || undefined,
|
||||||
|
});
|
||||||
|
navigate('/');
|
||||||
|
} catch (err: any) {
|
||||||
|
setError(err.response?.data?.detail || 'Registration failed. Please try again.');
|
||||||
|
} finally {
|
||||||
|
setLoading(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
return (
|
||||||
|
<Container maxWidth="sm">
|
||||||
|
<Box
|
||||||
|
sx={{
|
||||||
|
minHeight: '100vh',
|
||||||
|
display: 'flex',
|
||||||
|
alignItems: 'center',
|
||||||
|
justifyContent: 'center',
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
<Paper
|
||||||
|
elevation={3}
|
||||||
|
sx={{
|
||||||
|
p: 4,
|
||||||
|
width: '100%',
|
||||||
|
maxWidth: 400,
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
<Typography variant="h4" component="h1" gutterBottom align="center">
|
||||||
|
Site11 Console
|
||||||
|
</Typography>
|
||||||
|
<Typography variant="h6" component="h2" gutterBottom align="center" color="text.secondary">
|
||||||
|
Create Account
|
||||||
|
</Typography>
|
||||||
|
|
||||||
|
{error && (
|
||||||
|
<Alert severity="error" sx={{ mb: 2 }}>
|
||||||
|
{error}
|
||||||
|
</Alert>
|
||||||
|
)}
|
||||||
|
|
||||||
|
<Box component="form" onSubmit={handleSubmit} sx={{ mt: 2 }}>
|
||||||
|
<TextField
|
||||||
|
fullWidth
|
||||||
|
label="Email"
|
||||||
|
name="email"
|
||||||
|
type="email"
|
||||||
|
value={formData.email}
|
||||||
|
onChange={handleChange}
|
||||||
|
margin="normal"
|
||||||
|
required
|
||||||
|
autoFocus
|
||||||
|
disabled={loading}
|
||||||
|
/>
|
||||||
|
<TextField
|
||||||
|
fullWidth
|
||||||
|
label="Username"
|
||||||
|
name="username"
|
||||||
|
value={formData.username}
|
||||||
|
onChange={handleChange}
|
||||||
|
margin="normal"
|
||||||
|
required
|
||||||
|
disabled={loading}
|
||||||
|
inputProps={{ minLength: 3, maxLength: 50 }}
|
||||||
|
/>
|
||||||
|
<TextField
|
||||||
|
fullWidth
|
||||||
|
label="Full Name"
|
||||||
|
name="full_name"
|
||||||
|
value={formData.full_name}
|
||||||
|
onChange={handleChange}
|
||||||
|
margin="normal"
|
||||||
|
disabled={loading}
|
||||||
|
/>
|
||||||
|
<TextField
|
||||||
|
fullWidth
|
||||||
|
label="Password"
|
||||||
|
name="password"
|
||||||
|
type="password"
|
||||||
|
value={formData.password}
|
||||||
|
onChange={handleChange}
|
||||||
|
margin="normal"
|
||||||
|
required
|
||||||
|
disabled={loading}
|
||||||
|
inputProps={{ minLength: 6 }}
|
||||||
|
/>
|
||||||
|
<TextField
|
||||||
|
fullWidth
|
||||||
|
label="Confirm Password"
|
||||||
|
name="confirmPassword"
|
||||||
|
type="password"
|
||||||
|
value={formData.confirmPassword}
|
||||||
|
onChange={handleChange}
|
||||||
|
margin="normal"
|
||||||
|
required
|
||||||
|
disabled={loading}
|
||||||
|
/>
|
||||||
|
<Button
|
||||||
|
type="submit"
|
||||||
|
fullWidth
|
||||||
|
variant="contained"
|
||||||
|
size="large"
|
||||||
|
sx={{ mt: 3, mb: 2 }}
|
||||||
|
disabled={loading}
|
||||||
|
>
|
||||||
|
{loading ? 'Creating account...' : 'Sign Up'}
|
||||||
|
</Button>
|
||||||
|
|
||||||
|
<Box sx={{ textAlign: 'center' }}>
|
||||||
|
<Typography variant="body2">
|
||||||
|
Already have an account?{' '}
|
||||||
|
<Link component={RouterLink} to="/login" underline="hover">
|
||||||
|
Sign In
|
||||||
|
</Link>
|
||||||
|
</Typography>
|
||||||
|
</Box>
|
||||||
|
</Box>
|
||||||
|
</Paper>
|
||||||
|
</Box>
|
||||||
|
</Container>
|
||||||
|
);
|
||||||
|
};
|
||||||
|
|
||||||
|
export default Register;
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user