070032006e
Major architectural transformation from synchronous to asynchronous processing: ## Pipeline Services (8 microservices) - pipeline-scheduler: APScheduler for 30-minute periodic job triggers - pipeline-rss-collector: RSS feed collection with deduplication (7-day TTL) - pipeline-google-search: Content enrichment via Google Search API - pipeline-ai-summarizer: AI summarization using Claude API (claude-sonnet-4-20250514) - pipeline-translator: Translation using DeepL Pro API - pipeline-image-generator: Image generation with Replicate API (Stable Diffusion) - pipeline-article-assembly: Final article assembly and MongoDB storage - pipeline-monitor: Real-time monitoring dashboard (port 8100) ## Key Features - Redis-based job queue with deduplication - Asynchronous processing with Python asyncio - Shared models and queue manager for inter-service communication - Docker containerization for all services - Container names standardized with site11_ prefix ## Removed Services - Moved to backup: google-search, rss-feed, news-aggregator, ai-writer ## Configuration - DeepL Pro API: 3abbc796-2515-44a8-972d-22dcf27ab54a - Claude Model: claude-sonnet-4-20250514 - Redis Queue TTL: 7 days for deduplication 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
49 lines
2.3 KiB
Python
49 lines
2.3 KiB
Python
"""
|
|
Queue Models for AI Writer Service
|
|
Redis 큐에서 사용할 데이터 모델 정의
|
|
"""
|
|
from pydantic import BaseModel, Field
|
|
from typing import Optional, List, Dict, Any
|
|
from datetime import datetime
|
|
from enum import Enum
|
|
|
|
class JobStatus(str, Enum):
|
|
"""작업 상태"""
|
|
PENDING = "pending"
|
|
PROCESSING = "processing"
|
|
COMPLETED = "completed"
|
|
FAILED = "failed"
|
|
SKIPPED = "skipped"
|
|
|
|
class NewsJobData(BaseModel):
|
|
"""큐에 들어갈 뉴스 작업 데이터"""
|
|
job_id: str = Field(..., description="작업 고유 ID")
|
|
keyword: str = Field(..., description="원본 검색 키워드")
|
|
rss_title: str = Field(..., description="RSS 제목")
|
|
rss_link: Optional[str] = Field(None, description="RSS 링크")
|
|
rss_published: Optional[str] = Field(None, description="RSS 발행일")
|
|
google_results: List[Dict[str, Any]] = Field(default_factory=list, description="구글 검색 결과")
|
|
style: str = Field("professional", description="기사 스타일")
|
|
created_at: datetime = Field(default_factory=datetime.now, description="작업 생성 시간")
|
|
priority: int = Field(0, description="우선순위 (높을수록 우선)")
|
|
retry_count: int = Field(0, description="재시도 횟수")
|
|
max_retries: int = Field(3, description="최대 재시도 횟수")
|
|
|
|
class JobResult(BaseModel):
|
|
"""작업 결과"""
|
|
job_id: str = Field(..., description="작업 고유 ID")
|
|
status: JobStatus = Field(..., description="작업 상태")
|
|
article_id: Optional[str] = Field(None, description="생성된 기사 ID")
|
|
error_message: Optional[str] = Field(None, description="에러 메시지")
|
|
processing_time: Optional[float] = Field(None, description="처리 시간(초)")
|
|
completed_at: Optional[datetime] = Field(None, description="완료 시간")
|
|
|
|
class QueueStats(BaseModel):
|
|
"""큐 통계"""
|
|
pending_jobs: int = Field(..., description="대기 중인 작업 수")
|
|
processing_jobs: int = Field(..., description="처리 중인 작업 수")
|
|
completed_jobs: int = Field(..., description="완료된 작업 수")
|
|
failed_jobs: int = Field(..., description="실패한 작업 수")
|
|
total_jobs: int = Field(..., description="전체 작업 수")
|
|
workers_active: int = Field(..., description="활성 워커 수")
|
|
average_processing_time: Optional[float] = Field(None, description="평균 처리 시간(초)") |