feat: Implement async queue-based news pipeline with microservices
Major architectural transformation from synchronous to asynchronous processing: ## Pipeline Services (8 microservices) - pipeline-scheduler: APScheduler for 30-minute periodic job triggers - pipeline-rss-collector: RSS feed collection with deduplication (7-day TTL) - pipeline-google-search: Content enrichment via Google Search API - pipeline-ai-summarizer: AI summarization using Claude API (claude-sonnet-4-20250514) - pipeline-translator: Translation using DeepL Pro API - pipeline-image-generator: Image generation with Replicate API (Stable Diffusion) - pipeline-article-assembly: Final article assembly and MongoDB storage - pipeline-monitor: Real-time monitoring dashboard (port 8100) ## Key Features - Redis-based job queue with deduplication - Asynchronous processing with Python asyncio - Shared models and queue manager for inter-service communication - Docker containerization for all services - Container names standardized with site11_ prefix ## Removed Services - Moved to backup: google-search, rss-feed, news-aggregator, ai-writer ## Configuration - DeepL Pro API: 3abbc796-2515-44a8-972d-22dcf27ab54a - Claude Model: claude-sonnet-4-20250514 - Redis Queue TTL: 7 days for deduplication 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
30
backup-services/google-search/backend/app/config.py
Normal file
30
backup-services/google-search/backend/app/config.py
Normal file
@ -0,0 +1,30 @@
|
||||
from pydantic_settings import BaseSettings
|
||||
from typing import Optional
|
||||
|
||||
class Settings(BaseSettings):
|
||||
# Google Custom Search API 설정
|
||||
google_api_key: Optional[str] = None
|
||||
google_search_engine_id: Optional[str] = None
|
||||
|
||||
# SerpAPI 설정 (대안)
|
||||
serpapi_key: Optional[str] = None
|
||||
|
||||
# Redis 캐싱 설정
|
||||
redis_host: str = "redis"
|
||||
redis_port: int = 6379
|
||||
redis_db: int = 2
|
||||
cache_ttl: int = 3600 # 1시간
|
||||
|
||||
# 검색 설정
|
||||
max_results: int = 10
|
||||
default_language: str = "ko"
|
||||
default_country: str = "kr"
|
||||
|
||||
# 서비스 설정
|
||||
service_name: str = "Google Search Service"
|
||||
debug: bool = True
|
||||
|
||||
class Config:
|
||||
env_file = ".env"
|
||||
|
||||
settings = Settings()
|
||||
Reference in New Issue
Block a user