feat: Integrate News Engine Console with Pipeline Monitor service

Backend Integration:
- Created PipelineClient for communicating with Pipeline Monitor (port 8100)
- Added proxy endpoints in monitoring.py:
  * /api/v1/monitoring/pipeline/stats - Queue status and article counts
  * /api/v1/monitoring/pipeline/health - Pipeline service health
  * /api/v1/monitoring/pipeline/queues/{name} - Queue details
  * /api/v1/monitoring/pipeline/workers - Worker status

Frontend Integration:
- Added Pipeline Monitor API functions to monitoring.ts
- Updated Monitoring page to display:
  * Redis queue status (keyword, rss, search, summarize, assembly)
  * Article statistics (today, total, active keywords)
  * Pipeline health status
  * Worker status for each pipeline type

Architecture:
- Console acts as API Gateway, proxying requests to Pipeline Monitor
- Pipeline Monitor (services/pipeline/monitor) manages:
  * RSS Collector, Google Search, AI Summarizer, Article Assembly workers
  * Redis queues for async job processing
  * MongoDB for article and keyword storage

This integration allows the News Engine Console to monitor and display
real-time pipeline activity, queue status, and worker health.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
jungwoo choi
2025-11-04 22:06:46 +09:00
parent d6ae03f42b
commit 7d29b7ca85
4 changed files with 331 additions and 1 deletions

View File

@ -4,6 +4,7 @@ from datetime import datetime
from app.core.auth import get_current_active_user, User
from app.core.database import get_database
from app.core.pipeline_client import get_pipeline_client, PipelineClient
from app.services.monitoring_service import MonitoringService
router = APIRouter()
@ -135,3 +136,58 @@ async def get_error_summary(
"""
summary = await monitoring_service.get_error_summary(hours=hours)
return summary
# =============================================================================
# Pipeline Monitor Proxy Endpoints
# =============================================================================
@router.get("/pipeline/stats")
async def get_pipeline_stats(
current_user: User = Depends(get_current_active_user),
pipeline_client: PipelineClient = Depends(get_pipeline_client)
):
"""
Get pipeline statistics from Pipeline Monitor service
Returns queue status, article counts, and worker info
"""
return await pipeline_client.get_stats()
@router.get("/pipeline/health")
async def get_pipeline_health(
current_user: User = Depends(get_current_active_user),
pipeline_client: PipelineClient = Depends(get_pipeline_client)
):
"""
Get Pipeline Monitor service health status
"""
return await pipeline_client.get_health()
@router.get("/pipeline/queues/{queue_name}")
async def get_queue_details(
queue_name: str,
current_user: User = Depends(get_current_active_user),
pipeline_client: PipelineClient = Depends(get_pipeline_client)
):
"""
Get details for a specific pipeline queue
Returns queue length, processing count, failed count, and preview of items
"""
return await pipeline_client.get_queue_details(queue_name)
@router.get("/pipeline/workers")
async def get_pipeline_workers(
current_user: User = Depends(get_current_active_user),
pipeline_client: PipelineClient = Depends(get_pipeline_client)
):
"""
Get status of all pipeline workers
Returns active worker counts for each pipeline type
"""
return await pipeline_client.get_workers()