feat: Integrate News Engine Console with Pipeline Monitor service
Backend Integration:
- Created PipelineClient for communicating with Pipeline Monitor (port 8100)
- Added proxy endpoints in monitoring.py:
* /api/v1/monitoring/pipeline/stats - Queue status and article counts
* /api/v1/monitoring/pipeline/health - Pipeline service health
* /api/v1/monitoring/pipeline/queues/{name} - Queue details
* /api/v1/monitoring/pipeline/workers - Worker status
Frontend Integration:
- Added Pipeline Monitor API functions to monitoring.ts
- Updated Monitoring page to display:
* Redis queue status (keyword, rss, search, summarize, assembly)
* Article statistics (today, total, active keywords)
* Pipeline health status
* Worker status for each pipeline type
Architecture:
- Console acts as API Gateway, proxying requests to Pipeline Monitor
- Pipeline Monitor (services/pipeline/monitor) manages:
* RSS Collector, Google Search, AI Summarizer, Article Assembly workers
* Redis queues for async job processing
* MongoDB for article and keyword storage
This integration allows the News Engine Console to monitor and display
real-time pipeline activity, queue status, and worker health.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@ -4,6 +4,7 @@ from datetime import datetime
|
||||
|
||||
from app.core.auth import get_current_active_user, User
|
||||
from app.core.database import get_database
|
||||
from app.core.pipeline_client import get_pipeline_client, PipelineClient
|
||||
from app.services.monitoring_service import MonitoringService
|
||||
|
||||
router = APIRouter()
|
||||
@ -135,3 +136,58 @@ async def get_error_summary(
|
||||
"""
|
||||
summary = await monitoring_service.get_error_summary(hours=hours)
|
||||
return summary
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Pipeline Monitor Proxy Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/pipeline/stats")
|
||||
async def get_pipeline_stats(
|
||||
current_user: User = Depends(get_current_active_user),
|
||||
pipeline_client: PipelineClient = Depends(get_pipeline_client)
|
||||
):
|
||||
"""
|
||||
Get pipeline statistics from Pipeline Monitor service
|
||||
|
||||
Returns queue status, article counts, and worker info
|
||||
"""
|
||||
return await pipeline_client.get_stats()
|
||||
|
||||
|
||||
@router.get("/pipeline/health")
|
||||
async def get_pipeline_health(
|
||||
current_user: User = Depends(get_current_active_user),
|
||||
pipeline_client: PipelineClient = Depends(get_pipeline_client)
|
||||
):
|
||||
"""
|
||||
Get Pipeline Monitor service health status
|
||||
"""
|
||||
return await pipeline_client.get_health()
|
||||
|
||||
|
||||
@router.get("/pipeline/queues/{queue_name}")
|
||||
async def get_queue_details(
|
||||
queue_name: str,
|
||||
current_user: User = Depends(get_current_active_user),
|
||||
pipeline_client: PipelineClient = Depends(get_pipeline_client)
|
||||
):
|
||||
"""
|
||||
Get details for a specific pipeline queue
|
||||
|
||||
Returns queue length, processing count, failed count, and preview of items
|
||||
"""
|
||||
return await pipeline_client.get_queue_details(queue_name)
|
||||
|
||||
|
||||
@router.get("/pipeline/workers")
|
||||
async def get_pipeline_workers(
|
||||
current_user: User = Depends(get_current_active_user),
|
||||
pipeline_client: PipelineClient = Depends(get_pipeline_client)
|
||||
):
|
||||
"""
|
||||
Get status of all pipeline workers
|
||||
|
||||
Returns active worker counts for each pipeline type
|
||||
"""
|
||||
return await pipeline_client.get_workers()
|
||||
|
||||
Reference in New Issue
Block a user