Commit Graph

13 Commits

Author SHA1 Message Date
07088e60e9 feat: Implement backend core functionality for news-engine-console
Phase 1 Backend Implementation:
-  MongoDB data models (Keyword, Pipeline, User, Application)
-  Pydantic schemas for all models with validation
-  KeywordService: Full CRUD, filtering, pagination, stats, toggle status
-  PipelineService: Full CRUD, start/stop/restart, logs, config management
-  Keywords API: 8 endpoints with complete functionality
-  Pipelines API: 11 endpoints with complete functionality
-  Updated TODO.md to reflect completion

Key Features:
- Async MongoDB operations with Motor
- Comprehensive filtering and pagination support
- Pipeline logging system
- Statistics tracking for keywords and pipelines
- Proper error handling with HTTP status codes
- Type-safe request/response models

Files Added:
- models/: 4 data models with PyObjectId support
- schemas/: 4 schema modules with Create/Update/Response patterns
- services/: KeywordService (234 lines) + PipelineService (332 lines)

Files Modified:
- api/keywords.py: 40 → 212 lines (complete implementation)
- api/pipelines.py: 25 → 300 lines (complete implementation)
- TODO.md: Updated checklist with completed items

Next Steps:
- UserService with authentication
- ApplicationService for OAuth2
- MonitoringService
- Redis integration
- Frontend implementation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-04 16:24:14 +09:00
7649844023 feat: Initialize News Engine Console project
Create comprehensive news pipeline management and monitoring system
with backend API structure and detailed implementation roadmap.

Core Features (7):
1. Keyword Management - Pipeline keyword CRUD and control
2. Pipeline Monitoring - Processing stats and utilization metrics
3. Pipeline Control - Step-wise start/stop and scheduling
4. Logging System - Pipeline status logs and error tracking
5. User Management - User CRUD with role-based access (Admin/Editor/Viewer)
6. Application Management - OAuth2/JWT-based Application CRUD
7. System Monitoring - Service health checks and resource monitoring

Technology Stack:
- Backend: FastAPI + Motor (MongoDB async) + Redis
- Frontend: React 18 + TypeScript + Material-UI v7 (planned)
- Auth: JWT + OAuth2
- Infrastructure: Docker + Kubernetes

Project Structure:
- backend/app/api/ - 5 API routers (keywords, pipelines, users, applications, monitoring)
- backend/app/core/ - Core configurations (config, database, auth)
- backend/app/models/ - Data models (planned)
- backend/app/services/ - Business logic (planned)
- backend/app/schemas/ - Pydantic schemas (planned)
- frontend/ - React application (planned)
- k8s/ - Kubernetes manifests (planned)

Documentation:
- README.md - Project overview, current status, API endpoints, DB schema
- TODO.md - Detailed implementation plan for next sessions

Current Status:
 Project structure initialized
 Backend basic configuration (config, database, auth)
 API router skeletons (5 routers)
 Requirements and environment setup
🚧 Models, services, and schemas pending
📋 Frontend implementation pending
📋 Docker and Kubernetes deployment pending

Next Steps (See TODO.md):
1. MongoDB schema and indexes
2. Pydantic schemas with validation
3. Service layer implementation
4. Redis integration
5. Login/authentication API
6. Frontend basic setup

This provides a solid foundation for building a comprehensive
news pipeline management console system.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-04 16:16:09 +09:00
e40f50005d docs: Add comprehensive News API developer guide
Create external developer-focused API documentation for News API service
with practical integration examples for frontend systems.

Features:
- 10 major sections covering all API endpoints
- Complete TypeScript type definitions
- Real-world React/Next.js integration examples
- Axios client setup and React Query patterns
- Infinite scroll implementation
- Error handling strategies
- Performance optimization tips

API Coverage:
- Articles API (6 endpoints): list, latest, search, detail, categories
- Outlets API (3 endpoints): list, detail, outlet articles
- Comments API (3 endpoints): list, create, count
- Multi-language support (9 languages)
- Pagination and filtering

Code Examples:
- Copy-paste ready code snippets
- React hooks and components
- Next.js App Router examples
- React Query integration
- Infinite scroll with Intersection Observer
- Client-side caching strategies

Developer Experience:
- TypeScript-first approach
- Practical, executable examples
- Quick start guide
- API reference table
- Error handling patterns
- Performance optimization tips

Target Audience: External frontend developers integrating with News API

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-04 08:33:59 +09:00
de0d548b7a docs: Add comprehensive technical interview guide
- Create TECHNICAL_INTERVIEW.md with 20 technical questions
- Cover Backend (5), Frontend (4), DevOps (6), Data/API (3), Problem Solving (2)
- Include detailed answers with code examples
- Use Obsidian-compatible callout format for collapsible answers
- Add evaluation criteria (Junior/Mid/Senior levels)
- Include practical coding challenge (Comments service)

Technical areas covered:
- API Gateway vs Service Mesh architecture
- FastAPI async/await and Motor vs PyMongo
- Microservice communication (REST, Pub/Sub, gRPC)
- Database strategies and JWT security
- React 18 features and TypeScript integration
- Docker multi-stage builds and K8s deployment strategies
- Health checks, monitoring, and logging
- RESTful API design and MongoDB schema modeling
- Traffic handling and failure scenarios

fix: Update Services.tsx with TypeScript fixes
- Fix ServiceType enum import (use value import, not type-only)
- Fix API method name: checkHealthAll → checkAllHealth
- Ensure proper enum usage in form data

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-03 08:26:00 +09:00
e60e531cdc feat: Phase 2 - Service Management CRUD API (Backend)
Backend Implementation:
- Service model with comprehensive fields (name, url, type, status, health_endpoint)
- Complete CRUD API endpoints for service management
- Health check mechanism with httpx and response time tracking
- Service status tracking (healthy/unhealthy/unknown)
- Service type categorization (backend, frontend, database, cache, etc.)

API Endpoints:
- GET /api/services - Get all services
- POST /api/services - Create new service
- GET /api/services/{id} - Get service by ID
- PUT /api/services/{id} - Update service
- DELETE /api/services/{id} - Delete service
- POST /api/services/{id}/health-check - Check specific service health
- POST /api/services/health-check/all - Check all services health

Frontend Preparation:
- TypeScript type definitions for Service
- Service API client with full CRUD methods
- Health check client methods

Files Added:
- backend/app/models/service.py - Service data model
- backend/app/schemas/service.py - Request/response schemas
- backend/app/services/service_service.py - Business logic
- backend/app/routes/services.py - API route handlers
- frontend/src/types/service.ts - TypeScript types
- frontend/src/api/service.ts - API client

Updated:
- backend/app/main.py - Added services router
- docs/PROGRESS.md - Added Phase 2 status

Next: Frontend UI implementation (Services list page, Add/Edit modal, Health monitoring)

🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-28 16:44:33 +09:00
f4b75b96a5 feat: Phase 1 - Complete authentication system with JWT
Backend Implementation (FastAPI + MongoDB):
- JWT authentication with access/refresh tokens
- User registration and login endpoints
- Password hashing with bcrypt (fixed 72-byte limit)
- Protected endpoints with JWT middleware
- Token refresh mechanism
- Role-Based Access Control (RBAC) structure
- Pydantic v2 models and async MongoDB with Motor
- API endpoints: /api/auth/register, /api/auth/login, /api/auth/me, /api/auth/refresh

Frontend Implementation (React + TypeScript + Material-UI):
- Login and Register pages with validation
- AuthContext for global authentication state
- API client with Axios interceptors for token refresh
- Protected routes with automatic redirect
- User profile display in navigation
- Logout functionality

Technical Achievements:
- Resolved bcrypt 72-byte limit (replaced passlib with native bcrypt)
- Fixed Pydantic v2 compatibility (PyObjectId, ConfigDict)
- Implemented automatic token refresh on 401 errors
- Created comprehensive test suite for all auth endpoints

Docker & Kubernetes:
- Backend image: yakenator/site11-console-backend:latest
- Frontend image: yakenator/site11-console-frontend:latest
- Deployed to site11-pipeline namespace
- Nginx reverse proxy configuration

Documentation:
- CONSOLE_ARCHITECTURE.md - Complete system architecture
- PHASE1_COMPLETION.md - Detailed completion report
- PROGRESS.md - Updated with Phase 1 status

All authentication endpoints tested and verified working.

🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-28 16:23:07 +09:00
07579ea9f5 docs: Add News API deployment guide and SAPIENS services
- Add comprehensive deployment guide in CLAUDE.md
  - Quick deploy commands for News API
  - Version management strategy (Major/Minor/Patch)
  - Rollback procedures
- Add detailed DEPLOYMENT.md for News API service
- Update docker-compose.yml with SAPIENS platform services
  - Add sapiens-web with PostgreSQL (port 3005, 5433)
  - Add sapiens-web2 with PostgreSQL (port 3006, 5434)
  - Configure health checks and dependencies

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-22 09:20:55 +09:00
86ca214dd8 feat: Add source_keyword-based article queries for dynamic outlet articles
- Add get_articles_by_source_keyword method to query articles by entities
- Search across entities.people, entities.organizations, and entities.groups
- Deprecate get_articles_by_ids method in favor of dynamic queries
- Support pagination for outlet article listings

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 16:53:09 +09:00
e467e76d02 feat: Refactor outlets with multilingual support and dynamic queries
- Replace static articles array with dynamic source_keyword queries
- Use MongoDB _id as unique identifier for outlets
- Add multilingual translations (9 languages: ko, en, zh_cn, zh_tw, ja, fr, de, es, it)
- Add OutletService for database operations
- Add outlet migration script with Korean source_keyword matching
- Remove JSON file-based outlet loading
- Add /outlets/{outlet_id}/articles endpoint for dynamic article retrieval

This resolves the design issues with:
1. Static articles array requiring constant updates
2. Lack of multilingual support for outlet names/descriptions
3. Broken image URLs
4. Korean entity matching for article queries

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-13 16:52:34 +09:00
deb52e51f2 feat: Add comment system and outlets data to News API
- Add comment models and service with CRUD operations
- Add comment endpoints (GET, POST, count)
- Add outlets-extracted.json with people/topics/companies data
- Fix database connection in comment_service to use centralized get_database()

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-10 18:52:12 +09:00
68cc70118f fix: Sync News API models with actual MongoDB schema
## 🔧 Model Synchronization
Updated Pydantic models to match actual article structure in MongoDB

### Changes
- **Article Model**: Complete restructure to match MongoDB documents
  - Added Subtopic, Reference, Entities nested models
  - Changed created_at to Union[str, datetime] with serializer
  - Added all pipeline metadata fields (job_id, keyword_id, etc.)
  - Added translation & image fields
  - Changed category (single) to categories (array)

- **ArticleSummary Model**: Updated for list responses
  - Synced with actual MongoDB structure
  - Added news_id, categories array, images array

- **ArticleService**: Fixed category filtering
  - Changed "category" to "categories" (array field)
  - Updated search to include subtopics and source_keyword
  - Implemented MongoDB aggregation for category list

### Verified Fields
 news_id, title, summary, created_at, language
 subtopics (array of {title, content[]})
 categories (array), entities (nested object)
 references (array), source_keyword, source_count
 pipeline_stages, job_id, keyword_id, processing_time
 images (array), image_prompt, translated_languages

### Testing
- Validated with actual English articles (20,966 total)
- Search functionality working (15,298 AI-related articles)
- Categories endpoint returning 1000+ unique categories
- All datetime fields properly serialized to ISO format

🤖 Generated with [Claude Code](https://claude.ai/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-03 17:27:26 +09:00
dca130d300 feat: Add News API service for multi-language article delivery
## 🚀 New Service: News API
Multi-language RESTful API service for serving AI-generated news articles

### Features
- **9 Language Support**: ko, en, zh_cn, zh_tw, ja, fr, de, es, it
- **FastAPI Backend**: Async MongoDB integration with Motor
- **Comprehensive Endpoints**:
  - List articles with pagination
  - Get latest articles
  - Search articles by keyword
  - Get article by ID
  - Get categories by language
- **Production Ready**: Auto-scaling, health checks, K8s deployment

### Technical Stack
- FastAPI 0.104.1 + Uvicorn
- Motor 3.3.2 (async MongoDB driver)
- Pydantic 2.5.0 for data validation
- Docker containerized
- Kubernetes ready with HPA

### API Endpoints
```
GET /api/v1/{lang}/articles          # List articles with pagination
GET /api/v1/{lang}/articles/latest   # Latest articles
GET /api/v1/{lang}/articles/search   # Search articles
GET /api/v1/{lang}/articles/{id}     # Get by ID
GET /api/v1/{lang}/categories        # Get categories
```

### Deployment Options
1. **Local K8s**: `kubectl apply -f k8s/news-api/`
2. **Docker Hub**: `./scripts/deploy-news-api.sh dockerhub`
3. **Kind**: `./scripts/deploy-news-api.sh kind`

### Performance
- Response Time: <50ms (p50), <200ms (p99)
- Auto-scaling: 2-10 pods based on CPU/Memory
- Supports 1000+ req/sec

### Files Added
- services/news-api/backend/ - FastAPI service implementation
- k8s/news-api/ - Kubernetes deployment manifests
- scripts/deploy-news-api.sh - Automated deployment script
- Comprehensive READMEs for service and K8s deployment

🤖 Generated with [Claude Code](https://claude.ai/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-03 17:24:06 +09:00
e3c28f796a Initial commit - cleaned repository 2025-09-28 20:41:57 +09:00