aimond/multilingual-translation
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add complete NLLB-200 support with all 204 FLORES-200 languages

Browse Source 
Updated dual model system to fully support both M2M100 and NLLB-200:

**NLLB-200 Model (204 languages)**
- Added all 204 FLORES-200 language codes to nllb200_lang_codes dictionary
- Updated language code mappings with FLORES-200 format (xxx_Yyyy)
- Added 24+ NLLB-exclusive languages including:
  - Southeast Asian: Acehnese, Balinese, Banjar, Buginese, Minangkabau
  - South Asian: Assamese, Awadhi, Bhojpuri, Chhattisgarhi, Magahi, Maithili, Meitei, Odia, Santali
  - African: Akan, Bambara, Bemba, Chokwe, Dyula, Fon, Kikuyu, Kimbundu, Kongo, Luba-Kasai, Luo, Mossi, Nuer
  - Arabic dialects: Mesopotamian, Najdi, Moroccan, Egyptian, Tunisian, South/North Levantine
  - European regional: Asturian, Friulian, Latgalian, Ligurian, Limburgish, Lombard, Norwegian Nynorsk/Bokmål, Occitan, Sardinian, Sicilian, Silesian, Venetian
  - Other: Dzongkha, Fijian, Guarani, Kabyle, Kabuverdianu, Papiamento, Quechua, Samoan, Sango, Shan, Tamasheq, Tibetan, Tok Pisin

**Updated Files**
- app/translator.py: Complete NLLB-200 language mappings (204 languages)
- app/main.py: Added display names for all 204+ language codes
- README.md: Updated with dual model system, NLLB-200 details, license info
- CLAUDE.md: Updated developer documentation with model architecture

**Testing**
- Verified M2M100: 105 languages working ✅
- Verified NLLB-200: 204 languages working ✅
- Tested NLLB-exclusive languages (Bemba, Fon, etc.) ✅

**License Information**
- M2M100: Apache 2.0 - Commercial use allowed
- NLLB-200: CC-BY-NC 4.0 - Non-commercial only

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
jungwoo choi
2025-11-11 16:19:50 +09:00
parent 5a99d081ab
commit 578be1fd55
4 changed files with 387 additions and 250 deletions
Download Patch File Download Diff File Expand all files Collapse all files

96
CLAUDE.md
View File

@ -4,7 +4,11 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Project Overview
This is a Malaysian language translation API service built with FastAPI and Hugging Face Transformers. It provides bidirectional translation between Malay (Bahasa Melayu) and English using Helsinki-NLP's OPUS-MT neural machine translation models.
This is a multilingual translation API service built with FastAPI and Hugging Face Transformers. It provides any-to-any translation between up to 204 languages using Facebook's M2M100 and NLLB-200 models.
**Dual Model System:**
- **M2M100 (default)**: 105 languages, Apache 2.0 License, commercial use allowed
- **NLLB-200 (optional)**: 204 languages, CC-BY-NC 4.0 License, non-commercial only
## Development Commands
@ -43,17 +47,28 @@ docker-compose up -d --build
```bash
# Health check
curl http://localhost:8000/health
curl http://localhost:8001/health
# Translate Malay to English
curl -X POST "http://localhost:8000/api/translate" \
# Translate Malay to English (M2M100, default)
curl -X POST "http://localhost:8001/api/translate" \
-H "Content-Type: application/json" \
-d '{"text": "Selamat pagi", "source_lang": "ms", "target_lang": "en"}'
# Translate English to Malay
curl -X POST "http://localhost:8000/api/translate" \
# Translate English to Korean (M2M100)
curl -X POST "http://localhost:8001/api/translate" \
-H "Content-Type: application/json" \
-d '{"text": "Good morning", "source_lang": "en", "target_lang": "ms"}'
-d '{"text": "Good morning", "source_lang": "en", "target_lang": "ko", "model": "m2m100"}'
# Translate English to Bemba (NLLB-200 exclusive language)
curl -X POST "http://localhost:8001/api/translate" \
-H "Content-Type: application/json" \
-d '{"text": "Welcome", "source_lang": "en", "target_lang": "bem", "model": "nllb200"}'
# Get supported languages for M2M100
curl http://localhost:8001/api/supported-languages?model=m2m100
# Get supported languages for NLLB-200
curl http://localhost:8001/api/supported-languages?model=nllb200
```
## Architecture
@ -63,18 +78,23 @@ curl -X POST "http://localhost:8000/api/translate" \
1. **app/main.py** - FastAPI application with endpoint definitions
- Lifespan events handle model preloading on startup
- CORS middleware configured for cross-origin requests
- Three main endpoints: root (`/`), health (`/health`), translate (`/api/translate`)
- Main endpoints: root (`/`), health (`/health`), translate (`/api/translate`), supported-languages (`/api/supported-languages`)
- Includes lang_names dictionary with display names for all 204+ language codes
2. **app/translator.py** - Translation service singleton
- Manages loading and caching of translation models
- Manages loading and caching of both M2M100 and NLLB-200 models
- Automatically detects and uses GPU if available (CUDA)
- Supports lazy loading - models are loaded on first use or preloaded at startup
- Model naming convention: `Helsinki-NLP/opus-mt-{source}-{target}`
- Model support:
- M2M100: `facebook/m2m100_418M` (105 languages)
- NLLB-200: `facebook/nllb-200-distilled-600M` (204 languages, FLORES-200 format)
- Language code mapping for both models (m2m100_lang_codes, nllb200_lang_codes)
3. **app/models.py** - Pydantic schemas for request/response validation
- `TranslationRequest`: Validates input (text, source_lang, target_lang)
- `TranslationRequest`: Validates input (text, source_lang, target_lang, model)
- `TranslationResponse`: Structured output with metadata
- `LanguageCode` enum: Only "ms" and "en" are supported
- `HealthResponse`: Health check response
- Model parameter accepts "m2m100" (default) or "nllb200"
4. **app/config.py** - Configuration management using pydantic-settings
- Loads settings from environment variables or `.env` file
@ -83,18 +103,25 @@ curl -X POST "http://localhost:8000/api/translate" \
### Translation Flow
1. Request received at `/api/translate` endpoint
2. Pydantic validates request schema
3. TranslationService determines appropriate model based on language pair
4. Model is loaded if not already cached in memory
5. Text is tokenized, translated, and decoded
6. Response includes original text, translation, and model metadata
2. Pydantic validates request schema (including optional model parameter)
3. TranslationService selects model based on `model` parameter (m2m100 or nllb200)
4. Language codes are validated against the selected model's supported languages
5. Model is loaded if not already cached in memory
6. Text is tokenized with model-specific language codes:
- M2M100: Uses simple codes (e.g., "en", "ko")
- NLLB-200: Uses FLORES-200 format (e.g., "eng_Latn", "kor_Hang")
7. Translation generated using the model
8. Response includes original text, translation, and model metadata
### Model Caching
- Models are downloaded to `MODEL_CACHE_DIR` (default: `./models/`)
- Once downloaded, models persist across restarts
- In Docker, use volume mount to persist models
- First translation request may be slow due to model download (~300MB per model)
- First translation request may be slow due to model download:
- M2M100: ~1.6GB
- NLLB-200: ~2.5GB
- Both models can be cached simultaneously
### Device Selection
@ -112,20 +139,28 @@ Environment variables (see `.env.example`):
## Common Tasks
### Adding New Language Pairs
### Adding New Language Codes
To add support for additional languages:
The system currently supports all 105 M2M100 languages and all 204 NLLB-200 languages. To add new language code mappings:
1. Check if Helsinki-NLP has an OPUS-MT model for the language pair at https://huggingface.co/Helsinki-NLP
2. Update `app/models.py` - Add new language code to `LanguageCode` enum
3. Update `app/translator.py` - Add model mapping in `_get_model_name()` method
4. Update `app/main.py` - Add language info to `/api/supported-languages` endpoint
1. **For M2M100**: Update `m2m100_lang_codes` dictionary in `app/translator.py`
- Format: `"user_code": "m2m100_code"` (e.g., `"en": "en"`)
2. **For NLLB-200**: Update `nllb200_lang_codes` dictionary in `app/translator.py`
- Format: `"user_code": "flores_code"` (e.g., `"en": "eng_Latn"`)
- Reference: https://github.com/facebookresearch/flores/blob/main/flores200/README.md
3. **Display Names**: Add entries to `lang_names` dictionary in `app/main.py`
- Format: `"code": {"name": "English Name", "native": "Native Name"}`
### Modifying Translation Behavior
Translation parameters are in `app/translator.py` in the `translate()` method:
- Adjust `max_length` in tokenizer call to handle longer texts
- Modify generation parameters passed to `model.generate()` for different translation strategies
- Model-specific behavior:
- M2M100: Uses `tokenizer.get_lang_id()` for target language
- NLLB-200: Uses `tokenizer.convert_tokens_to_ids()` for target language
### Production Deployment
@ -139,5 +174,14 @@ For production use:
## API Documentation
When the server is running, interactive API documentation is available at:
- Swagger UI: http://localhost:8000/docs
- ReDoc: http://localhost:8000/redoc
- Swagger UI: http://localhost:8001/docs (Docker) or http://localhost:8000/docs (local)
- ReDoc: http://localhost:8001/redoc (Docker) or http://localhost:8000/redoc (local)
## Model Licenses
**IMPORTANT**: Be aware of licensing when deploying:
- **M2M100**: Apache 2.0 License - Commercial use allowed ✅
- **NLLB-200**: CC-BY-NC 4.0 License - Non-commercial use only ⚠️
Always use M2M100 for commercial applications. Only use NLLB-200 for research, education, or personal non-commercial projects.

155
README.md
View File

@ -1,38 +1,52 @@
# Multilingual Translation API
105개 이상의 언어를 지원하는 다국어 번역 REST API 서비스입니다. 상업적 사용이 가능한 Apache 2.0 라이선스 모델을 사용합니다.
최대 204개 언어를 지원하는 다국어 번역 REST API 서비스입니다. 상업적 사용이 가능한 M2M100 모델(105개 언어)을 기본으로 제공하며, 비상업적 용도로 NLLB-200 모델(204개 언어)을 선택할 수 있습니다.
## 주요 기능
- **105개 이상의 언어 지원** - 모든 언어 간 양방향 번역 가능 (Any-to-Any Translation)
- **상업적 사용 가능** - Apache 2.0 라이선스의 Facebook M2M100 모델 사용
- **고성능 번역** - 418M 파라미터 신경망 기계 번역 모델
- **최대 204개 언어 지원** - 모든 언어 간 양방향 번역 가능 (Any-to-Any Translation)
- **듀얼 모델 시스템**
- **M2M100** (기본): 105개 언어, Apache 2.0 라이선스, **상업적 사용 가능** ✅
- **NLLB-200** (옵션): 204개 언어, CC-BY-NC 4.0 라이선스, 비상업적 용도만 가능
- **고성능 번역** - 최신 신경망 기계 번역 모델
- **FastAPI 기반** - 고성능 비동기 REST API
- **Docker 지원** - 간편한 배포 및 확장
- **자동 API 문서화** - Swagger UI 제공
- **GPU 자동 감지** - CUDA 사용 가능 시 자동으로 GPU 활용
## 지원 언어 (105개)
## 지원 언어
### 주요 언어
English, 中文 (Chinese), Español (Spanish), العربية (Arabic), हिन्दी (Hindi), বাংলা (Bengali), Português (Portuguese), Русский (Russian), 日本語 (Japanese), Deutsch (German), Français (French), 한국어 (Korean), Italiano (Italian), Türkçe (Turkish)
### M2M100 모델 (105개 언어, 기본)
**상업적 사용 가능 ✅ Apache 2.0 License**
### 동남아시아 언어
Bahasa Melayu (Malay), Bahasa Indonesia (Indonesian), ไทย (Thai), Tiếng Việt (Vietnamese), Tagalog, မြန်မာဘာသာ (Burmese), ភាសាខ្មែរ (Khmer), ລາວ (Lao), Filipino, Cebuano, Basa Jawa (Javanese), Basa Sunda (Sundanese)
주요 언어: English, 中文 (Chinese), Español (Spanish), العربية (Arabic), हिन्दी (Hindi), বাংলা (Bengali), Português (Portuguese), Русский (Russian), 日本語 (Japanese), Deutsch (German), Français (French), 한국어 (Korean), Italiano (Italian), Türkçe (Turkish)
### 남아시아 언어
اردو (Urdu), தமிழ் (Tamil), తెలుగు (Telugu), मराठी (Marathi), ગુજરાતી (Gujarati), ಕನ್ನಡ (Kannada), മലയാളം (Malayalam), ਪੰਜਾਬੀ (Punjabi), नेपाली (Nepali), සිංහල (Sinhala)
동남아시아: Bahasa Melayu (Malay), Bahasa Indonesia (Indonesian), ไทย (Thai), Tiếng Việt (Vietnamese), Tagalog, မြန်မာဘာသာ (Burmese), ភាសាខ្មែរ (Khmer), ລາວ (Lao)
### 유럽 언어
Polski (Polish), Nederlands (Dutch), Українська (Ukrainian), Română (Romanian), Svenska (Swedish), Dansk (Danish), Suomi (Finnish), Norsk (Norwegian), Čeština (Czech), Slovenčina (Slovak), Magyar (Hungarian), Български (Bulgarian), Српски (Serbian), Hrvatski (Croatian), Slovenščina (Slovenian), Eesti (Estonian), Latviešu (Latvian), Lietuvių (Lithuanian), Ελληνικά (Greek), עברית (Hebrew), فارسی (Persian), Català (Catalan), Euskara (Basque), Galego (Galician), Cymraeg (Welsh), Gaeilge (Irish), Gàidhlig (Scottish Gaelic), Íslenska (Icelandic), Malti (Maltese)
남아시아: اردو (Urdu), தமிழ் (Tamil), తెలుగు (Telugu), मराठी (Marathi), ગુજરાતી (Gujarati), ಕನ್ನಡ (Kannada), മലയാളം (Malayalam), ਪੰਜਾਬੀ (Punjabi), नेपाली (Nepali), සිංහල (Sinhala)
### 아프리카 언어
Kiswahili (Swahili), አማርኛ (Amharic), Hausa, Igbo, Yorùbá (Yoruba), isiZulu (Zulu), isiXhosa (Xhosa), Afrikaans, Malagasy, Luganda, Lingála (Lingala), chiShona (Shona), Soomaali (Somali), Sesotho, Chichewa
유럽: Polski (Polish), Nederlands (Dutch), Українська (Ukrainian), Română (Romanian), Svenska (Swedish), Dansk (Danish), Suomi (Finnish), Norsk (Norwegian), Čeština (Czech), Slovenčina (Slovak), Magyar (Hungarian), Български (Bulgarian), Српski (Serbian), Hrvatski (Croatian), Slovenščina (Slovenian), Eesti (Estonian), Latviešu (Latvian), Lietuvių (Lithuanian), Ελληνικά (Greek), עברית (Hebrew), فارسی (Persian), Català (Catalan), Euskara (Basque), Galego (Galician), Cymraeg (Welsh), Gaeilge (Irish), Gàidhlig (Scottish Gaelic), Íslenska (Icelandic), Malti (Maltese)
### 기타 언어
Azərbaycan (Azerbaijani), ქართული (Georgian), Қазақша (Kazakh), Oʻzbekcha (Uzbek), Монгол (Mongolian), Shqip (Albanian), Հայերեն (Armenian), Беларуская (Belarusian), Bosanski (Bosnian), Esperanto, Frysk (Frisian), ʻŌlelo Hawaiʻi (Hawaiian), Hmong, Kreyòl ayisyen (Haitian Creole), Kurdî (Kurdish), Кыргызча (Kyrgyz), Latina (Latin), Lëtzebuergesch (Luxembourgish), Te Reo Māori (Maori), Македонски (Macedonian), پښتو (Pashto), Тоҷикӣ (Tajik), Türkmençe (Turkmen), ئۇيغۇرچە (Uyghur), ייִדיש (Yiddish)
아프리카: Kiswahili (Swahili), አማርኛ (Amharic), Hausa, Igbo, Yorùbá (Yoruba), isiZulu (Zulu), isiXhosa (Xhosa), Afrikaans, Malagasy, Luganda, Lingála (Lingala), chiShona (Shona), Soomaali (Somali), Sesotho, Chichewa
**전체 언어 목록은 `/api/supported-languages` 엔드포인트에서 확인하실 수 있습니다.**
기타: Azərbaycan (Azerbaijani), ქართული (Georgian), Қазақша (Kazakh), Oʻzbekcha (Uzbek), Монгол (Mongolian), Shqip (Albanian), Հայերեն (Armenian), Беларуская (Belarusian), Bosanski (Bosnian), Esperanto, Frysk (Frisian), ʻŌlelo Hawaiʻi (Hawaiian), Hmong, Kreyòl ayisyen (Haitian Creole), Kurdî (Kurdish), Кыргызча (Kyrgyz), Latina (Latin), Lëtzebuergesch (Luxembourgish), Te Reo Māori (Maori), Македонски (Macedonian), پښتو (Pashto), Тоҷикӣ (Tajik), Türkmençe (Turkmen), ئۇيغۇرچە (Uyghur), ייִדיש (Yiddish)
### NLLB-200 모델 (204개 언어, 옵션)
**비상업적 용도만 가능 ⚠️ CC-BY-NC 4.0 License**
M2M100의 105개 언어에 더해 다음 99개 추가 언어 지원:
- **동남아시아**: Acehnese, Balinese, Banjar, Buginese, Minangkabau
- **남아시아**: Assamese, Awadhi, Bhojpuri, Chhattisgarhi, Magahi, Maithili, Meitei, Odia, Santali
- **아프리카**: Akan, Bambara, Bemba, Chokwe, Dyula, Fon, Kikuyu, Kimbundu, Kongo, Luba-Kasai, Luo, Mossi, Nuer, Northern Sotho, Tumbuka, Umbundu, Wolof
- **중동/아랍**: Mesopotamian Arabic, Najdi Arabic, Moroccan Arabic, Egyptian Arabic, Tunisian Arabic, South Levantine Arabic, North Levantine Arabic, Dari, Central Kurdish, Sindhi
- **유럽**: Asturian, Friulian, Latgalian, Ligurian, Limburgish, Lombard, Norwegian Nynorsk, Norwegian Bokmål, Occitan, Sardinian, Sicilian, Silesian, Venetian
- **중앙아시아**: Bashkir, Crimean Tatar, South Azerbaijani, Tatar
- **기타**: Dzongkha, Fijian, Guarani, Kabyle, Kabuverdianu, Papiamento, Quechua, Samoan, Sango, Shan, Tamasheq, Tibetan, Tok Pisin
- 그 외 다양한 지역 언어 및 방언
**전체 언어 목록 및 언어 코드는 다음 엔드포인트에서 확인:**
- M2M100: `/api/supported-languages?model=m2m100`
- NLLB-200: `/api/supported-languages?model=nllb200`
## 빠른 시작
@ -76,6 +90,8 @@ python run.py
### 번역 요청
#### M2M100 모델 (기본, 상업적 사용 가능)
```bash
# 말레이어 → 영어
curl -X POST "http://localhost:8001/api/translate" \
@ -86,31 +102,14 @@ curl -X POST "http://localhost:8001/api/translate" \
"target_lang": "en"
}'
# 영어 → 한국어
# 영어 → 한국어 (model 파라미터 생략 시 m2m100이 기본값)
curl -X POST "http://localhost:8001/api/translate" \
-H "Content-Type: application/json" \
-d '{
"text": "Hello, how are you?",
"source_lang": "en",
"target_lang": "ko"
}'
# 영어 → 방글라데시어
curl -X POST "http://localhost:8001/api/translate" \
-H "Content-Type: application/json" \
-d '{
"text": "Good morning",
"source_lang": "en",
"target_lang": "bn"
}'
# 일본어 → 영어
curl -X POST "http://localhost:8001/api/translate" \
-H "Content-Type: application/json" \
-d '{
"text": "こんにちは、元気ですか？",
"source_lang": "ja",
"target_lang": "en"
"target_lang": "ko",
"model": "m2m100"
}'
```
@ -125,15 +124,61 @@ curl -X POST "http://localhost:8001/api/translate" \
}
```
### 지원 언어 확인
#### NLLB-200 모델 (204개 언어, 비상업적 용도)
```bash
curl http://localhost:8001/api/supported-languages
# 영어 → 벵골어 (NLLB-200 사용)
curl -X POST "http://localhost:8001/api/translate" \
-H "Content-Type: application/json" \
-d '{
"text": "Good morning",
"source_lang": "en",
"target_lang": "bn",
"model": "nllb200"
}'
# 영어 → Bemba (NLLB-200 전용 언어)
curl -X POST "http://localhost:8001/api/translate" \
-H "Content-Type: application/json" \
-d '{
"text": "Welcome to our service",
"source_lang": "en",
"target_lang": "bem",
"model": "nllb200"
}'
```
응답:
```json
{
"original_text": "Good morning",
"translated_text": "শুভ সকাল",
"source_lang": "en",
"target_lang": "bn",
"model_used": "facebook/nllb-200-distilled-600M"
}
```
### 지원 언어 확인
```bash
# M2M100 지원 언어 (105개)
curl http://localhost:8001/api/supported-languages?model=m2m100
# NLLB-200 지원 언어 (204개)
curl http://localhost:8001/api/supported-languages?model=nllb200
```
응답:
```json
{
"model": {
"name": "M2M100",
"languages": 105,
"license": "Apache 2.0",
"commercial_use": true,
"model_id": "facebook/m2m100_418M"
},
"languages": [
{
"code": "af",
@ -184,7 +229,9 @@ curl http://localhost:8001/health
- **FastAPI**: 고성능 비동기 웹 프레임워크
- **Transformers (Hugging Face)**: 머신러닝 모델 로딩 및 추론
- **PyTorch**: 딥러닝 프레임워크
- **M2M100 (Facebook)**: 418M 파라미터 다국어 번역 모델
- **번역 모델**:
- **M2M100 (Facebook)**: 418M 파라미터, 105개 언어 (기본)
- **NLLB-200 (Facebook)**: 600M 파라미터, 204개 언어 (옵션)
- **Docker & Docker Compose**: 컨테이너화 및 배포
- **Pydantic**: 데이터 검증 및 설정 관리
- **Uvicorn**: ASGI 서버
@ -197,7 +244,7 @@ site13/
│ ├── __init__.py
│ ├── main.py # FastAPI 애플리케이션 및 엔드포인트
│ ├── models.py # Pydantic 데이터 모델
│ ├── translator.py # 번역 서비스 로직 (M2M100)
│ ├── translator.py # 번역 서비스 로직 (M2M100 + NLLB-200)
│ └── config.py # 설정 관리
├── models/ # 다운로드된 번역 모델 캐시 (자동 생성)
├── requirements.txt # Python 의존성
@ -220,11 +267,13 @@ site13/
## 성능 최적화
- 첫 실행 시 M2M100 모델을 다운로드하므로 시간이 걸릴 수 있습니다 (~1.5GB)
- 첫 실행 시 선택한 모델을 다운로드합니다:
- M2M100: ~1.6GB
- NLLB-200: ~2.5GB
- 모델은 `models/` 디렉토리에 캐시되어 재사용됩니다
- GPU가 있는 경우 자동으로 감지하여 사용합니다
- Docker volume을 사용하여 모델을 영구 저장합니다
- 시작 시 말레이어↔영어 모델을 미리 로드합니다
- 시작 시 번역 모델을 미리 로드하여 첫 요청 응답 시간을 단축합니다
## 개발
@ -240,19 +289,31 @@ docker-compose up -d --build
## 라이선스
### 모델 라이선스
이 프로젝트는 **Facebook M2M100 모델**을 사용합니다.
#### M2M100 모델 (기본)
- **라이선스**: Apache 2.0 License
- **상업적 사용**: ✅ 허용됨
- **상업적 사용**: ✅ **허용됨**
- **모델 크기**: ~1.6GB
- **언어 수**: 105개
- **모델 정보**: https://huggingface.co/facebook/m2m100_418M
#### NLLB-200 모델 (옵션)
- **라이선스**: CC-BY-NC 4.0 License
- **상업적 사용**: ⚠️ **비상업적 용도만 가능**
- **모델 크기**: ~2.5GB
- **언어 수**: 204개
- **모델 정보**: https://huggingface.co/facebook/nllb-200-distilled-600M
- **주의**: 연구, 교육, 개인 프로젝트 등 비상업적 목적으로만 사용 가능합니다.
### 프로젝트 코드
프로젝트 코드는 자유롭게 사용 가능합니다.
## 제한사항
- 텍스트 길이: 1-5000 자
- 모델 크기: ~1.5GB (디스크 공간 필요)
- 모델 크기: M2M100 ~1.6GB, NLLB-200 ~2.5GB (디스크 공간 필요)
- Python 3.11 이하 버전 필요 (PyTorch 호환성)
- NLLB-200 모델은 비상업적 용도로만 사용 가능
## 트러블슈팅

87
app/main.py
View File

@ -245,91 +245,111 @@ async def get_supported_languages(model: str = "m2m100"):
"yi": {"name": "Yiddish", "native": "ייִדיש"},
# Additional NLLB-200 exclusive languages
"ace_arab": {"name": "Acehnese (Arabic script)", "native": "أتشيه"},
"ace": {"name": "Acehnese", "native": "Acèh"},
"acm": {"name": "Mesopotamian Arabic", "native": "عراقي"},
"acq": {"name": "Ta'izzi-Adeni Arabic", "native": "تعزية-عدنية"},
"aeb": {"name": "Tunisian Arabic", "native": "تونسي"},
"ajp": {"name": "South Levantine Arabic", "native": "شامي"},
"als": {"name": "Tosk Albanian", "native": "Toskë"},
"aka": {"name": "Akan", "native": "Akan"},
"apc": {"name": "North Levantine Arabic", "native": "شامي شمالي"},
"ar_latn": {"name": "Arabic (Latin script)", "native": "Arabic (Latin)"},
"ars": {"name": "Najdi Arabic", "native": "نجدي"},
"ary": {"name": "Moroccan Arabic", "native": "الدارجة"},
"arz": {"name": "Egyptian Arabic", "native": "مصري"},
"asm": {"name": "Assamese", "native": "অসমীয়া"},
"as": {"name": "Assamese", "native": "অসমীয়া"},
"ast": {"name": "Asturian", "native": "Asturianu"},
"awa": {"name": "Awadhi", "native": "अवधी"},
"ayr": {"name": "Central Aymara", "native": "Aymar aru"},
"azb": {"name": "South Azerbaijani", "native": "تۆرکجه"},
"bak": {"name": "Bashkir", "native": "Башҡортса"},
"ba": {"name": "Bashkir", "native": "Башҡортса"},
"bam": {"name": "Bambara", "native": "Bamanankan"},
"ban": {"name": "Balinese", "native": "Basa Bali"},
"bem": {"name": "Bemba", "native": "Ichibemba"},
"bho": {"name": "Bhojpuri", "native": "भोजपुरी"},
"bjn_arab": {"name": "Banjar (Arabic script)", "native": "بنجر"},
"bjn": {"name": "Banjar", "native": "Bahasa Banjar"},
"bod": {"name": "Tibetan", "native": "བོད་སྐད་"},
"bo": {"name": "Tibetan", "native": "བོད་སྐད་"},
"bug": {"name": "Buginese", "native": "Basa Ugi"},
"crh": {"name": "Crimean Tatar", "native": "Qırımtatar tili"},
"cjk": {"name": "Chokwe", "native": "Chokwe"},
"ckb": {"name": "Central Kurdish", "native": "کوردیی ناوەندی"},
"dik": {"name": "Southwestern Dinka", "native": "Thuɔŋjäŋ"},
"dyu": {"name": "Dyula", "native": "Jula"},
"dzo": {"name": "Dzongkha", "native": "རྫོང་ཁ"},
"dz": {"name": "Dzongkha", "native": "རྫོང་ཁ"},
"ee": {"name": "Ewe", "native": "Eʋegbe"},
"fo": {"name": "Faroese", "native": "Føroyskt"},
"fj": {"name": "Fijian", "native": "Na Vosa Vakaviti"},
"fon": {"name": "Fon", "native": "Fɔngbe"},
"fur": {"name": "Friulian", "native": "Furlan"},
"fuv": {"name": "Nigerian Fulfulde", "native": "Fulfulde"},
"gaz": {"name": "West Central Oromo", "native": "Oromoo"},
"grn": {"name": "Guarani", "native": "Avañe'ẽ"},
"om": {"name": "West Central Oromo", "native": "Oromoo"},
"gn": {"name": "Guarani", "native": "Avañe'ẽ"},
"hne": {"name": "Chhattisgarhi", "native": "छत्तीसगढ़ी"},
"ilo": {"name": "Iloko", "native": "Ilokano"},
"kab": {"name": "Kabyle", "native": "Taqbaylit"},
"kac": {"name": "Jingpho", "native": "Jinghpaw"},
"kam": {"name": "Kamba", "native": "Kikamba"},
"kas": {"name": "Kashmiri", "native": "कॉशुर"},
"ks": {"name": "Kashmiri", "native": "کٲشُر"},
"ks_deva": {"name": "Kashmiri (Devanagari)", "native": "कॉशुर"},
"kbp": {"name": "Kabiyè", "native": "Kabɩyɛ"},
"kea": {"name": "Kabuverdianu", "native": "Kabuverdianu"},
"khk": {"name": "Halh Mongolian", "native": "Монгол хэл"},
"kin": {"name": "Kinyarwanda", "native": "Ikinyarwanda"},
"ki": {"name": "Kikuyu", "native": "Gĩkũyũ"},
"rw": {"name": "Kinyarwanda", "native": "Ikinyarwanda"},
"kmb": {"name": "Kimbundu", "native": "Kimbundu"},
"knc_arab": {"name": "Kanuri (Arabic script)", "native": "كانوري"},
"knc": {"name": "Kanuri", "native": "Kanuri"},
"kg": {"name": "Kongo", "native": "Kikongo"},
"lij": {"name": "Ligurian", "native": "Ligure"},
"lim": {"name": "Limburgish", "native": "Limburgs"},
"lin": {"name": "Lingala", "native": "Lingála"},
"li": {"name": "Limburgish", "native": "Limburgs"},
"lmo": {"name": "Lombard", "native": "Lombard"},
"ltg": {"name": "Latgalian", "native": "Latgalīšu"},
"lua": {"name": "Luba-Kasai", "native": "Tshiluba"},
"luo": {"name": "Luo", "native": "Dholuo"},
"lus": {"name": "Mizo", "native": "Mizo ṭawng"},
"mag": {"name": "Magahi", "native": "मगही"},
"mai": {"name": "Maithili", "native": "मैथिली"},
"min_arab": {"name": "Minangkabau (Arabic)", "native": "مينڠكاباو"},
"min": {"name": "Minangkabau", "native": "Baso Minangkabau"},
"mni": {"name": "Meitei", "native": "মৈতৈলোন্"},
"mos": {"name": "Mossi", "native": "Mooré"},
"mri": {"name": "Maori", "native": "Te Reo Māori"},
"nn": {"name": "Norwegian Nynorsk", "native": "Nynorsk"},
"nb": {"name": "Norwegian Bokmål", "native": "Bokmål"},
"nso": {"name": "Northern Sotho", "native": "Sesotho sa Leboa"},
"nus": {"name": "Nuer", "native": "Thok Naath"},
"ory": {"name": "Odia", "native": "ଓଡ଼ିଆ"},
"oc": {"name": "Occitan", "native": "Occitan"},
"or": {"name": "Odia", "native": "ଓଡ଼ିଆ"},
"pag": {"name": "Pangasinan", "native": "Pangasinan"},
"pap": {"name": "Papiamento", "native": "Papiamentu"},
"prs": {"name": "Dari", "native": "دری"},
"quy": {"name": "Ayacucho Quechua", "native": "Chanka Qhichwa"},
"run": {"name": "Rundi", "native": "Ikirundi"},
"sag": {"name": "Sango", "native": "Sängö"},
"san": {"name": "Sanskrit", "native": "संस्कृतम्"},
"qu": {"name": "Ayacucho Quechua", "native": "Chanka Qhichwa"},
"rn": {"name": "Rundi", "native": "Ikirundi"},
"sg": {"name": "Sango", "native": "Sängö"},
"sa": {"name": "Sanskrit", "native": "संस्कृतम्"},
"sat": {"name": "Santali", "native": "ᱥᱟᱱᱛᱟᱲᱤ"},
"scn": {"name": "Sicilian", "native": "Sicilianu"},
"shn": {"name": "Shan", "native": "လိၵ်ႈတႆး"},
"srd": {"name": "Sardinian", "native": "Sardu"},
"sm": {"name": "Samoan", "native": "Gagana Sāmoa"},
"sd": {"name": "Sindhi", "native": "سنڌي"},
"sc": {"name": "Sardinian", "native": "Sardu"},
"ss": {"name": "Swazi", "native": "SiSwati"},
"szl": {"name": "Silesian", "native": "Ślōnski"},
"taq": {"name": "Tamasheq", "native": "Tamasheq"},
"tat": {"name": "Tatar", "native": "Татарча"},
"tir": {"name": "Tigrinya", "native": "ትግርኛ"},
"taq_tfng": {"name": "Tamasheq (Tifinagh)", "native": "ⵜⴰⵎⴰⵛⴰⵆ"},
"tt": {"name": "Tatar", "native": "Татарча"},
"ti": {"name": "Tigrinya", "native": "ትግርኛ"},
"tpi": {"name": "Tok Pisin", "native": "Tok Pisin"},
"tsn": {"name": "Tswana", "native": "Setswana"},
"tso": {"name": "Tsonga", "native": "Xitsonga"},
"tn": {"name": "Tswana", "native": "Setswana"},
"ts": {"name": "Tsonga", "native": "Xitsonga"},
"tum": {"name": "Tumbuka", "native": "Chitumbuka"},
"twi": {"name": "Twi", "native": "Twi"},
"tw": {"name": "Twi", "native": "Twi"},
"tzm": {"name": "Central Atlas Tamazight", "native": "ⵜⴰⵎⴰⵣⵉⵖⵜ"},
"uig": {"name": "Uyghur", "native": "ئۇيغۇرچە"},
"umb": {"name": "Umbundu", "native": "Umbundu"},
"vec": {"name": "Venetian", "native": "Vèneto"},
"war": {"name": "Waray", "native": "Winaray"},
"wol": {"name": "Wolof", "native": "Wolof"},
"xho": {"name": "Xhosa", "native": "isiXhosa"},
"ydd": {"name": "Eastern Yiddish", "native": "ייִדיש"},
"yor": {"name": "Yoruba", "native": "Yorùbá"},
"wo": {"name": "Wolof", "native": "Wolof"},
"yue": {"name": "Cantonese", "native": "粵語"},
"zho_hant": {"name": "Chinese (Traditional)", "native": "繁體中文"},
"zh_hant": {"name": "Chinese (Traditional)", "native": "繁體中文"},
}
# Get all supported language codes from translator based on model type
@ -354,11 +374,12 @@ async def get_supported_languages(model: str = "m2m100"):
"model_id": "facebook/m2m100_418M"
},
"nllb200": {
"name": "NLLB-200",
"languages": 200,
"name": "NLLB-200 (FLORES-200)",
"languages": 204,
"license": "CC-BY-NC 4.0",
"commercial_use": False,
"model_id": "facebook/nllb-200-distilled-600M"
"model_id": "facebook/nllb-200-distilled-600M",
"note": "Includes multiple script variants for some languages"
}
}

299
app/translator.py
View File

@ -143,203 +143,214 @@ class TranslationService:
"yi": "yi", # Yiddish
}
# NLLB-200 supported language codes (200 languages)
# NLLB uses different format: xxx_Yyyy (language_Script)
# Full list: https://huggingface.co/facebook/nllb-200-distilled-600M
# NLLB-200 supported language codes (204 languages from FLORES-200)
# NLLB uses format: xxx_Yyyy (language_Script)
# Full list: https://github.com/facebookresearch/flores/blob/main/flores200/README.md
self.nllb200_lang_codes = {
# Major languages
"en": "eng_Latn", # English
"zh": "zho_Hans", # Chinese (Simplified)
"es": "spa_Latn", # Spanish
"ar": "arb_Arab", # Arabic (Standard)
"hi": "hin_Deva", # Hindi
"bn": "ben_Beng", # Bengali
"pt": "por_Latn", # Portuguese
"ru": "rus_Cyrl", # Russian
"ja": "jpn_Jpan", # Japanese
"de": "deu_Latn", # German
"fr": "fra_Latn", # French
"ko": "kor_Hang", # Korean
"it": "ita_Latn", # Italian
"tr": "tur_Latn", # Turkish
"vi": "vie_Latn", # Vietnamese
"th": "tha_Thai", # Thai
"pl": "pol_Latn", # Polish
"nl": "nld_Latn", # Dutch
"uk": "ukr_Cyrl", # Ukrainian
"ro": "ron_Latn", # Romanian
# Southeast Asian languages
"ms": "zsm_Latn", # Malay (Standard)
"id": "ind_Latn", # Indonesian
"tl": "tgl_Latn", # Tagalog
"my": "mya_Mymr", # Burmese
"km": "khm_Khmr", # Khmer
"lo": "lao_Laoo", # Lao
# South Asian languages
"ur": "urd_Arab", # Urdu
"ta": "tam_Taml", # Tamil
"te": "tel_Telu", # Telugu
"mr": "mar_Deva", # Marathi
"gu": "guj_Gujr", # Gujarati
"kn": "kan_Knda", # Kannada
"ml": "mal_Mlym", # Malayalam
"pa": "pan_Guru", # Punjabi
"ne": "npi_Deva", # Nepali
"si": "sin_Sinh", # Sinhala
# European languages
"sv": "swe_Latn", # Swedish
"da": "dan_Latn", # Danish
"fi": "fin_Latn", # Finnish
"no": "nob_Latn", # Norwegian (Bokmål)
"cs": "ces_Latn", # Czech
"sk": "slk_Latn", # Slovak
"hu": "hun_Latn", # Hungarian
"bg": "bul_Cyrl", # Bulgarian
"sr": "srp_Cyrl", # Serbian
"hr": "hrv_Latn", # Croatian
"sl": "slv_Latn", # Slovenian
"et": "est_Latn", # Estonian
"lv": "lvs_Latn", # Latvian
"lt": "lit_Latn", # Lithuanian
"el": "ell_Grek", # Greek
"he": "heb_Hebr", # Hebrew
"fa": "pes_Arab", # Persian
# African languages
"sw": "swh_Latn", # Swahili
"am": "amh_Ethi", # Amharic
"ha": "hau_Latn", # Hausa
"ig": "ibo_Latn", # Igbo
"yo": "yor_Latn", # Yoruba
"zu": "zul_Latn", # Zulu
"xh": "xho_Latn", # Xhosa
"af": "afr_Latn", # Afrikaans
"sn": "sna_Latn", # Shona
"so": "som_Latn", # Somali
# Other languages
"az": "azj_Latn", # Azerbaijani (North)
"ka": "kat_Geor", # Georgian
"kk": "kaz_Cyrl", # Kazakh
"uz": "uzn_Latn", # Uzbek (Northern)
"mn": "khk_Cyrl", # Mongolian (Halh)
"sq": "als_Latn", # Albanian
"hy": "hye_Armn", # Armenian
"be": "bel_Cyrl", # Belarusian
"bs": "bos_Latn", # Bosnian
"ca": "cat_Latn", # Catalan
"ceb": "ceb_Latn", # Cebuano
"cy": "cym_Latn", # Welsh
"eo": "epo_Latn", # Esperanto
"eu": "eus_Latn", # Basque
"gl": "glg_Latn", # Galician
"is": "isl_Latn", # Icelandic
"jv": "jav_Latn", # Javanese
"ku": "kmr_Latn", # Kurdish (Kurmanji)
"ky": "kir_Cyrl", # Kyrgyz
"la": "lat_Latn", # Latin
"lb": "ltz_Latn", # Luxembourgish
"lg": "lug_Latn", # Luganda
"mg": "plt_Latn", # Malagasy
"mk": "mkd_Cyrl", # Macedonian
"mt": "mlt_Latn", # Maltese
"ny": "nya_Latn", # Chichewa
"ps": "pbt_Arab", # Pashto (Southern)
"st": "sot_Latn", # Sesotho
"su": "sun_Latn", # Sundanese
"tg": "tgk_Cyrl", # Tajik
"tk": "tuk_Latn", # Turkmen
"ug": "uig_Arab", # Uyghur
# Additional NLLB-200 exclusive languages (examples, 95 more)
"ace_arab": "ace_Arab", # Acehnese (Arabic script)
"ace": "ace_Latn", # Acehnese
"acm": "acm_Arab", # Mesopotamian Arabic
"acq": "acq_Arab", # Ta'izzi-Adeni Arabic
"aeb": "aeb_Arab", # Tunisian Arabic
"af": "afr_Latn", # Afrikaans
"ajp": "ajp_Arab", # South Levantine Arabic
"als": "als_Latn", # Tosk Albanian
"aka": "aka_Latn", # Akan
"am": "amh_Ethi", # Amharic
"apc": "apc_Arab", # North Levantine Arabic
"ar": "arb_Arab", # Arabic (Standard)
"ar_latn": "arb_Latn", # Arabic (Latin script)
"ars": "ars_Arab", # Najdi Arabic
"ary": "ary_Arab", # Moroccan Arabic
"arz": "arz_Arab", # Egyptian Arabic
"asm": "asm_Beng", # Assamese
"as": "asm_Beng", # Assamese
"ast": "ast_Latn", # Asturian
"awa": "awa_Deva", # Awadhi
"ayr": "ayr_Latn", # Central Aymara
"azb": "azb_Arab", # South Azerbaijani
"bak": "bak_Cyrl", # Bashkir
"az": "azj_Latn", # Azerbaijani
"ba": "bak_Cyrl", # Bashkir
"bam": "bam_Latn", # Bambara
"ban": "ban_Latn", # Balinese
"be": "bel_Cyrl", # Belarusian
"bem": "bem_Latn", # Bemba
"bn": "ben_Beng", # Bengali
"bho": "bho_Deva", # Bhojpuri
"bjn_arab": "bjn_Arab", # Banjar (Arabic script)
"bjn": "bjn_Latn", # Banjar
"bod": "bod_Tibt", # Tibetan
"bo": "bod_Tibt", # Tibetan
"bs": "bos_Latn", # Bosnian
"bug": "bug_Latn", # Buginese
"crh": "crh_Latn", # Crimean Tatar
"bg": "bul_Cyrl", # Bulgarian
"ca": "cat_Latn", # Catalan
"ceb": "ceb_Latn", # Cebuano
"cs": "ces_Latn", # Czech
"cjk": "cjk_Latn", # Chokwe
"ckb": "ckb_Arab", # Central Kurdish
"crh": "crh_Latn", # Crimean Tatar
"cy": "cym_Latn", # Welsh
"da": "dan_Latn", # Danish
"de": "deu_Latn", # German
"dik": "dik_Latn", # Southwestern Dinka
"dyu": "dyu_Latn", # Dyula
"dzo": "dzo_Tibt", # Dzongkha
"dz": "dzo_Tibt", # Dzongkha
"el": "ell_Grek", # Greek
"en": "eng_Latn", # English
"eo": "epo_Latn", # Esperanto
"et": "est_Latn", # Estonian
"eu": "eus_Latn", # Basque
"ee": "ewe_Latn", # Ewe
"fo": "fao_Latn", # Faroese
"fj": "fij_Latn", # Fijian
"fi": "fin_Latn", # Finnish
"fon": "fon_Latn", # Fon
"fr": "fra_Latn", # French
"fur": "fur_Latn", # Friulian
"fuv": "fuv_Latn", # Nigerian Fulfulde
"gaz": "gaz_Latn", # West Central Oromo
"grn": "grn_Latn", # Guarani
"om": "gaz_Latn", # West Central Oromo
"gd": "gla_Latn", # Scottish Gaelic
"ga": "gle_Latn", # Irish
"gl": "glg_Latn", # Galician
"gn": "grn_Latn", # Guarani
"gu": "guj_Gujr", # Gujarati
"ht": "hat_Latn", # Haitian Creole
"ha": "hau_Latn", # Hausa
"he": "heb_Hebr", # Hebrew
"hi": "hin_Deva", # Hindi
"hne": "hne_Deva", # Chhattisgarhi
"hr": "hrv_Latn", # Croatian
"hu": "hun_Latn", # Hungarian
"hy": "hye_Armn", # Armenian
"ig": "ibo_Latn", # Igbo
"ilo": "ilo_Latn", # Iloko
"id": "ind_Latn", # Indonesian
"is": "isl_Latn", # Icelandic
"it": "ita_Latn", # Italian
"jv": "jav_Latn", # Javanese
"ja": "jpn_Jpan", # Japanese
"kab": "kab_Latn", # Kabyle
"kac": "kac_Latn", # Jingpho
"kam": "kam_Latn", # Kamba
"kas": "kas_Arab", # Kashmiri
"kn": "kan_Knda", # Kannada
"ks": "kas_Arab", # Kashmiri (Arabic)
"ks_deva": "kas_Deva", # Kashmiri (Devanagari)
"ka": "kat_Geor", # Georgian
"kk": "kaz_Cyrl", # Kazakh
"kbp": "kbp_Latn", # Kabiyè
"kea": "kea_Latn", # Kabuverdianu
"khk": "khk_Cyrl", # Halh Mongolian
"kin": "kin_Latn", # Kinyarwanda
"mn": "khk_Cyrl", # Mongolian (Halh)
"km": "khm_Khmr", # Khmer
"ki": "kik_Latn", # Kikuyu
"rw": "kin_Latn", # Kinyarwanda
"ky": "kir_Cyrl", # Kyrgyz
"kmb": "kmb_Latn", # Kimbundu
"ku": "kmr_Latn", # Kurdish (Kurmanji)
"knc_arab": "knc_Arab", # Kanuri (Arabic script)
"knc": "knc_Latn", # Kanuri
"kg": "kon_Latn", # Kongo
"ko": "kor_Hang", # Korean
"lo": "lao_Laoo", # Lao
"lij": "lij_Latn", # Ligurian
"lim": "lim_Latn", # Limburgish
"lin": "lin_Latn", # Lingala
"li": "lim_Latn", # Limburgish
"ln": "lin_Latn", # Lingala
"lt": "lit_Latn", # Lithuanian
"lmo": "lmo_Latn", # Lombard
"ltg": "ltg_Latn", # Latgalian
"lb": "ltz_Latn", # Luxembourgish
"lua": "lua_Latn", # Luba-Kasai
"lg": "lug_Latn", # Luganda
"luo": "luo_Latn", # Luo
"lus": "lus_Latn", # Mizo
"lv": "lvs_Latn", # Latvian
"mag": "mag_Deva", # Magahi
"mai": "mai_Deva", # Maithili
"ml": "mal_Mlym", # Malayalam
"mr": "mar_Deva", # Marathi
"min_arab": "min_Arab", # Minangkabau (Arabic)
"min": "min_Latn", # Minangkabau
"mk": "mkd_Cyrl", # Macedonian
"mt": "mlt_Latn", # Maltese
"mni": "mni_Beng", # Meitei
"mos": "mos_Latn", # Mossi
"mri": "mri_Latn", # Maori
"mi": "mri_Latn", # Maori
"my": "mya_Mymr", # Burmese
"nl": "nld_Latn", # Dutch
"nn": "nno_Latn", # Norwegian Nynorsk
"nb": "nob_Latn", # Norwegian Bokmål
"ne": "npi_Deva", # Nepali
"nso": "nso_Latn", # Northern Sotho
"nus": "nus_Latn", # Nuer
"ory": "ory_Orya", # Odia
"ny": "nya_Latn", # Chichewa
"oc": "oci_Latn", # Occitan
"or": "ory_Orya", # Odia
"pag": "pag_Latn", # Pangasinan
"pa": "pan_Guru", # Punjabi
"pap": "pap_Latn", # Papiamento
"ps": "pbt_Arab", # Pashto (Southern)
"fa": "pes_Arab", # Persian
"mg": "plt_Latn", # Malagasy
"pl": "pol_Latn", # Polish
"pt": "por_Latn", # Portuguese
"prs": "prs_Arab", # Dari
"quy": "quy_Latn", # Ayacucho Quechua
"run": "run_Latn", # Rundi
"sag": "sag_Latn", # Sango
"san": "san_Deva", # Sanskrit
"sat": "sat_Beng", # Santali
"qu": "quy_Latn", # Ayacucho Quechua
"ro": "ron_Latn", # Romanian
"rn": "run_Latn", # Rundi
"ru": "rus_Cyrl", # Russian
"sg": "sag_Latn", # Sango
"sa": "san_Deva", # Sanskrit
"sat": "sat_Olck", # Santali
"scn": "scn_Latn", # Sicilian
"shn": "shn_Mymr", # Shan
"srd": "srd_Latn", # Sardinian
"si": "sin_Sinh", # Sinhala
"sk": "slk_Latn", # Slovak
"sl": "slv_Latn", # Slovenian
"sm": "smo_Latn", # Samoan
"sn": "sna_Latn", # Shona
"sd": "snd_Arab", # Sindhi
"so": "som_Latn", # Somali
"st": "sot_Latn", # Sesotho
"es": "spa_Latn", # Spanish
"sq": "als_Latn", # Albanian (Tosk)
"sc": "srd_Latn", # Sardinian
"sr": "srp_Cyrl", # Serbian
"ss": "ssw_Latn", # Swazi
"su": "sun_Latn", # Sundanese
"sv": "swe_Latn", # Swedish
"sw": "swh_Latn", # Swahili
"szl": "szl_Latn", # Silesian
"taq": "taq_Latn", # Tamasheq
"tat": "tat_Cyrl", # Tatar
"tir": "tir_Ethi", # Tigrinya
"ta": "tam_Taml", # Tamil
"taq": "taq_Latn", # Tamasheq (Latin)
"taq_tfng": "taq_Tfng", # Tamasheq (Tifinagh)
"tt": "tat_Cyrl", # Tatar
"te": "tel_Telu", # Telugu
"tg": "tgk_Cyrl", # Tajik
"tl": "tgl_Latn", # Tagalog
"th": "tha_Thai", # Thai
"ti": "tir_Ethi", # Tigrinya
"tpi": "tpi_Latn", # Tok Pisin
"tsn": "tsn_Latn", # Tswana
"tso": "tso_Latn", # Tsonga
"tn": "tsn_Latn", # Tswana
"ts": "tso_Latn", # Tsonga
"tk": "tuk_Latn", # Turkmen
"tum": "tum_Latn", # Tumbuka
"twi": "twi_Latn", # Twi
"tr": "tur_Latn", # Turkish
"tw": "twi_Latn", # Twi
"tzm": "tzm_Tfng", # Central Atlas Tamazight
"uig": "uig_Arab", # Uyghur
"ug": "uig_Arab", # Uyghur
"uk": "ukr_Cyrl", # Ukrainian
"umb": "umb_Latn", # Umbundu
"ur": "urd_Arab", # Urdu
"uz": "uzn_Latn", # Uzbek (Northern)
"vec": "vec_Latn", # Venetian
"vi": "vie_Latn", # Vietnamese
"war": "war_Latn", # Waray
"wol": "wol_Latn", # Wolof
"xho": "xho_Latn", # Xhosa
"ydd": "ydd_Hebr", # Eastern Yiddish
"yor": "yor_Latn", # Yoruba
"wo": "wol_Latn", # Wolof
"xh": "xho_Latn", # Xhosa
"yi": "ydd_Hebr", # Eastern Yiddish
"yo": "yor_Latn", # Yoruba
"yue": "yue_Hant", # Cantonese
"zho_hant": "zho_Hant", # Chinese (Traditional)
"zh": "zho_Hans", # Chinese (Simplified)
"zh_hant": "zho_Hant", # Chinese (Traditional)
"ms": "zsm_Latn", # Malay (Standard)
"zu": "zul_Latn", # Zulu
}
def get_supported_languages(self, model_type: str = "m2m100") -> Dict[str, str]: