openobserve/scripts/translations

Translation Management

Automated translation system for OpenObserve using AWS Translate.

Overview

This system automatically translates the English locale file (en.json) into multiple languages during the build process. It intelligently preserves existing translations and only translates new keys, making it safe to run repeatedly.

🚀 How It Works

Automatic workflow triggered by en.json changes:

1. Developer updates en.json and pushes to any branch (main, develop, feature branches)
2. GitHub Action detects the change and automatically triggers
3. Translation script runs using AWS Translate to update all language files
4. Changes are committed back to the same branch automatically
5. Build workflows use updated files - all subsequent builds have fresh translations

This means translations are always up-to-date without any manual intervention!

Supported Languages

• 🇹🇷 Turkish (tr)
• 🇨🇳 Chinese (zh)
• 🇫🇷 French (fr)
• 🇪🇸 Spanish (es)
• 🇩🇪 German (de)
• 🇮🇹 Italian (it)
• 🇵🇹 Portuguese (pt)
• 🇯🇵 Japanese (ja)
• 🇰🇷 Korean (ko)
• 🇳🇱 Dutch (nl)

How It Works

1. Source File: All translations originate from web/src/locales/languages/en.json
2. Translation: New keys are automatically translated using AWS Translate
3. Preservation: Existing translations are never overwritten
4. Nested Support: Handles nested JSON structures correctly

Local Development

Prerequisites

1. Python 3.11+
2. AWS Credentials with access to AWS Translate service
3. AWS CLI configured or environment variables set

Setup

# From the web directory
npm run translate:setup

Or manually:

cd scripts/translations
pip3 install -r requirements.txt

Configure AWS Credentials for Local Development

Option 1 - AWS CLI (recommended):

aws configure

Option 2 - IAM Role (if running on EC2/ECS):

# Uses instance profile automatically
# No configuration needed

Option 3 - Environment variables:

export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
export AWS_DEFAULT_REGION=us-east-1

Option 4 - Assume IAM Role locally:

aws sts assume-role \
  --role-arn arn:aws:iam::ACCOUNT_ID:role/GitHubActions-TranslationRole \
  --role-session-name local-translation

# Then export the credentials from the output

Note: GitHub Actions uses OIDC and assumes the IAM role automatically. Local development can use any of the above methods.

Running Translations

Translate all languages:

# From the web directory
npm run translate

# Or directly
cd scripts/translations
python3 main.py

Translate specific languages:

cd scripts/translations
python3 main.py fr es de

GitHub Actions Workflow

Automatic Translation Updates

The workflow (.github/workflows/update-translations.yml) automatically runs when:

• Trigger: Any push that modifies web/src/locales/languages/en.json
• Branches: All branches (**)
• Action:
  1. Runs Python translation script
  2. Updates all language JSON files
  3. Commits changes back to the same branch
  4. Subsequent builds use the updated files

Setup Requirements

The workflow uses IAM Role with OIDC for secure, credential-less authentication.

IAM Role Configuration

The workflow assumes an IAM role via OIDC. The role ARN is not hardcoded — it is read from the AWS_TRANSLATE_ROLE_ARN repository secret so the AWS account ID is not exposed in source control:

• IAM Role: stored in the AWS_TRANSLATE_ROLE_ARN secret
• Region: us-east-1

You can point this secret at the same role used by other OpenObserve workflows (e.g., build-pr-image.yml).

Required IAM Permissions

Ensure the GitHubActionsRole has the following permission for AWS Translate:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "translate:TranslateText"
      ],
      "Resource": "*"
    }
  ]
}

Note: If you need to add this permission, attach a policy to the existing GitHubActionsRole in AWS IAM Console.

GitHub Repository Setup

Add one repository secret so the workflow can authenticate via OIDC without exposing the AWS account ID:

Secret	Value
AWS_TRANSLATE_ROLE_ARN	arn:aws:iam::<ACCOUNT_ID>:role/<RoleName>

Set it under Settings → Secrets and variables → Actions → New repository secret (or via gh secret set AWS_TRANSLATE_ROLE_ARN). The workflow fails fast with a clear error if this secret is missing. No long-lived AWS keys are stored — only the role ARN, which is assumed via GitHub's OIDC provider.

Workflow Behavior

graph TD
    A[Push Code to Branch] --> B[Update Translations Workflow]
    B --> C{en.json Changed?}
    C -->|Yes| D[Run Translation Script]
    C -->|No| E[Skip Translation]
    D --> F[Commit Updated Languages]
    E --> G[Mark Complete]
    F --> G
    G --> H[Build Workflows Start]
    H --> I[Build with Latest Translations]

Workflow Execution Order:

1. Push code → update-translations.yml runs
2. Checks if en.json changed
   • If YES: Translates and commits back to branch
   • If NO: Skips (fast, ~10 seconds)
3. If translations were committed → New push event
4. Build workflows trigger on the new commit → Use updated translations

Key Features:

• ✅ Always runs - Checks every push for en.json changes
• ✅ Smart detection - Only translates if en.json actually changed
• ✅ Non-blocking - Quick skip if no translation needed
• ✅ Auto-commit - Updates committed back to same branch
• ✅ Natural flow - Translation commit triggers builds automatically
• ✅ Works everywhere - All branches (main, develop, feature branches)

Manual Workflow Trigger

You can also run translations manually:

1. Go to Actions tab in GitHub
2. Select Update Translations workflow
3. Click Run workflow
4. (Optional) Specify specific languages: fr es de
5. Translations will be committed to the current branch

File Structure

scripts/translations/
├── README.md           # This file
├── main.py            # Entry point for translation script
├── translator.py      # Core translation logic
└── requirements.txt   # Python dependencies

web/src/locales/languages/
├── en.json           # Source file (English)
├── tr.json           # Turkish translations
├── zh.json           # Chinese translations
├── fr.json           # French translations
├── es.json           # Spanish translations
├── de.json           # German translations
├── it.json           # Italian translations
├── pt.json           # Portuguese translations
├── ja.json           # Japanese translations
├── ko.json           # Korean translations
└── nl.json           # Dutch translations

Adding New Languages

1. Add language code to get_supported_languages() in translator.py
2. Update the README to reflect the new language
3. Run the translation script

Troubleshooting

Workflow Not Detecting en.json Changes

Symptom: You changed en.json but workflow says "en.json not modified"

Solution 1 - Check workflow logs:

1. Go to Actions → Update Translations → Click the run
2. Look at "Check if en.json was modified" step
3. It shows debug info: event type, before/after SHAs, and changed files

Solution 2 - Use force flag:

1. Go to Actions → Update Translations
2. Click Run workflow
3. Check "Force translation even if en.json not detected as changed"
4. Click Run workflow

Solution 3 - Debug detection:

# Locally check what git sees
git show --name-only --pretty="" HEAD
# Should show web/src/locales/languages/en.json

# Check last push
git diff HEAD~1 --name-only

Root cause: The detection uses git diff to compare commits. If:

• Multiple commits in one push → Uses github.event.before and github.sha
• First commit on branch → Uses git show HEAD
• Manual trigger → Uses git show HEAD

AWS Credentials Error

ERROR: No credentials for the translation service.

Solution: IAM role should be automatically assumed via OIDC. Check:

1. Workflow has permissions: id-token: write
2. The AWS_TRANSLATE_ROLE_ARN secret is set and points to a valid role
3. Role has translate:TranslateText permission

Import Error

ModuleNotFoundError: No module named 'boto3'

Solution: Run npm run translate:setup or pip3 install -r requirements.txt

Translation Quality Issues

• Machine translations are not perfect
• Review translations before merging PRs
• Consider manual review for critical UI text
• Native speakers should review translations

Build Has Old Translations

Symptom: Build doesn't have the latest translations

Solution: Check if translation workflow ran and committed:

1. Go to Actions → Find the Update Translations run
2. Check if it detected en.json changes and committed translations
3. If translations were committed, the commit should trigger a new build
4. The new build will have updated translations

Workflow Example

Scenario: Adding New UI Text

1. Developer adds new text to en.json:

   {
     "dashboard": {
       "newFeature": "This is a new feature"
     }
   }
   

2. Merge the en.json change to main:

   git add web/src/locales/languages/en.json
   git commit -m "feat: add new dashboard feature text"
   # open a PR and merge to main
   

3. Workflow automatically (on main only):

   • Triggers because web/src/locales/languages/en.json changed
   • Runs translation script
   • Translates only the new or modified keys to all 10 languages
   • Commits updated fr.json, es.json, etc. plus .translation_state.json

4. Build workflows:

   • Use the newly updated translation files
   • No additional steps needed

Why main only? Running on every feature branch made AWS Translate bill for the same strings repeatedly (per branch, per rebase, again on merge). Gating to main translates each string once, when it actually lands.

Change detection (.translation_state.json)

scripts/translations/.translation_state.json records, per locale, a hash of the English source each translated value was derived from. On every run the script:

• Translates a key only when it is new, missing in a target file, or its English source text changed since the last run (so editing an existing label re-translates it).
• Keeps already-translated text whose source is unchanged — it is never re-sent to AWS, and English is never "translated" to English.
• Prunes keys that were removed from en.json.
• Bootstraps safely: the first run after this file is introduced adopts existing translations as-is (no costly full re-translation, no overwriting manual fixes).

Commit .translation_state.json together with the translation files — it is the source of truth that keeps subsequent runs incremental.

Best Practices

1. Review Commits: Check auto-generated translation commits for accuracy
2. Test in UI: Verify translations display correctly in the application
3. Manual Fixes: Manual edits to a key are preserved until its English source changes
4. Context Matters: Some terms may need manual translation for proper context
5. Land on main: Translations are generated when en.json is merged to main

Cost Considerations

AWS Translate pricing (as of 2024):

• $15 per million characters

Cost Per Workflow Run:

Update Type	Characters	Languages	Total Chars	Cost
Small (10 strings)	500	10	5,000	$0.08
Medium (50 strings)	2,500	10	25,000	$0.38
Large (200 strings)	10,000	10	100,000	$1.50
Full translation	60,000	10	600,000	$9.00

Monthly Estimates:

Active Development:

• 10-15 en.json updates/month
• Average 20 new strings per update
• Monthly cost: $0.80 - $1.20

Production:

• 5-8 en.json updates/month
• Average 15 new strings per update
• Monthly cost: $0.40 - $0.80

First-time full translation: ~$9 (one-time)

Cost Optimization:

• ✅ Only new or modified keys are translated (unchanged text is never re-sent)
• ✅ Runs on main only, and only when en.json changes (no per-branch re-billing)
• ✅ Superseded runs are cancelled (concurrency with cancel-in-progress)
• ✅ Safety cap (TRANSLATION_MAX_KEYS, default 5000) blocks accidental mass re-translation
• ✅ Failed AWS calls are retried next run, not silently billed as English
• ✅ Typical monthly cost: Under $2

Alternative Translation Services

To use a different service, modify translator.py:

• Google Cloud Translate
• DeepL API
• Azure Translator
• OpenAI GPT-4 (for context-aware translations)

Support

For issues or questions:

1. Check this README first
2. Review existing GitHub issues
3. Create a new issue with the translations label