10 Watts, 46 Languages: The Case for Right-Sized AI

Abstract

Model size has become the default proxy for capability. For government and enterprise workloads, which are specific, repetitive and sensitive, the more useful criterion is fit: how well a model matches its task, its data and the place it has to run. We report evaluation results for two of the smallest models in the Arabic.AI family. LLM-T (5B parameters) reaches COMET 94.0 Arabic→English and 91.0 English→Arabic on FLORES, and 93.80 English→Arabic on WMT24++, its strongest of eight measured directions. LLM-N (1B parameters) covers 46 languages with a macro COMET of 88.58 from English and 88.01 into English across 90 FLORES-200 directions, translates 91,080 sentences in 1,058 seconds on a single workstation, and runs at about 15 tokens per second on a device capped at 10 watts, within 0.1 COMET of its workstation quality. We discuss what these results mean for right-sizing models in sovereign deployments.

1. Introduction

Most enterprise and government AI workloads look nothing like an open-ended chat. They are the same document types, the same language pairs and the same terminology, repeated thousands of times a day, often inside infrastructure the organization controls.

For this kind of work, a model’s value shows up as consistency, latency, cost and control. Recent research makes the same case for agentic systems: agents perform a small number of specialized tasks repeatedly, and small models fine-tuned for those tasks can do the work efficiently, with larger models reserved for open-ended reasoning [2].

Arabic.AI builds models across a wide range of sizes (Figure 1). This note focuses on the two smallest, both translation models, and asks a practical question: what can a right-sized model deliver?

2. Models

Table 1. Models at a glance

LLM-T is built for Arabic-centric production translation. Its 12 high-accuracy languages are Arabic, Chinese, English, French, German, Hindi, Italian, Japanese, Korean, Portuguese, Russian and Spanish. Custom glossaries apply client terminology consistently across domains.

LLM-N is built for coverage and efficiency: English paired with 45 languages, small enough to run on edge hardware.

3. Evaluation setup

• Metric. COMET on a 0 to 100 scale. LLM-N uses the wmt22-comet-da checkpoint.
• LLM-T. FLORES (Arabic↔English) and WMT24++ (8 directions from English), plus an internal Arabic dialect benchmark.
• LLM-N. FLORES-200 devtest, 1,012 sentences per direction, 90 directions (45 languages × 2), greedy decoding.
• Throughput. LLM-N was timed on the full 91,080-sentence run on one workstation. The on-device quality check compares the same sample sentences on device and on the workstation; it is not a comparison of full 1,012-sentence averages.

4. Results

4.1 LLM-T: depth in Arabic

On FLORES, LLM-T scores COMET 94.0 Arabic→English and 91.0 English→Arabic. On WMT24++, Arabic is its strongest target among eight directions from English, at 93.80, followed by German at 90.61 (Figure 2). The model performs particularly well on colloquial Arabic in our internal dialect benchmark.

4.2 Arabic across both models 

Figure 3 places both models’ FLORES Arabic scores side by side. The models were evaluated under different setups, so these gaps are indicative rather than a controlled head-to-head comparison. LLM-T’s reported Arabic scores are about 4 to 7 COMET points higher, consistent with its design goal. LLM-N holds Arabic at 87.20 into English and 86.73 from English while covering 44 other languages.

4.3 LLM-N: breadth across 45 languages

LLM-N scores a macro COMET of 88.58 from English and 88.01 into English. Into English, quality is even across all 45 languages, ranging from 85.98 (Burmese) to 90.34 (Danish). From English, scores range from 80.99 (Hindi) to 91.87 (Indonesian) (Figures 4 and 5). 

Figure 5. LLM-N per-language COMET, from and into English (FLORES-200 devtest). Reproduced from the Arabic.AI technical report.

The four lowest from-English targets (Hindi, Urdu, Khmer, Lao) translate into English at 86.8 to 89.3. This suggests the gap lies in generating these languages rather than understanding them, which points to where future training effort is best spent.

4.4 LLM-N: throughput and on-device deployment

On a single workstation, LLM-N translated the full 91,080-sentence run in 1,058 seconds (about 18 minutes), producing 3,242,332 output tokens at 3,064 tokens per second, or 86.1 sentences per second. Only 9 of 91,080 outputs reached the length limit.

On a device with power capped at 10 watts, the 46-language model runs at about 15 tokens per second at full precision. On the sample sentences evaluated on both systems, on-device quality stays within 0.1 COMET of the workstation (Figure 6). This is a sample comparison, not a comparison of full 1,012-sentence averages. The speed figures use different hardware and batch settings and are not a controlled head-to-head comparison.

5. Discussion: right-sizing for sovereign deployment

The two models answer different questions, and that is the point.

Table 4. Matching workloads to models

For governments and regulated enterprises, these properties are what make sovereignty practical. A model that fits on a workstation or a 10-watt device can run in a private cloud, on premises, or fully offline. Control over where the model runs, who holds the data and who answers for the outcome then stays with the organization. Larger models remain essential for open-ended reasoning. The strongest architecture treats models as a team: the right model for each task, all under the organization’s control.

6. Limitations

• The two models were evaluated under different setups, so cross-model gaps (Figure 3) are indicative until LLM-T is rerun under LLM-N’s protocol.
• Speed figures were measured on different hardware and batch settings. They describe each model’s own run and are not a head-to-head comparison.
• Per-dialect scores for LLM-T and scores for its 40 or more additional languages are not yet reported.
• The on-device result is a sample comparison against the workstation, not against the full 1,012-sentence average.

7. Conclusion

A 5B model reaching COMET 94.0 on Arabic→English, and a 1B model covering 46 languages while running at about 15 tokens per second on a device capped at 10 watts, illustrate the case for right-sizing AI to a well-defined task. In the reported sample comparison, the on-device model stays within 0.1 COMET of workstation quality. For sovereign deployments, the practical choice is the model that meets the workload’s quality, speed and infrastructure requirements—not simply the largest model available.

References 

1. Arabic.AI. LLM-T and LLM-N Translation Models: Technical Report. October 2026.
2. Belcak, P. et al. Small Language Models are the Future of Agentic AI. NVIDIA Research, 2025. https://arxiv.org/abs/2506.02153
3. Goyal, N. et al. The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation. TACL, 2022. FLORES-200 extension, 2022.
4. Rei, R. et al. COMET: A Neural Framework for MT Evaluation. EMNLP, 2020.

Related reading: Enterprises Do Not Adopt an AI Model: They Adopt a Workflow · Sovereignty Is a Measurement Problem