Specialized pt-BR embeddings

This report presents benchmark evidence for mk-embeddings-pt, an embedding model specialized for Brazilian Portuguese (pt-BR). The evaluation uses standard MTEB benchmarks built by Brazilian academic institutions, comparing our model against the multilingual baseline multilingual-e5-large on identical tasks and hardware. The goal is to demonstrate that mk-embeddings-pt is genuinely specialized for pt-BR — not merely a multilingual model with incidental Portuguese coverage — and that its Portuguese semantic quality is equivalent to or better than what English-native embeddings achieve in English.

​

Results summary

mk-embeddings-pt scores +12.7% higher on Portuguese than English, confirming genuine pt-BR specialization. multilingual-e5-large scores −1.1% on Portuguese vs English, confirming it is English-biased.

​

Benchmark methodology

All evaluations use the MTEB (Massive Text Embedding Benchmark) framework, the standard for embedding model evaluation. Both models were evaluated on identical hardware (Apple M-series, MPS backend) with the same MTEB task configs.

​

Portuguese benchmarks

​

English benchmarks (for specialization comparison)

​

Portuguese benchmark results

​

Semantic textual similarity

STS tasks measure how well embeddings capture semantic similarity between sentence pairs. These are the most direct measure of embedding quality for retrieval and RAG applications. On the two native Brazilian Portuguese STS benchmarks (SICK-BR and Assin2), mk-embeddings-pt outperforms the multilingual baseline by +4.9 to +14.2 points. The STSBenchmarkMultilingualSTS result favors multilingual-e5-large because this benchmark is a machine-translated version of the English STS Benchmark — multilingual models trained on English STS data have an inherent advantage here. The native pt-BR benchmarks (SICK-BR, Assin2) are more representative of real Portuguese semantic understanding.

​

Pair classification and entailment

mk-embeddings-pt is substantially better at recognizing textual entailment in pt-BR — a critical capability for RAG systems that need to determine whether a retrieved passage actually supports a claim.

​

Classification

multilingual-e5-large leads on the MASSIVE classification tasks. These tasks test cross-lingual transfer from English training data — an area where large multilingual models have an inherent advantage due to their training distribution. However, classification accuracy is not the primary requirement for a retrieval-focused embedding model. For the Brazilian-specific task (toxic tweets), both models score similarly, with mk-embeddings-pt marginally ahead.

​

Language specialization analysis

The specialization delta — the difference between a model’s Portuguese STS score and its English STS score — is the key indicator of whether a model is genuinely specialized for Portuguese or merely multilingual with English bias.

​

English STS baselines

​

Specialization delta

mk-embeddings-pt sacrifices English performance to achieve superior Portuguese quality. This is the expected signature of a genuinely specialized model — it performs best in its target language and intentionally trades off performance in other languages. multilingual-e5-large shows the opposite pattern: it is marginally better on English than Portuguese, confirming that it is a general-purpose multilingual model, not a Portuguese specialist.

​

Cross-language parity

A key requirement is that pt-BR embedding quality should be equivalent to what English-native embeddings achieve in English. The Portuguese STS average for mk-embeddings-pt (0.8088) is within 1 point of the English STS average for multilingual-e5-large (0.8170). This demonstrates cross-language parity — Brazilian Portuguese users get embedding quality equivalent to what English users expect.

​

Comparison with published Portuguese benchmarks

The Serafim paper (Santos et al., 2024) provides additional context for Portuguese embedding performance. Published scores on overlapping benchmarks: mk-embeddings-pt outperforms published multilingual and English-only baselines by +11.5 to +23.5 points on the native Brazilian Portuguese ASSIN2 benchmark.

​

Training data

All training and evaluation data comes from publicly available Brazilian academic benchmarks. ASSIN and ASSIN2 are the standard benchmarks for Brazilian Portuguese semantic understanding, produced by the Interinstitutional Center for Computational Linguistics (NILC) at the University of São Paulo.

​

Sovereign AI compliance

The model can be further fine-tuned on domain-specific data (legal, government, regulatory) without exposing classified documents to external services.

​

Recommendations

​

For retrieval and RAG

Use mk-embeddings-pt for all Portuguese retrieval pipelines. The +14.2 point advantage on SICK-BR-STS and +6.2 point advantage on Assin2RTE translate directly to better retrieval relevance and more accurate entailment detection in RAG systems.

​

For classification

For intent classification and scenario classification tasks, multilingual-e5-large remains stronger due to its larger multilingual training distribution. Consider using a hybrid approach: mk-embeddings-pt for retrieval and a separate classifier for categorization.

​

For further specialization

The model can be fine-tuned on domain-specific Brazilian Portuguese data using CoSENT loss for STS optimization or contrastive learning for retrieval. Recommended domains for government deployment: legal texts, regulatory documents, public service workflows.

​

References

1. Real et al., 2020. “The ASSIN 2 Shared Task: a Portuguese Semantic Similarity Evaluation”
2. Santos et al., 2024. “Serafim: Portuguese Sentence Embeddings” (arXiv:2407.19527)
3. Enevoldsen et al., 2025. “MMTEB: Massive Multilingual Text Embedding Benchmark” (arXiv:2502.13595)
4. Wang et al., 2024. “Multilingual E5 Text Embeddings” (arXiv:2402.05672)
5. Souza et al., 2020. “BERTimbau: Pretrained BERT Models for Brazilian Portuguese”
6. MTEB Leaderboard — https://huggingface.co/spaces/mteb/leaderboard