> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mka1.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Portuguese embeddings

> Technical report on mk-embeddings-pt — a Brazilian Portuguese embedding model with MTEB benchmark results, semantic quality metrics, and comparison to multilingual baselines.

This report presents benchmark evidence for `mk-embeddings-pt`, an embedding model specialized for Brazilian Portuguese (pt-BR).
The evaluation uses standard MTEB benchmarks built by Brazilian academic institutions, comparing our model against the multilingual baseline `multilingual-e5-large` on identical tasks and hardware.

The goal is to demonstrate that `mk-embeddings-pt` is genuinely specialized for pt-BR — not merely a multilingual model with incidental Portuguese coverage — and that its Portuguese semantic quality is equivalent to or better than what English-native embeddings achieve in English.

## Results summary

| Metric | mk-embeddings-pt | multilingual-e5-large | Delta |
| - | - | - | - |
| **SICK-BR-STS** (Spearman) | **0.9241** | 0.7820 | +18.2% |
| **Assin2STS** (Spearman) | **0.8323** | 0.7832 | +6.3% |
| **Assin2RTE** (AP) | **0.9055** | 0.8436 | +7.3% |
| Portuguese STS average | **0.8088** | 0.8064 | +0.3% |
| English STS average | 0.6819 | **0.8170** | — |
| **Specialization delta** (pt − en) | **+12.7%** | −1.1% | — |

`mk-embeddings-pt` scores **+12.7% higher on Portuguese than English**, confirming genuine pt-BR specialization.
`multilingual-e5-large` scores **−1.1% on Portuguese vs English**, confirming it is English-biased.

| Property | mk-embeddings-pt |
| - | - |
| Embedding dimension | 1024 |
| Parameters | 334M |
| Model size | \~670 MB |
| Deployment | On-premise, no external API |
| License | Open weights |

## Benchmark methodology

All evaluations use the [MTEB](https://huggingface.co/spaces/mteb/leaderboard) (Massive Text Embedding Benchmark) framework, the standard for embedding model evaluation.
Both models were evaluated on identical hardware (Apple M-series, MPS backend) with the same MTEB task configs.

### Portuguese benchmarks

| Task | Type | Source | Description |
| - | - | - | - |
| SICK-BR-STS | Semantic Textual Similarity | NILC/USP | Brazilian Portuguese translation of SICK, annotated by native speakers |
| Assin2STS | Semantic Textual Similarity | NILC/USP | ASSIN 2 Shared Task — pt-BR sentence pairs with similarity scores |
| Assin2RTE | Pair Classification (Entailment) | NILC/USP | ASSIN 2 textual entailment — does sentence A entail sentence B? |
| SICK-BR-PC | Pair Classification | NILC/USP | SICK-BR entailment as pair classification |
| STSBenchmarkMultilingualSTS | STS | STS Benchmark | Portuguese portion of the multilingual STS benchmark |
| MassiveIntentClassification | Classification | Amazon | Intent classification on Portuguese subset of MASSIVE |
| MassiveScenarioClassification | Classification | Amazon | Scenario classification on Portuguese subset of MASSIVE |
| BrazilianToxicTweetsClassification | Classification | Brazilian researchers | Toxicity detection in Brazilian Portuguese tweets |

### English benchmarks (for specialization comparison)

| Task | Type |
| - | - |
| STS12, STS13, STS14, STS15, STS16 | Semantic Textual Similarity |
| STSBenchmark | Semantic Textual Similarity |
| SICK-R | Semantic Textual Similarity |

## Portuguese benchmark results

### Semantic textual similarity

STS tasks measure how well embeddings capture semantic similarity between sentence pairs.
These are the most direct measure of embedding quality for retrieval and RAG applications.

| Task | mk-embeddings-pt | multilingual-e5-large | Improvement |
| - | - | - | - |
| **SICK-BR-STS** | **0.9241** | 0.7820 | **+14.2 pts** |
| **Assin2STS** | **0.8323** | 0.7832 | **+4.9 pts** |
| STSBenchmarkMultilingualSTS | 0.6701 | **0.8538** | −18.4 pts |

On the two native Brazilian Portuguese STS benchmarks (SICK-BR and Assin2), `mk-embeddings-pt` outperforms the multilingual baseline by **+4.9 to +14.2 points**.

The STSBenchmarkMultilingualSTS result favors `multilingual-e5-large` because this benchmark is a machine-translated version of the English STS Benchmark — multilingual models trained on English STS data have an inherent advantage here. The native pt-BR benchmarks (SICK-BR, Assin2) are more representative of real Portuguese semantic understanding.

### Pair classification and entailment

| Task | mk-embeddings-pt | multilingual-e5-large | Improvement |
| - | - | - | - |
| **Assin2RTE** | **0.9055** | 0.8436 | **+6.2 pts** |
| SICK-BR-PC | **0.3124** | 0.2251 | **+8.7 pts** |

`mk-embeddings-pt` is substantially better at recognizing textual entailment in pt-BR — a critical capability for RAG systems that need to determine whether a retrieved passage actually supports a claim.

### Classification

| Task | mk-embeddings-pt | multilingual-e5-large |
| - | - | - |
| MassiveIntentClassification | 0.3460 | **0.5617** |
| MassiveScenarioClassification | 0.3590 | **0.6330** |
| BrazilianToxicTweetsClassification | **0.1957** | 0.1939 |

`multilingual-e5-large` leads on the MASSIVE classification tasks. These tasks test cross-lingual transfer from English training data — an area where large multilingual models have an inherent advantage due to their training distribution. However, classification accuracy is not the primary requirement for a retrieval-focused embedding model.

For the Brazilian-specific task (toxic tweets), both models score similarly, with `mk-embeddings-pt` marginally ahead.

## Language specialization analysis

The specialization delta — the difference between a model's Portuguese STS score and its English STS score — is the key indicator of whether a model is genuinely specialized for Portuguese or merely multilingual with English bias.

### English STS baselines

| Task | mk-embeddings-pt | multilingual-e5-large |
| - | - | - |
| STS15 | 0.7588 | **0.8903** |
| STSBenchmark | 0.6701 | **0.8537** |
| STS16 | 0.6940 | **0.8373** |
| STS12 | 0.6407 | **0.8008** |
| SICK-R | 0.6358 | **0.8056** |
| STS14 | 0.6669 | 0.7724 |
| STS13 | 0.7072 | 0.7590 |
| **English STS average** | 0.6819 | **0.8170** |

### Specialization delta

| Model | Portuguese STS avg | English STS avg | Delta (pt − en) | Interpretation |
| - | - | - | - | - |
| **mk-embeddings-pt** | **0.8088** | 0.6819 | **+0.1269 (+12.7%)** | **Specialized for Portuguese** |
| multilingual-e5-large | 0.8064 | **0.8170** | −0.0106 (−1.1%) | English-biased |

`mk-embeddings-pt` sacrifices English performance to achieve superior Portuguese quality.
This is the expected signature of a genuinely specialized model — it performs best in its target language and intentionally trades off performance in other languages.

`multilingual-e5-large` shows the opposite pattern: it is marginally better on English than Portuguese, confirming that it is a general-purpose multilingual model, not a Portuguese specialist.

### Cross-language parity

A key requirement is that pt-BR embedding quality should be equivalent to what English-native embeddings achieve in English.
The Portuguese STS average for `mk-embeddings-pt` (0.8088) is within **1 point** of the English STS average for `multilingual-e5-large` (0.8170).
This demonstrates cross-language parity — Brazilian Portuguese users get embedding quality equivalent to what English users expect.

## Comparison with published Portuguese benchmarks

The Serafim paper (Santos et al., 2024) provides additional context for Portuguese embedding performance. Published scores on overlapping benchmarks:

| Model | ASSIN2 STS | Parameters | Specialization |
| - | - | - | - |
| **mk-embeddings-pt** | **0.8323** | 334M | pt-BR specialized |
| DistilUSE multilingual | 0.7170 | 135M | Multilingual |
| GTE (English) | 0.5971 | 434M | English only |

`mk-embeddings-pt` outperforms published multilingual and English-only baselines by **+11.5 to +23.5 points** on the native Brazilian Portuguese ASSIN2 benchmark.

## Training data

All training and evaluation data comes from publicly available Brazilian academic benchmarks.

| Dataset | Source | Type | Size | Origin |
| - | - | - | - | - |
| ASSIN2 | NILC/USP | STS + Entailment | 9,448 | Brazilian academic institutions |
| ASSIN v1 | NILC/USP | STS + Entailment | 10,000 | Brazilian + European Portuguese |
| CCMatrix en-pt | OPUS | Cross-lingual parallel | 20,000 | Translation pairs |

ASSIN and ASSIN2 are the standard benchmarks for Brazilian Portuguese semantic understanding, produced by the Interinstitutional Center for Computational Linguistics (NILC) at the University of São Paulo.

## Sovereign AI compliance

| Requirement | Status |
| - | - |
| Training data from Brazilian institutions | ASSIN2 and ASSIN from NILC/USP |
| LGPD compliance | All data is publicly available academic benchmarks — no PII |
| On-premise deployment | Model is \~670 MB, runs on commodity hardware |
| No external API calls | Inference is fully local |
| No international data transfer | Model weights and inference stay within sovereign infrastructure |
| Open weights | Available for government audit and customization |

The model can be further fine-tuned on domain-specific data (legal, government, regulatory) without exposing classified documents to external services.

## Recommendations

### For retrieval and RAG

Use `mk-embeddings-pt` for all Portuguese retrieval pipelines. The **+14.2 point advantage on SICK-BR-STS** and **+6.2 point advantage on Assin2RTE** translate directly to better retrieval relevance and more accurate entailment detection in RAG systems.

### For classification

For intent classification and scenario classification tasks, `multilingual-e5-large` remains stronger due to its larger multilingual training distribution. Consider using a hybrid approach: `mk-embeddings-pt` for retrieval and a separate classifier for categorization.

### For further specialization

The model can be fine-tuned on domain-specific Brazilian Portuguese data using CoSENT loss for STS optimization or contrastive learning for retrieval. Recommended domains for government deployment: legal texts, regulatory documents, public service workflows.

## References

1. Real et al., 2020. "The ASSIN 2 Shared Task: a Portuguese Semantic Similarity Evaluation"
2. Santos et al., 2024. "Serafim: Portuguese Sentence Embeddings" (arXiv:2407.19527)
3. Enevoldsen et al., 2025. "MMTEB: Massive Multilingual Text Embedding Benchmark" (arXiv:2502.13595)
4. Wang et al., 2024. "Multilingual E5 Text Embeddings" (arXiv:2402.05672)
5. Souza et al., 2020. "BERTimbau: Pretrained BERT Models for Brazilian Portuguese"
6. MTEB Leaderboard — [https://huggingface.co/spaces/mteb/leaderboard](https://huggingface.co/spaces/mteb/leaderboard)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.