ModernBERT
The original BERT paper came out in 2018, around 7 years ago at time of writing. However, it is still referred to and used as a strong baseline in a number of NLP tasks. ModernBERT was created by HuggingFace. ModernBERT is a drop-in replacement for use in problems where BERT may have previously been used and like the original, has a base and a large variant.
Comparison Table
|
BERT Base
|
ModernBERT Base
|
BERT Large
|
ModernBERT Large
|
|
|---|---|---|---|---|
|
# Params
|
110M
|
149M
|
340M
|
395M
|
|
Context Size
|
512
|
8192
|
512
|
8192
|
|
BEIR
|
38.9
|
41.6
|
38.9
|
44.0
|
mmBERT is a modern multi-lingual encoder-only model based on ModernBERT.