Open-source lemmatizing analyzer for ES 9.x: +18.8% F1@10 vs standard on MIRACL English (reproducible benchmark)

Announcements Community Ecosystem
1

Hi all,

We measured how much BM25 relevance depends on the analyzer alone, and wanted to share the results and the plugin with the community.

Setup: Elasticsearch 9.5.4, plain BM25, MIRACL English (32.9M passages, 799 queries), official trec_eval.

  • vs standard: F1@10 +18.8%, nDCG@10 +15.7%, Recall@100 +14.8%
  • vs built-in english (Porter): +7.6% to +8.5% on every top-10 metric

The main gains come from dictionary lemmas instead of stemming: Porter can't map sold→sell or built→build, and it over-merges majority→major and United→unit.

Limits: one dataset, English only, lexical only.

Disclosure: I work at Bitext, which built the analyzer. It's Apache 2.0 and installs as an ES 9.x stable plugin (also OpenSearch 3.4+ and Solr 10).

Benchmark (fully reproducible): GitHub - bitext/bitext-search-benchmarks: Reproducible MIRACL lexical-search benchmarks to measure the benefits that the Bitext English Analyzer adds to Elasticsearch · GitHub

Feedback is welcome, especially cases where it does worse than the english analyzer on your own data.

2 Likes