How to emulate elasticsearch.yml in elasticsearch 5.0

Elastic Stack Elasticsearch
1

I recently switched to elasticsearch 5.0 and some of the things in elasticsearch.yml from 2.x don't seem to be working. I get an error on startup that warns my to define my index parameters using curl instead of elasticsearch.yml, the suggested modifcations did not work - and I determined I should be using the index template. Anyway, my es 2.x elasticsearch.yml looks like this

index :
  number_of_shards: 1
  number_of_replicas: 1
  similarity:
    default:
      type: BM25
      b: 0.0
      k1: 1.2
    norm_bm25:
      type: BM25
      b: 0.75
      k1: 1.2

And I've tried to use templates to replace this using

curl -XPUT 'http://localhost:9200/_template/template1' -d '{
  "template" : "*",  
  "settings.number_of_replicas" : "1",
  "settings.number_of_shards" : "1",
  "settings.similarity.default.type" : "BM25",
  "settings.similarity.default.b" : "0.0",
  "settings.similarity.default.k1" : "1.2",
  "settings.similarity.norm_bm25.type" : "BM25",
  "settings.similarity.norm_bm25.b" : "0.75",
  "settings.similarity.norm_bm25.k1" : "1.2"
}'

However, I seem to be getting errors from my elasticsearch.js (now at version 12.01) when I try to create a new index with the call (this worked fine with elasticsearch 2.x)

 client.indices.create({
      index: indexName,
      body: { settings: { 
          number_of_shards: 1,
          similarity : "norm_bm25"
      } 
}. ... error stuff

The error is

'{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"unknown setting    [index.similarity] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"}],"type":"illegal_argument_exception","reason":"unknown setting [index.similarity] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"},"status":400}'

I'm thinking the issue is with my template, but it is installed without warning. The problem could also be in elasticsearch.js - though elasticsearch.js 11.01, for which it worked before, also fails with es 5.

Thanks for any help.

John

2

Hi,

The default similarity has been renamed to classic:

https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_settings_changes.html#_similarity_settings

3

Even though the default similarity is now classic, it looks like default is defined as type BM25 (though not totally clear from the documentation) https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html . I think things may be failing because of my definition of norm_bm25?

Nevertheless, I tried changing default (above) to,

"settings.similarity.classic.type" : "BM25",
"settings.similarity.classic.b" : "0.0",
"settings.similarity.classic.k1" : "1.2",

And still get the same problem. Also, in the elasticsearch.js snippet I've changed norm_bm25 to both classic and default with the same result.

John

4

This works for me:

$ curl -XPUT "http://172.17.42.1:9200/myindex" -d'
{
  "settings": {
    "similarity": {
      "my_similarity": {
        "type": "BM25",
        "b": "0.0",
        "k1": "1.2"
      }
    }
  }
}
'
{"acknowledged":true,"shards_acknowledged":true}

I recommend you format your template in a similar way.

5

What if I want it to work for all indexes by default? I'll have 10s of thousands of relatively small tables.

6

You need to do:

curl -XPOST "http://172.17.42.1:9200/myindex/_close"

curl -XPUT "http://172.17.42.1:9200/myindex/_settings" -d'
{
"similarity": {
"my_similarity": {
"type": "BM25",
"b": "0.0",
"k1": "1.2"
}
}
}'

curl -XPOST "http://172.17.42.1:9200/myindex/_open"

It doesn't appear you can dynamically set these on an open index. You can also try using _all but will fail on any open index.