Getting error while trying to create knn index on elasticsearch version 8.7.1

[Sharad_Nautiyal]

Below is the request I am sending while creating knn index:

PUT posting
{
  "settings": {
    "index": {
      "number_of_shards" :20,
      "number_of_replicas": 1,
      "knn":{
        "algo_param":{
          "ef_search":40,
          "ef_construction":40,
          "m": "4"
        }
      }
    },
    "knn": true
  }, 
  "mappings": {
    "properties": {
      "vector": {
        "type": "knn_vector",
        "dimension": 384
      }
    }
  }
}

Below is the error I am facing:-
"reason": "unknown setting [index.knn.algo_param.m] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

As per error it is saying plugin missing but if I go by the documentation knn is available on 8.0 version onwards. Kindly help on how to resolve this, if I need to install external plugin please share steps to do so.

[Christian_Dahlqvist]

I do not see these parameters anywhere in the official Elasticsearch documentation. It does however, based on quick google search, seem like these parameters may be used in Opensearch. Where did you get these parameters from?

[Serena_Chou]

Hey there, please check how you've set the parameters. These documents might help to get you started kNN search API | Elasticsearch Guide [8.9] | Elastic
k-nearest neighbor (kNN) search | Elasticsearch Guide [8.9] | Elastic

and we have a great repository of code that can help you to bootstrap this. Here are some notebooks you can use to try it out https://github.com/elastic/elasticsearch-labs/tree/main/notebooks

[Sharad_Nautiyal]

@Sharad_Nautiyal

reaking changes documentation for removed settings

Hi Thanks for the response.
@Christian_Dahlqvist I was trying to explore data vectorization and then performing semantic search on that data in elasticsearch using knn following note which is also referred by @Serena_Chou "k-nearest neighbor (kNN) search | Elasticsearch Guide [8.11] | Elastic" . Although the code shared above has type: knn_vector which I was randomly trying.

@Serena_Chou thanks I followed the example mentioned in the git repo you have shared. In example "https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/00-quick-start.ipynb" they have mentioned under column mappings type as dense_vector as below:
"title_vector": {
"type": "dense_vector",
"dims": 384,
"index": "true",
"similarity": "cosine"
}
If you refer to my installed elk setup on version 8.7.1 I don't see any data type as vector, screenshot from installation below:-

Is this option only present on Elastic cloud version and not available in on-premise version. Please help to let me know.

[joemcelroy (Joseph McElroy)]

hi there @Sharad_Nautiyal,

dense_vector should be a supported data type (see documentation) Could be that its not showing up on the autocomplete but is supported.

Joe