Is indexing with HNSW required for kNN to work?

Elastic Stack Elasticsearch
1

While setting up some search templates for kNN, I discovered HNSW and the parameters for index options.

"index_options": {
          "type": "hnsw",
          "m": 32,
          "ef_construction": 100
        }

When indexing all my data, I didn't set the type to "hnsw". Does this mean kNN will not work at all? Or lower accuracy? I'm trying to understand the impact before I spend the time reindexing.

2

Hi @mmaccou !

"type": "hnsw" is the default (and the only value supported) for index_options on dense_vector field types. You don't have to explicitly set that on your index for knn to work.

There are two ways of performing kNN search:

The parameter that controls the indexing of the field (and thus, approximate knn search) is index (see dense_vector field type parameters).

Depending of the Elasticsearch version you used to create the index, it will be indexed by default (8.11 or higher) or not (8.10 and previous).

If you are using approximate kNN:

  • If you created your index before 8.11 and didn't have "index": true as a parameter for your dense_vector field type, you will need to create a new index and ensure your dense_vector field is indexed. You can use reindex for transferring your data to the new index.
  • If you created your index on version 8.11 or above, you don't need to do anything for approximate knn to work.

If you are using exact kNN, you need no index changes as you don't need your field to be indexed.

I hope that helped!

3

Very helpful thank you! I confirmed I have 8.11, so just to confirm, both of the index mappings below are essentially identical given the default settings.

"properties": {
      "ml": {
        "properties": {
          "inference": {
            "properties": {
              "chunkedContent": {
                "properties": {
                  "predicted_value": {
                    "type": "dense_vector",
                    "dims": 768,
                    "index": true,
                    "similarity": "cosine",
                    "index_options": {
                      "type": "hnsw",
                      "m": 32,
                      "ef_construction": 100

"properties": {
      "ml": {
        "properties": {
          "inference": {
            "properties": {
              "chunkedContent": {
                "properties": {
                  "predicted_value": {
                    "type": "dense_vector",
                    "dims": 768,
                    "index": true,
                    "similarity": "cosine"
4

From the knn point of view, they are equivalent as both are indexed explicitly.

The only change is m option in index_options that change from the default (16). It affects the internal HNSW structure, but doesn't affect the usage from the knn point of view.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.