Convert dense vector to float

Elastic Stack Elasticsearch
1

Hi im using ELK stack V 8.13.3
i having an issue in my index:
i dont know why but I pushed 8 vectors and the mapper set them as dense_vector(no activity from my side to set this type was done)
now this 8 vectors can change the size and I got mapper error.
my goal is to change them into float.
i tried to create new index with new float mapping to problematic fields and reindex the data to new index excluding the dense vector(currently the data in dense_vector can be descarted)
but I got always 2 parameters with the same name but different types in new index:
1.float
2.dense vector
my devtoolcode:

PUT /rf_equipment_data_simple_new_2025
{
  "mappings": {
    "properties": {
      "ES_FLATNESS_AVERAGE_RES": { 
        "type": "nested",
        "properties": { "values": { "type": "float" } }
      },
      "SEMASK_AVERAGE_SPECTRUM_OFR_TRACE": { 
        "type": "nested",
        "properties": { "values": { "type": "float" } }
      },
      "SEMASK_MAX_SPECTRUM_TRACE": { 
        "type": "nested",
        "properties": { "values": { "type": "float" } }
      },
      "TRACE_SE_MASK_AVG_RBW_100": { 
        "type": "nested",
        "properties": { "values": { "type": "float" } }
      },
      "TRACE_SE_MASK_AVG_RBW_1000": { 
        "type": "nested",
        "properties": { "values": { "type": "float" } }
      },
      "SEMASK_AVERAGE_SPECTRUM_TRACE": { 
        "type": "nested",
        "properties": { "values": { "type": "float" } }
      },
      "TRACE_SE_MASK_MAX_RBW_100": { 
        "type": "nested",
        "properties": { "values": { "type": "float" } }
      },
      "TRACE_SE_MASK_MAX_RBW_1000": { 
        "type": "nested",
        "properties": { "values": { "type": "float" } }
      }
    }
  }
}

MAPPING SHOWS ONLY FLOATS:

GET rf_equipment_data_simple_new_2025/_mapping

REINDEX EXCLUDE DENSE_VECTOR

POST _reindex?wait_for_completion=false
{
  "source": {
    "index": ["rf_equipment_data-000001", "rf_equipment_data-000002", "rf_equipment_data-000003"],  
    "size": 1000
  },
  "dest": {
    "index": "rf_equipment_data_simple_new_2025"
  },
  "script": {
    "source": """
    // Remove all fields that were previously dense_vector
    def keysToRemove = [];
    
    for (entry in ctx._source.entrySet()) {
      // Check if the field was a dense_vector (by checking for an object with a "dims" key)
      if (entry.getValue() instanceof Map && entry.getValue().containsKey("dims")) {
        keysToRemove.add(entry.getKey());
      }
    }

    // Remove all identified dense_vector fields
    for (key in keysToRemove) {
      ctx._source.remove(key);
    }

    // Add a success message to confirm reindexing
    ctx._source["reindex_status"] = "SUCCESS";
    """
  }
}

any help please?
in addition I want my current ILM will be applied to new index, is it possible?

2

Hello, just a couple things off the top of my head:

  • Are you able to reindex the dense_vectors to an array of floats with a new name? Then reindex to a third, moving over only the floats?
  • Are you able to explicitly remove the dense_vectors by field name in your script?
  • Check to make sure all of the dense vectors you want to convert and remove are of the exact same nested structure. There may be discrepancies in how they're stored

You can see if changing your dynamic mapping in your reindex helps. Here's the doc

And yes, you can enact your ILM in your new index, like so:

PUT /rf_equipment_data_simple_new_2025
{
  "settings": {
    "index.lifecycle.name": "your_ilm_policy_name",
    "index.lifecycle.rollover_alias": "your_alias_if_applicable"
  },
  "mappings": {
    "dynamic": "strict", #if you want to try this out
    "properties": {
      "ES_FLATNESS_AVERAGE_RES": { 
        "type": "nested",
        "properties": { "values": { "type": "float" } }
      }, ...
}

You can also create a index_template that adds your ILM to all ensuing indexes.

These are just some ideas without reproduction efforts. Let me know how it goes.

3

Hi @Justin_Castilla and thanks for the quick response!
regarding bullet #1:
how should I do it? I tried multiple times and when the source index has dense_vector -> it moves on to the newly generated one.

regarding bullet #1:
already tried -> same result.

By the way, I realized that on my ELK stack version, each vector above the size of 128 will be automatically inserted as a dense vector, right?