Combining two fields into one during Reindex

Elastic Stack Elasticsearch
1

Hi,

We have a requirement wherein we want to combine two fields into one field during reindex.
For Example:
Original Index:
field1: firstname
field2: lastname

After reindex:
Name: firstname lastname

I have checked some documents on reindex and _update_by_query but I am not able to find my requirement.Can this be done using reindex? or Is there any other way?

Thanks,
Nikhil

1 Like
2

Hello,

Any suggestions?

Thanks,
Nikhil

3

Did you try copy_to ?

Defined in mappings :
"mappings": {
"_doc": {
"properties": {
"region_name": {
"type": "keyword",
"copy_to": "locations_combined"
},
"country_name": {
"type": "keyword",
"copy_to": "locations_combined"
},
"city_name": {
"type": "keyword",
"copy_to": "locations_combined"
},
"locations_combined": {
"type": "text"
}
...

4

Hi @newbie1,

I tried using copy_to. I created mappings for the new index and then reindexed the data into the new index but the field is not visible. Since it doesn't create any field, I cannot use it in visualizations.

Can it be done using update_by_query to add a new field which is a concatenated value of two fields?

Thanks,
nikhil

5

That's strange, did you check the mapping of the new index to be sure to have defined the copy_to field ? Did you try with a sample ?

6

Have you tried using a script inside the reindex request? I have personally done this many times. You could combine the two fields into one and then use ctx._source.remove("[property name]") to remove the extra field.

7

Hi @newbie1,

Yes, I have tried with sample data.After using reindex and copy_to to create a new field(containing combined value from other two fields ), it will create the new field and you can also use search based on the new field but the new field will not visible in "Discover".

8

Hi @radoslav.sholev,

Yes, I have tried various combination of using scripts during reindex but I did't get any expected results.

My expectation is to create to a new field(containing values from two other fields) during reindex.

Example:
For Example:

Original Index: Containing two fields
firstname: Ryan
lastname: Gosling

After reindex: Contains only one field
name: Ryan Gosling

Expectation is to combine two values into one during reindex

9

On this example could you provide the mapping of original index and teh mapping of destination index. A good test woul be to set dynamic to strict like that you will see if there is an issue during reindexation.

10

@Nikhil04 here is a simple example of what I am trying to say. Downside is that the target index will have to create a mapping for the two fields, but you don't have to store them in the document. As I wrote before, the fields can be removed in the reindex script. Here is the example:

PUT index_4
{
  "mappings": {
    "properties": {
      "firstName":{
        "type":"text"
      },
      "lastName":{
        "type":"text"
      }
    }
  }
}

PUT index_4/_doc/1
{
  "firstName": "A",
  "secondName": "B"
}

PUT index_5
{
  "mappings": {
    "properties": {
      "name":{
        "type":"text"
      }
    }
  }
}

POST _reindex
{
  "source": {
    "index": "index_4"
  },
  "dest": {
    "index": "index_5"
  },
  "script": {
    "source": "ctx._source.name=ctx._source.firstName + ' ' + ctx._source.secondName"
  }
}

GET index_5/_search
2 Likes
11

Thank you @radoslav.sholev for your help!

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.