Copy some fields from one index to another

I am thinking if it's possible to merge two indices to holt my data

  1. first_index:
PUT first_index
{
  "mappings": {
    "dynamic": "false",
    "_source": {
      "excludes": ["content"]
    },
    "properties": {
      "content": {
        "type": "text"
      },
      "title": {
        "type": "text"
      }
    }
  }
}
  1. second_index:
PUT second_index
{
  "mappings": {
    "dynamic": "false",
    "properties": {
      "content": {
        "type": "text",
        "index": false
      }
    }
  }
}

As you can see, in the first_index the field content is marked to not be stored in _source and the same field is not indexed in second_index. I want to use it as a backup for future reindexing.
When I do reindexing of the first_index I'm about to lose the content field. Is there some way how to fill the content field from the second_index?

POST _reindex
{
  "source": {
    "index": "first_index"
  },
  "dest": {
    "index": "third_index"
  }
}

I mean I will not be able to find in third_index's content field

Reindex API | Elasticsearch Guide [8.2] | Elastic might work, where the pipeline pulls the second index in.

2 Likes

Well, but is Reindex API able to merge documents or do I have to store all documents twice?
I mean in first_index I have document with all searcheable fields content and title in my example, but to save the space I was thinking to have only field content.
Documents are not merged, but rewritten (they miss title field) when I try to use this

POST _reindex
{
  "source": {
    "index": ["first_index", "second_index"]
  },
  "dest": {
    "index": "third_index"
  }
}

I suppose there is no such merging built-in function for re-index.

I'm not sure it works, but one possibility is to set enrich processor on the destination third_index for the reindex from first_index. You can add some fields from the matched document in the second index.

If you look at the link, it shows you can use an ingest pipeline. That approach should be able to do the merge for you.

1 Like

Thank you @warkolm , I missed you said about the ingest pipelines. I was just repeating the same thing you said.

Yep, no worries :slight_smile:

1 Like

Yes, you can achieve it using a simple script, you don't have to use any processor for it, step by step thing to achieve it.

  1. Create a new index with a single field only, for example
PUT /second_index
{
   "mappings" : {
       "properties" : {
           "content" : {
               "type" : "text"
           }
       }
   }
}

Now, use the reindex API as shown below

PUT /_reindex
{
    "source": {
        "index": "first_index"
    },
    "dest": {
        "index": "second_index"
    },
    "script": {
         "source": "ctx._source.content=ctx._source.content + ' ' + ctx._source.title"
    }
}

Note: I am assuming your first index had content and title field and this simple script will just merge the values of these two fields and store it in content field of second_index. Hope this helps.

Update: You can test it, by simply indexing 2 documents(I already did) and search on second index as

POST second_index/_search

Response

"hits": [
            {
                "_index": "second_index",
                "_type": "_doc",
                "_id": "1",
                "_score": 1.0,
                "_source": {
                    "title": "title",
                    "content": "content title"
                }
            },
            {
                "_index": "second_index",
                "_type": "_doc",
                "_id": "2",
                "_score": 1.0,
                "_source": {
                    "title": "latest title",
                    "content": "latest content latest title"
                }
            }
        ]

Hi Amit,
thanks for your response, but this is not what I wand. I don't want to merge two fields from the same index to the new one. This is easy.
I want to merge field field title from first_index and field content from second_index into the new third_index.

Got it, let me try and get back to you.

1 Like