Copy some fields from one index to another

Elastic Stack Elasticsearch
1

I am thinking if it's possible to merge two indices to holt my data

  1. first_index:
PUT first_index
{
  "mappings": {
    "dynamic": "false",
    "_source": {
      "excludes": ["content"]
    },
    "properties": {
      "content": {
        "type": "text"
      },
      "title": {
        "type": "text"
      }
    }
  }
}
  1. second_index:
PUT second_index
{
  "mappings": {
    "dynamic": "false",
    "properties": {
      "content": {
        "type": "text",
        "index": false
      }
    }
  }
}

As you can see, in the first_index the field content is marked to not be stored in _source and the same field is not indexed in second_index. I want to use it as a backup for future reindexing.
When I do reindexing of the first_index I'm about to lose the content field. Is there some way how to fill the content field from the second_index?

POST _reindex
{
  "source": {
    "index": "first_index"
  },
  "dest": {
    "index": "third_index"
  }
}

I mean I will not be able to find in third_index's content field

2

Reindex API | Elasticsearch Guide [8.2] | Elastic might work, where the pipeline pulls the second index in.

2 Likes
3

Well, but is Reindex API able to merge documents or do I have to store all documents twice?
I mean in first_index I have document with all searcheable fields content and title in my example, but to save the space I was thinking to have only field content.
Documents are not merged, but rewritten (they miss title field) when I try to use this

POST _reindex
{
  "source": {
    "index": ["first_index", "second_index"]
  },
  "dest": {
    "index": "third_index"
  }
}
4

I suppose there is no such merging built-in function for re-index.

I'm not sure it works, but one possibility is to set enrich processor on the destination third_index for the reindex from first_index. You can add some fields from the matched document in the second index.

5

If you look at the link, it shows you can use an ingest pipeline. That approach should be able to do the merge for you.

1 Like
6

Thank you @warkolm , I missed you said about the ingest pipelines. I was just repeating the same thing you said.

7

Yep, no worries :slight_smile:

1 Like
8

Yes, you can achieve it using a simple script, you don't have to use any processor for it, step by step thing to achieve it.

  1. Create a new index with a single field only, for example
PUT /second_index
{
   "mappings" : {
       "properties" : {
           "content" : {
               "type" : "text"
           }
       }
   }
}

Now, use the reindex API as shown below

PUT /_reindex
{
    "source": {
        "index": "first_index"
    },
    "dest": {
        "index": "second_index"
    },
    "script": {
         "source": "ctx._source.content=ctx._source.content + ' ' + ctx._source.title"
    }
}

Note: I am assuming your first index had content and title field and this simple script will just merge the values of these two fields and store it in content field of second_index. Hope this helps.

Update: You can test it, by simply indexing 2 documents(I already did) and search on second index as

POST second_index/_search

Response

"hits": [
            {
                "_index": "second_index",
                "_type": "_doc",
                "_id": "1",
                "_score": 1.0,
                "_source": {
                    "title": "title",
                    "content": "content title"
                }
            },
            {
                "_index": "second_index",
                "_type": "_doc",
                "_id": "2",
                "_score": 1.0,
                "_source": {
                    "title": "latest title",
                    "content": "latest content latest title"
                }
            }
        ]
9

Hi Amit,
thanks for your response, but this is not what I wand. I don't want to merge two fields from the same index to the new one. This is easy.
I want to merge field field title from first_index and field content from second_index into the new third_index.

10

Got it, let me try and get back to you.

1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.