Creating an index from another index with only certain fields and a different ID than the original index

Elastic Stack Elasticsearch
1

Is there anyway to create an index from another index but only taking certain fields, not all? The new index would have a new ID as well and include a new alias. I was looking at the reindex API and the clone API as maybe a possible solution however not sure if this is able to accomplish this. Another possible solution could be somehow making text file from the index and then transforming that file as needed to then reingest into the new index, however, this process seems to be the longest in terms of time as the original index may have more than 20 million records/documents.

An example of a document in the original document would be:

_id: field1-field2-field3-field4
field1: ***
field2: ***
field3: *****
field4: ****
field5: *****
field6: ******
field7: ******
field8: ******

The new index would then have :

_id: field2
field1: ***
field2: ***
field6: ******
field8: ******

The example is only for one record but it would be for the whole index, all documents in that index.

3

You should be able to use the reindex API together with an ingest pipeline that removed the unwanted fields.

4

When you say an ingest pipeline is that like referring to making another logstash config file with essentially remove fields options in it and then running that ingest process after the reindex API REST calls have been made?

5

The reindex is working however when trying to use this in the script portion to get the date from the previous index name to add to the new one it doesnt add it.

"ctx._index = ctx._index + '-' + (ctx._index.substring('testsubindex-'.length(), ctx._index.length()))"

I got this from the docs however this is giving me an error

"error" : {
"root_cause" : [
{
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"ctx._index = ctx._index + - + (ctx._index.substring('testsubindex-'.length(), ctx._index.length()))",
" ^---- HERE"
],
"script" : "ctx._index = ctx._index + - + (ctx._index.substring('testsubindex-'.length(), ctx._index.length()))",
"lang" : "painless"
}
],
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"ctx._index = ctx._index + - + (ctx._index.substring('testsubindex-'.length(), ctx._index.length()))",
" ^---- HERE"
],
"script" : "ctx._index = ctx._index + - + (ctx._index.substring('testsubindex-'.length(), ctx._index.length()))",
"lang" : "painless",
"caused_by" : {
"type" : "null_pointer_exception",
"reason" : null
}
},
"status" : 400

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.