Hi ES team,
I'm looking to merge 2 index's based off unique ID which is same across both the index and the desired index should look like,
ID, Email, Fax
ID, Email, Fax
Now the merged Index should be,
ID, Index1_Email, Index2_Email, Index1_Fax, Index2_ Fax
Hence the merge has taken place by field and not rows.
Appreciate your input's.
you can do this via python.
how big your index1 and index2 (how many records?)
use elasticsearch_dsl to scan ID create new record and push that to ELK.
Have a look at transforms, you can source both indexes, use a
pivot, group on
terms using the field
id. For the output fields you have several options (just ideas, not verified):
filter aggregation and
top_metrics as sub aggregation
top_metrics on top, sort by
_index ascending for index 1, descending to get index 2
You find further information in older posts: How to join two index - #3 by Hendrik_Muhs
Note however that these posts are old and e.g.
top_metrics has been added just recently. I talk about
top_metrics in a recent advent post: Dec 11th, 2021: [en] On a road trip with Transform
Thanks @Hendrik_Muhs ,
But the issue I seem to be stuck with an error as shown below.
Appreciate your help on the same.
Your call does not look correct to me, can you delete the id from the body and don't use the underscore in the path?
The id (
index_merge) must be specified as part of the path. You can specify it in the body, too, but than they must match. Leading underscores are not allowed.
, not sure if still I'm missing something.
Can you provide the version and environment you are using? It seems like you are either using a version that does not provide transform or you lack permissions.
@Hendrik_Muhs , I'm using version 7.10.2.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.