Merge 2 index based off fields and not rows

[Sugand]

Hi ES team,

I'm looking to merge 2 index's based off unique ID which is same across both the index and the desired index should look like,

Index 1
Fields
ID, Email, Fax

Index 2
Fields
ID, Email, Fax

Now the merged Index should be,
Index 3
ID, Index1_Email, Index2_Email, Index1_Fax, Index2_ Fax

Hence the merge has taken place by field and not rows.

Appreciate your input's.

[elasticforme (Sachin)]

you can do this via python.
how big your index1 and index2 (how many records?)

use elasticsearch_dsl to scan ID create new record and push that to ELK.

[Hendrik_Muhs]

Have a look at transforms, you can source both indexes, use a pivot, group on terms using the field id. For the output fields you have several options (just ideas, not verified):

• a filter aggregation and top_metrics as sub aggregation
• top_metrics on top, sort by _index ascending for index 1, descending to get index 2
• scripted_metric

You find further information in older posts: How to join two index - #3 by Hendrik_Muhs

Note however that these posts are old and e.g. top_metrics has been added just recently. I talk about top_metrics in a recent advent post: Dec 11th, 2021: [en] On a road trip with Transform

[Sugand]

Thanks @Hendrik_Muhs ,

But the issue I seem to be stuck with an error as shown below.

Appreciate your help on the same.

[Hendrik_Muhs]

Your call does not look correct to me, can you delete the id from the body and don't use the underscore in the path?

PUT _transform/index_merge
{
...
}

The id (index_merge) must be specified as part of the path. You can specify it in the body, too, but than they must match. Leading underscores are not allowed.

[Sugand]

Hi @Hendrik_Muhs , not sure if still I'm missing something.

Thanks,

[Hendrik_Muhs]

Can you provide the version and environment you are using? It seems like you are either using a version that does not provide transform or you lack permissions.

[Sugand]

@Hendrik_Muhs , I'm using version 7.10.2.