Best method to get 5 fields from inputs and duplicate in index2

[aaleman]

Hello, I was reading a lot because I want to now:

• What is the best way that masters recommend to next case?:

I have index1 with a lot of fields
I need this data into another index to have history, but this index2 just need 5 fields from inputs that are going to index1

At final I will have:
index 1 with all inputs data
Index2 with just 5 fields from inputs that are going to index1

What is the best way to achieve this? logstash purge? copy?

I was thinking to purge with whitelist field, but I'm not sure If I f#ck the index number 1 because I want to duplicate just 5 fields from inputs to index2 and not purge data that is going to index1...

My workflow is:

filebeat --> input logstash --> filter logstash --> output logstash

Actually, machine is sending json logs, logstash filtering using a field value, apply json, and go to output to index1 using filter by field value.

Maybe my explanation is a bit mesh...sorry!

[AquaX (Andreas Helmer)]

You can create a clone of the event and then use a conditional to check what the tag or type of the clone is (depending if you have ECS enabled).

This is a rough guide that I would try.

filbeat-> input logstash > filter logstash {
clone { type => ["index2"]}
if "index1" in [tags] {
#remove all the fields you don't need
}
output {
if "index2" in [tags] {
#output to your other elasticsearch index
}
}