Hello,
I tried to use clone plugin to get a single copy of the event.
However, it did not work. Version of our elastic stack - 7.9.2.
I have ids which are stored in one field - I use split on them and then I drop this field not to take so much space as in prod there are batches with 1000 records, but the target is to have one event (original) persisted and saved to have this batch once stored.
Here is a piece of my pipeline:
if [geimp][importResult][fail] {
# Generate hashBatchId
fingerprint {
method => "SHA1"
base64encode => true
source => [ "[geimp][importResult][fail]" ]
target => "[geimp][batchFingerprint]"
}
# Clone event to save the batch before message gets deleted
clone {
clones => [ "clonedEvent" ]
}
# Split events
split {
field => "[geimp][importResult][fail]"
target => "[geimp][importResult][fail]"
terminator => ", "
}
mutate {
remove_field => [ "[geimp][message]" ]
}
}