Logstash create value based on duplicate field

Elastic Stack Logstash
1

Hello,
I wan't know if it's possible to get duplicate value un field and create another field based on that value.
Supposed I have logs:
Field1, Field2, Field3
A,B, C
A,D, E
B, E, T
B, R, H
S, V, X
What I wan't to have is :
Field1, Field2, Field3, Field4
A,B, C, 1
A,D, E, 0 (because we know A)
B, E, T, 1
B, R, H, 0
S, V, X, 1

Thank you

2

You could probably do that with an aggregate or ruby filter. Use A/B etc as the task_id. Check if the map already contains something for the task_id. If so, set the field to 1, otherwise 0. Then add a value to the map. You would never purge entries from the map, so this would leak memory.

3

Thank you for the response.
But the problem is I have many value on Field1 and not just A or B.
In the given example, we have three different value but in real log I have at least 4000 distinct value.

4

I do not see why having thousand of possible values makes a difference. If it were millions of values I would be concerned, but thousands should be OK.

5

Thank you, gonna check this!

6

Hello @Badger,
It is a good solution and if I want to have the number of fields as count like :

Field1, field2, field3, field4, count_event
 A,B, C, 1,2
A,D, E, 0,2
B, E, T, 1,2
B, R, H, 0,2
S, V, X, 1,1

Regards

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.