How to display duplicate values of a particular field in Kibana


#1

I have log data as follows:
udp 81.0.0.1:1024 11.0.0.5:1024 --- ---
udp 81.0.0.1:1024 11.0.0.5:1024 --- ---
udp 81.0.0.1:1024 11.0.0.5:1024 --- ---
udp 81.0.0.1:1024 11.0.0.5:1024 --- ---
udp 81.0.0.1:1024 11.0.0.5:1024 --- ---
udp 81.0.0.1:1024 11.0.0.5:1024 --- ---
tcp 81.0.0.1:1024 11.0.0.5:52041 --- ---
udp 81.0.0.1:1027 11.0.0.5:1027 --- ---
udp 81.0.0.1:1026 11.0.0.5:1026 --- ---
udp 81.0.0.1:1026 11.0.0.5:1026 --- ---
udp 81.0.0.1:1026 11.0.0.5:1026 --- ---
udp 81.0.0.1:1025 11.0.0.5:1025 --- ---

I need to display the count of all duplicate entries in Kibana.
How can i do that ?


(Spencer Alger) #2

You first need to get your data into elasticsearch using one of the beats or logstash, but then once it's in elasticsearch a terms agg will count the number of times a specific term is found.

I would start by splitting those log lines into fields, something like protocol, srcip, srcport, destip, and destport. I would then run a terms aggregation on protocol to find how many of the logs were for tcp or udp and add a sub-aggregation for destip to see how many requests came to each uniq combination of protocol and destip.


(system) #3