I'm currently working in a company where we intend to use PacketBeat with ELK. It will initially only deal with DNS packets.
We don't have any real experience with ELK, so its kind of a challenge. I'm testing the latest version of PacketBeat that supports DNS over UDP.
The live capture mode works pretty good, except maybe for a behaviour I'm not sure is expected: whenever packets are replayed by a client wishing to resolve a domain name, ElasticSearch database only contains a few of them.
For example, a client (192.168.2.23) queries 192.168.2.208 to resolve a domain name :
dig +tries=3 @192.168.2.208 google.com ... ;; connection timed out; no servers could be reached
Since 192.168.2.208 is not a DNS Server, it is not listening on port 53 and that is why DNS packets are replayed 3 times by the client. 'replayed' means the packets stay exactely the same (as is the DNS ID) for all the requests. However, duplicate packets can also happen on any real DNS Server listening on port 53 and we want to make sure that every packet are sent to ElasticSearch, even if their are duplicates.
On 192.168.2.208, tcpdump shows that the 3 requests are received:
15:59:27.821199 IP 192.168.2.23.48362 > 192.168.2.208.53: 53727+ A? google.com. (28) 15:59:32.820852 IP 192.168.2.23.48362 > 192.168.2.208.53: 53727+ A? google.com. (28) 15:59:37.820623 IP 192.168.2.23.48362 > 192.168.2.208.53: 53727+ A? google.com. (28)
The problem is that only 2 of these requests appear in ElasticSearch.
It is worst when reading PCAP, because then only 1 packet of the previous dig command appears in ElasticSearch.
Finally, I would have two last questions :
Is it normal that when reading a PCAP file, packets data sent to ElasticSearch have a timestamp not equal to the timestamp in the PCAP file but takes its value from when PacketBeat sent the JSON formated data to ElasticSearch ?
We would like to have the original packets information in ElasticSearch.
Also, the '-t' option for reading PCAP doesn't put any packet in ElasticSearch even though PacketBeat prints 'dns.go:368: DBG Publishing transaction. DnsTuple src[192.168.2.23:56967] dst[192.168.2.208:53] transport[udp] id'. I tried the '-t' option with another protocol (http on port 80) but the result stays the same: no packets appeared in ElasticSearch.
Content of the PacketBeat config file :
shipper: refresh_topology_freq: 10 topology_expire: 15 interfaces: device: eth0 type: pcap protocols: dns: ports:  output: elasticsearch: enabled: true host: 127.0.0.1 port: 9200 save_topology: true
With thanks in advance