Sniffer not fully working

Elastic Stack Beats
1

I have installed the agent on ubuntu running HAPROXY with multiple ports backends. We have high traffic sites but it looks like nothing is really recording on elastcisearch.

sudo packetbeat -e -d "publish" -N does shows traffic just no traffic is being sent out.

2

It looks like its only saving 5 records

http://ip:9200/packetbeat-*/_search?pretty

Shows {
"took" : 4,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 4193,
"max_score" : 1.0,
"hits" : [ {
"_index" : "packetbeat-2015.05.29",

3

Hi, the -N disables publishing. What happens when you run it like packetbeat -e -d "publish"? Still only 5 records?

4

I'm having the same issue with capturing HTTP data. I'm running the following command:

packetbeat -e -d "Published" -c /etc/packetbeat/packetbeat.yml

I am only getting a few entries in my ES index. I see the following entries scrolling through my console window:

http.go:730: WARN Response from unknown transaction. Ignoring: TcpTuple src[X.X.X.X:59560] dst[X.X.X.X:80] stream_id[2743]
http.go:647: WARN Two requests without a response. Dropping old request

If i run the following I can see the http traffic:

packetbeat -e -d "http,httpdetailed" -c /etc/packetbeat/packetbeat.yml

http.go:237: DBG Header: Referer
http.go:247: DBG HV: http://z.cdn.turner.com/xslo/cvp/assets/container/2.0.4.0/cvp_main_container.swf
http.go:253: DBG Header: 'referer' Value: 'http://z.cdn.turner.com/xslo/cvp/assets/container/2.0.4.0/cvp_main_container.swf'
http.go:236: DBG Data: Cookie: IMRID=11880577-4427-4824-b761-5a82e486416f
Connection: keep-alive

http.go:237: DBG Header: Cookie
http.go:247: DBG HV: IMRID=11880577-4427-4824-b761-5a82e486416f
http.go:253: DBG Header: 'cookie' Value: 'IMRID=11880577-4427-4824-b761-5a82e486416f'
http.go:236: DBG Data: Connection: keep-alive

http.go:237: DBG Header: Connection
http.go:247: DBG HV: keep-alive
http.go:253: DBG Header: 'connection' Value: 'keep-alive'
http.go:385: DBG Empty content length, ignore body
http.go:647: WARN Two requests without a response. Dropping old request
http.go:655: DBG Received request with tuple: TcpTuple src[X.X.X.X:60988] dst[X.X.X.X:80] stream_id[30]
http.go:943: DBG Parameters:
^Csniffer.go:319: INFO Input finish. Processed 3068 packets. Have a nice day!

I was able to dump the traffic into a pcap file using the following:
packetbeat -e -dump /tmp/packetbeat/trace3.pcap

I opened the pcap with wireshark and could see all the traffic.

As a side note: The packetbeat server is virtual server connected to a mirrored port on my virtual switch (VMware distributed Switch). I configured it to listen on eth1.

Any thoughts as to why it's not sending the data into Elasticsearch (or the log file for that matter).

5

Can you send me that trace3.pcap file? You can send it to tudor@elastic.co.

6

Unfortunately I can't send the pcap for security reasons.

7

Ok, one thing to check is that in wireshark you don't see any "packet truncated during capture". This would be an indication that the snaplen setting is too low.

It could also be that there is some packet loss in the probably large response. Look in wireshark if you seen any notices of "previous tcp packet was not captured" or something like that.

8

Searched through my pcap file and didn't see any "packet truncated" messages nor "previous tcp packet was not captured" message.

9

Hmm, then I'm not sure. From the logs you posted already it looks like a request was correctly captured but no response for it. Can you post or send me a larger portion of the log file?