PacketBeat fails to extract HTTP from simple PCAP

Elastic Stack Beats
1

Hi,
I got trouble in extracting HTTP from my tcpdumped file
All I get is one tcp stream, but if I check with Wireshark I got full HTTP streams.

2017-10-13T09:40:52Z INFO packetbeat start running.
2017-10-13T09:40:52Z DBG  start flows worker
2017-10-13T09:40:52Z DBG  Waiting for the sniffer to finish
2017-10-13T09:40:52Z DBG  Packet number: 1
2017-10-13T09:40:52Z DBG  decode packet data
2017-10-13T09:40:52Z DBG  lock flows
2017-10-13T09:40:52Z DBG  flowid: add eth
2017-10-13T09:40:52Z DBG  worker wait start(2017-10-13 09:41:00 +0000 UTC): 7.639828107s
2017-10-13T09:40:52Z DBG  IPv4 packet
2017-10-13T09:40:52Z DBG  flowid: add ipv4
2017-10-13T09:40:52Z DBG  TCP packet
2017-10-13T09:40:52Z DBG  flowid: add tcp
2017-10-13T09:40:52Z DBG  flow id flags: 1041
2017-10-13T09:40:52Z DBG  get flow
2017-10-13T09:40:52Z DBG  lookup flow: {1041 0 255 255 255 12 255 255 255 255 255 20 255 1 0 1} => [144 177 28 86 238 13 248 202 184 68 225 84 10 1 1 115 10 1 5 37 80 0 128  99]
2017-10-13T09:40:52Z DBG  create new flow
2017-10-13T09:40:52Z DBG  unlock flows
2017-10-13T09:40:52Z DBG  Packet number: 2
2017-10-13T09:40:52Z DBG  decode packet data
2017-10-13T09:40:52Z DBG  lock flows
2017-10-13T09:40:52Z DBG  flowid: add eth
2017-10-13T09:40:52Z DBG  IPv4 packet
2017-10-13T09:40:52Z DBG  flowid: add ipv4
2017-10-13T09:40:52Z DBG  TCP packet
2017-10-13T09:40:52Z DBG  flowid: add tcp
2017-10-13T09:40:52Z DBG  flow id flags: 1041
2017-10-13T09:40:52Z DBG  get flow
2017-10-13T09:40:52Z DBG  lookup flow: {1041 0 255 255 255 12 255 255 255 255 255 20 255 1 0 1} => [144 177 28 86 238 13 248 202 184 68 225 84 10 1 1 115 10 1 5 37 80 0 128  99]
2017-10-13T09:40:52Z DBG  unlock flows
2017-10-13T09:40:52Z DBG  Packet number: 3
2017-10-13T09:40:52Z DBG  decode packet data
2017-10-13T09:40:52Z DBG  lock flows
2017-10-13T09:40:52Z DBG  flowid: add eth
2017-10-13T09:40:52Z DBG  IPv4 packet
2017-10-13T09:40:52Z DBG  flowid: add ipv4
2017-10-13T09:40:52Z DBG  TCP packet
2017-10-13T09:40:52Z DBG  flowid: add tcp
2017-10-13T09:40:52Z DBG  flow id flags: 1041
2017-10-13T09:40:52Z DBG  get flow
2017-10-13T09:40:52Z DBG  lookup flow: {1041 0 255 255 255 12 255 255 255 255 255 20 255 1 0 1} => [144 177 28 86 238 13 248 202 184 68 225 84 10 1 1 115 10 1 5 37 80 0 128  99]
2017-10-13T09:40:52Z DBG  unlock flows
2017-10-13T09:40:52Z DBG  Packet number: 4
2017-10-13T09:40:52Z DBG  decode packet data
2017-10-13T09:40:52Z DBG  lock flows
2017-10-13T09:40:52Z DBG  flowid: add eth
2017-10-13T09:40:52Z DBG  IPv4 packet
2017-10-13T09:40:52Z DBG  flowid: add ipv4
2017-10-13T09:40:52Z DBG  TCP packet
2017-10-13T09:40:52Z DBG  flowid: add tcp
2017-10-13T09:40:52Z DBG  flow id flags: 1041
2017-10-13T09:40:52Z DBG  get flow
2017-10-13T09:40:52Z DBG  lookup flow: {1041 0 255 255 255 12 255 255 255 255 255 20 255 1 0 1} => [144 177 28 86 238 13 248 202 184 68 225 84 10 1 1 115 10 1 5 37 80 0 128  99]
2017-10-13T09:40:52Z DBG  unlock flows
2017-10-13T09:40:52Z DBG  Packet number: 5
2017-10-13T09:40:52Z DBG  decode packet data
2017-10-13T09:40:52Z DBG  lock flows
2017-10-13T09:40:52Z DBG  flowid: add eth
2017-10-13T09:40:52Z DBG  IPv4 packet
2017-10-13T09:40:52Z DBG  flowid: add ipv4
2017-10-13T09:40:52Z DBG  TCP packet
2017-10-13T09:40:52Z DBG  flowid: add tcp
2017-10-13T09:40:52Z DBG  flow id flags: 1041
2017-10-13T09:40:52Z DBG  get flow
2017-10-13T09:40:52Z DBG  lookup flow: {1041 0 255 255 255 12 255 255 255 255 255 20 255 1 0 1} => [144 177 28 86 238 13 248 202 184 68 225 84 10 1 1 115 10 1 5 37 80 0 128  99]
2017-10-13T09:40:52Z DBG  unlock flows

[...]

pcap_wshark
pcap_wshark.png1297×548 54.1 KB

thanks for your help

2

How do you run packetbeat?

Are you looking at flows or are you refering to HTTP transactions from the HTTP module? Seems like you are more interested in the HTTP transactions.

HTTP is on top of TCP. The flows module only looks at src/dst + IP/Port address pairs + times out the flow if no more packets are seen in a TCP connection. Even TCP Reconnects with same ports might be counted into the same flow. There can be multiple HTTP transactions in one TCP connection.

3

Yes I had HTTP configured but found out it works better with specifying a port number in yml
ports: [80]
is there any way to catch all HTTP transaction regardless the port number, especially when reading from PCAP file ?

Now I got some results now but only halve of my requests are effectively stored in ES...
I expect 4 HTTP req/resp documents but only got 2 ...

2017-10-13T14:35:21Z INFO Total non-zero values: libbeat.es.call_count.PublishEvents=2 libbeat.es.publish.read_bytes=1076 libbeat.es.publish.write_bytes=3065 libbeat.es.published_and_acked_events=2 libbeat.publisher.messages_in_worker_queues=8 libbeat.publisher.published_events=4

4

nvm, I fixed it with -waitstop 10 :slight_smile:

5

Hi,
How did you solve this problem?
Where did you set up -waitstop 10 ?
I didnt find this property in yml.

6

Hi,
It's a command line option not a yml config

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.