Why packetbeat not capture this http request?

[shell.b2t]

Hi :

my env is packetbeat 5.1.1,include_body_for config as as follows：

include_body_for: ['image/webp','application/x-javascript','*/*','application/octet-stream','text/xml','application/xml','application/xhtml+xml','text/plain','text/html','application/json','text/javascript','application/javascript','application/x-www-form-urlencoded']

request as follows:

POST /*******/html5/login.do HTTP/1.1
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/3 (X13 Window NT 3.3 rv:20.17.8) Gecko/20100101 Firefox/20.17.8 Firefox/
Accept-Encoding: gzip, deflate, sdch
Accept-Language: zh-CN,zh;q=0.8
Content-Length: 39
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: kybpc.chexian.sinosig.com

username=****&password=****

but packetbeat not capture this request,why?

[steffens (Steffen Siering)]

Can you run packetbeat with htpp module debug enabled and share logs?

run packetbeat with -d 'http,httpdetailed' to enable most detailed http analyzer logs.

[shell.b2t]

Hi steffens:
thk !
i run pb with-e -d 'http,httpdetailed' the detail as follows:

2016/12/27 16:20:30.638603 http.go:409: DBG  Received response with tuple: TcpTuple src[10.10.246.44:7001] dst[10.10.228.39:43932] stream_id[382]
2016/12/27 16:20:30.638615 http.go:420: DBG  Response from unknown transaction. Ingoring.
2016/12/27 16:20:30.667246 http.go:329: DBG  Received FIN
2016/12/27 16:20:30.668631 http.go:329: DBG  Received FIN
2016/12/27 16:20:30.669150 http.go:329: DBG  Received FIN
2016/12/27 16:20:30.675667 http.go:329: DBG  Received FIN
2016/12/27 16:20:30.690571 http.go:329: DBG  Received FIN
2016/12/27 16:20:30.782974 http.go:329: DBG  Received FIN
2016/12/27 16:20:30.783021 http.go:329: DBG  Received FIN
2016/12/27 16:20:30.786207 http.go:267: DBG  Payload received: [POST /xxxxxxxx/html5/login.do?username=test11111111&password=test&checkcode=l28z HTTP/1.1
Host: test.xxxx.xxx.com
Connection: keep-alive
Content-Length: 0
Accept: application/json, text/javascript, */*; q=0.01
Origin: http://test.xxxx.xxx.com
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
Referer: http://test.xxxx.xxx.com/xxxxxxxx/html5/login.html
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.8
Cookie: JSESSIONID=-UPMXoxikuH1FeYaZrTRIh0t.undefined

]
2016/12/27 16:20:30.786272 http_parser.go:194: DBG  HTTP version 1.1
2016/12/27 16:20:30.786292 http_parser.go:307: DBG  Data: Host: test.xxxx.xxx.com
Connection: keep-alive
Content-Length: 0
Accept: application/json, text/javascript, */*; q=0.01
Origin: http://test.xxxx.xxx.com
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
Referer: http://test.xxxx.xxx.com/xxxxxxxx/html5/login.html
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.8
Cookie: JSESSIONID=-UPMXoxikuH1FeYaZrTRIh0t.undefined

but no data found in es.

[steffens (Steffen Siering)]

The log says

2016/12/27 16:20:30.638603 http.go:409: DBG  Received response with tuple: TcpTuple src[10.10.246.44:7001] dst[10.10.228.39:43932] stream_id[382]
2016/12/27 16:20:30.638615 http.go:420: DBG  Response from unknown transaction. Ingoring.
2016/12/27 16:20:30.667246 http.go:329: DBG  Received FIN
...
2016/12/27 16:20:30.786207 http.go:267: DBG  Payload received: [POST /xxxxxxxx/html5/login.do?username=test11111111&password=test&checkcode=l28z HTTP/1.1

interestingly the is a POST request being parsed after the response. Currently the http module requires a complete transaction in order to publish an transaction event. Plus, it requires the Request being parsed before the Response.

DO you use a pcap or life traffic? What kind of HTTP request are we dealing with. I think it's pretty much possible in HTTP to see part of the response before the request is finished (e.g. uploading big chunk of data/file?). This might indeed be a problem for packetbeat, right now.

We would need a trace (pcap file) to get an idea of what's actually happening here.

[shell.b2t]

thk steffens:

According to your tips,I found the reason。The problem in the port monitor。