Packetbeat not capturing HTTP traffic

Elastic Stack Beats
#1

Hi,

I've been trying with no success to capture HTTP traffic from network device with packetbeat.

It simply seams not to capture any HTTP traffic and send it to Logstash.

Here folows the packetbeat.yml:

packetbeat.interfaces:

  • device: p9p2
  • snaplen: 1514
    packetbeat.protocols:
  • type: dns
    enabled: true
    ports: [53]
    include_authorities: true
    include_additionals: true
  • type: http
    enabled: true
    ports: [80, 800, 8080, 8000, 5000, 8002]
    send_all_headers: true
    split_cookie: true
    output:
    logstash:
    hosts: ["localhost:5044"]
    packetbeat.logging:
    to_files: true
    files:
    path: /var/log/packetbeat
    name: packetbeat.log
    rotateeverybytes: 10485760
    keepfiles: 20

Here folows the logstash mvelk.conf file:

input {
beats {
port => 5044
}
}
output {

stdout { codec => rubydebug }

file {
path => "/var/log/logstash/pack-%{+YYYY-MM-dd}.txt"
}

elasticsearch {
hosts => ["172.27.47.165:9200"]
index => "packet-%{+YYYY.MM.dd}"
document_type => "http_req"
}

}

#2

How do you test HTTP not being captured?

You use plain HTTP or HTTPS (the later is encrypted and can not be captured)?

Is DNS working?

Have you checked packetbeat logs?

#3

HTTP is correctly logged in the packetbeat log but it's not sent to logstash.

I generated cap from packetbeat and the HTTP requests seems to be there.

But they are not sent to Logstash.

2017-11-08T22:02:52-02:00 DBG TCP packet
2017-11-08T22:02:52-02:00 DBG flowid: add tcp
2017-11-08T22:02:52-02:00 DBG flowid: add tcp connection id
2017-11-08T22:02:52-02:00 DBG tcp flow id: 0xc421475fc0
2017-11-08T22:02:52-02:00 DBG pkt.start_seq=1639101846 pkt.last_seq=1639104152 stream.last_seq=1639101846 (len=2306)
2017-11-08T22:02:52-02:00 DBG Payload received: [GET /static-files/PortalCliente/js/reestruturacaoFase2/script_homeproduto_listacontratos.js HTTP/1.1
Host: cliente.portoseguro.com.br
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36
Accept: /
Referer: https://cliente.portoseguro.com.br/portaldecliente/HomeProduto?prdcod=3&srvcod=1762
Accept-Encoding: gzip, deflate, br
Accept-Language: pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7
Cookie: _tc_p=MTc5LjExMC4yMC4xNjM=; _tc_g=LTIzLjU0MzE3ODU5OTk5OTk5NywtNDYuNjI5MTg0NQ==; fbm_1583040635248759=base_domain=.portoseguro.com.br; __CT_Data=gpv=4&apv_14023_www08=4; WRUID=0; _tc_id=65067841; _tc_order=16; com.silverpop.iMAWebCookie=c422553c-abda-cab2-a758-5c1edc560664; mmcore.tst=0.191; mmapi.store.p.0=%7B%22mmparams.d%22%3A%7B%7D%2C%22mmparams.p%22%3A%7B%22pd%22%3A%221541359436560%7C%5C%2217767186%7CIgAAAApVAgD7zk76SQ8AAREAAULjcWjpEQAxsiiVuSPVSIQAdUm3%2FtRIAAAAAP%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FABpjbGllbnRlLnBvcnRvc2VndXJvLmNvbS5icgJzDwUAAwACAAAAAADIhwEA%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F8CAC58AAAYBOgjKE0PAP%2F%2F%2F%2F8BTQ93D%2F%2F%2FDQAAAQAAAAABdSUBAN3EAQAAF0oAALr6rjqxTQ8A%2F%2F%2F%2F%2FwFND3cP%2F%2F8EAAABAAAAAAEGtgAAPyABAAAAAAAAAAFF%5C%22%22%2C%22srv%22%3A%221541359436564%7C%5C%22nycvwcgus11%5C%22%22%7D%7D; BIGipServerpool_portaldecliente=1007557292.20480.0000; JSESSIONID=JdNQhDMpj7RhzLnjM6NrGp28JG2DRHl2J6Mrw4WNzqjmcb6TtTQh!-1344245898!1867175345; __utmt=1; email=soniabrufly@gmail.com; telcel=14997098640; cpf=17815077838; PSUserID=5294CF3D7CBFFC6ADE5DB5DBA688D51ABD351962BFCA4EDB5B5FFB4F6B; PSSessionID=JdNQhDMpj7RhzLnjM6NrGp28JG2DRHl2J6Mrw4WNzqjmcb6TtTQh!-1344245898!1510182078894; chave=document=415274#1391512&session=JdNQhDMpj7RhzLnjM6NrGp28JG2DRHl2J6Mrw4WNzqjmcb6TtTQh!-1344245898!1510182078894&id=17815077838&hash=3369949EAAF7B0D9752FC9A483B7127E; captionID=email=17815077838&caption=17815077838&hash=9C56F8C3C38046BBB5EB4365B5D0CBF2; _ga=GA1.3.1660967633.1505754259; _gid=GA1.3.79418552.1510182088; flagPopupExibido=S; _tc_id=65067841; _tc_order=17; _gat=1; __utma=1.1660967633.1505754259.1509823212.1510182086.13; __utmb=1.2.10.1510182086; __utmc=1; __utmz=1.1510182086.13.13.utmcsr=google.com.br|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)
X-Forwarded-For: 179.110.111.113

]
2017-11-08T22:02:52-02:00 DBG HTTP version 1.1
2017-11-08T22:02:52-02:00 DBG Data: Host: cliente.portoseguro.com.br
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36
Accept: /
Referer: https://cliente.portoseguro.com.br/portaldecliente/HomeProduto?prdcod=3&srvcod=1762
Accept-Encoding: gzip, deflate, br
Accept-Language: pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7
Cookie: _tc_p=MTc5LjExMC4yMC4xNjM=; _tc_g=LTIzLjU0MzE3ODU5OTk5OTk5NywtNDYuNjI5MTg0NQ==; fbm_1583040635248759=base_domain=.portoseguro.com.br; __CT_Data=gpv=4&apv_14023_www08=4; WRUID=0; _tc_id=65067841; _tc_order=16; com.silverpop.iMAWebCookie=c422553c-abda-cab2-a758-5c1edc560664; mmcore.tst=0.191; mmapi.store.p.0=%7B%22mmparams.d%22%3A%7B%7D%2C%22mmparams.p%22%3A%7B%22pd%22%3A%221541359436560%7C%5C%2217767186%7CIgAAAApVAgD7zk76SQ8AAREAAULjcWjpEQAxsiiVuSPVSIQAdUm3%2FtRIAAAAAP%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FABpjbGllbnRlLnBvcnRvc2VndXJvLmNvbS5icgJzDwUAAwACAAAAAADIhwEA%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F8CAC58AAAYBOgjKE0PAP%2F%2F%2F%2F8BTQ93D%2F%2F%2FDQAAAQAAAAABdSUBAN3EAQAAF0oAALr6rjqxTQ8A%2F%2F%2F%2F%2FwFND3cP%2F%2F8EAAABAAAAAAEGtgAAPyABAAAAAAAAAAFF%5C%22%22%2C%22srv%22%3A%221541359436564%7C%5C%22nycvwcgus11%5C%22%22%7D%7D; BIGipServerpool_portaldecliente=1007557292.20480.0000; JSESSIONID=JdNQhDMpj7RhzLnjM6NrGp28JG2DRHl2J6Mrw4WNzqjmcb6TtTQh!-1344245898!1867175345; __utmt=1; email=soniabrufly@gmail.com; telcel=14997098640; cpf=17815077838; PSUserID=5294CF3D7CBFFC6ADE5DB5DBA688D51ABD351962BFCA4EDB5B5FFB4F6B; PSSessionID=JdNQhDMpj7RhzLnjM6NrGp28JG2DRHl2J6Mrw4WNzqjmcb6TtTQh!-1344245898!1510182078894; chave=document=415274#1391512&session=JdNQhDMpj7RhzLnjM6NrGp28JG2DRHl2J6Mrw4WNzqjmcb6TtTQh!-1344245898!1510182078894&id=17815077838&hash=3369949EAAF7B0D9752FC9A483B7127E; captionID=email=17815077838&caption=17815077838&hash=9C56F8C3C38046BBB5EB4365B5D0CBF2; _ga=GA1.3.1660967633.1505754259; _gid=GA1.3.79418552.1510182088; flagPopupExibido=S; _tc_id=65067841; _tc_order=17; _gat=1; __utma=1.1660967633.1505754259.1509823212.1510182086.13; __utmb=1.2.10.1510182086; __utmc=1; __utmz=1.1510182086.13.13.utmcsr=google.com.br|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)
X-Forwarded-For: 179.110.111.113

pcap file

pcap
pcap.png1784×994 110 KB

Logstash output

{"source":{"port":57656,"stats":{"net_bytes_total":9647,"net_packets_total":114},"mac":"00:1c:7f:60:30:36","ip":"172.16.235.3"},"transport":"tcp","dest":{"port":80,"mac":"00:00:0c:9f:f0:b9","ip":"172.26.14.91"},"type":"flow","tags":["beats_input_raw_event"],"start_time":"2017-11-09T00:00:45.937Z","@timestamp":"2017-11-09T00:01:20.000Z","vlan":3524,"connection_id":"AQAAAAAAAAA=","last_time":"2017-11-09T00:00:49.019Z","flow_id":"FQwA/wz/Dv//////FhoBAQEAAAyf8LkAHH9gMDbEDawaDlusEOsDUAA44QEAAAAAAAAA","beat":{"name":"li3233.portoseguro.brasil","hostname":"li3233.portoseguro.brasil","version":"6.0.0-rc1"},"final":true,"@version":"1","host":"li3233.portoseguro.brasil"}

#4

Packetbeat matches requests and responses into transactions. Only full transactions are published. I don't see any response in the log or the pcap's screenshot.

#5

Steffens,

What do you mean by "response"?

Is it the response that should be sent back by the web server?

Thanks

#6

yes, the HTTP Response.

#7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.