Packetbeat windows - all_headers not working?

Elastic Stack Beats
1

Sorry guys - I'm a newbie to ELK.
So I've got packetbeat working (sort of) fine on Win 10.
Local copy of Elasticsearch receiving direct docuemts from the beat. Kibana displaying everything - sort of.

Having edited the packetbeat.yml to include:
send_headers: ["User-Agent", "Cookie", "Set-Cookie"]
send_all_headers: true

(note: I'm guessing that if "send_all_headers: true" is enabled then I don't need the line before specifying user-agent, cookoie and set-cookie)

I then restarted the packetbeat service expecting to see some http headers. Nothing. Do I need to modify the template file in-order to see additional content in what was being captured.

I checked the PS script to make sure I wasn't editing a YML file that wasn't referenced by the service (wrong path, name, etc.) but it all seems fine.

Any suggestions gratefully appreciated. I got this working fine on my mac but for some reason Windows is proving ticky.

I've not yet got onto the subject of "how do I now index the new fields that are being passed" (a subsequent question I'm afraid).

Andrew

2

What Packetbeat version?

Could you please provide the complete config file you are using so that we can check subtle things like indentation. Surround the config with three backticks on each side to get it formatted correctly here.

3

Andrew - thanks for replying.

Version of packetbeat is: packetbeat-5.1.1-windows-x86_64
Full yml as follows:

#################### Packetbeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The packetbeat.full.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/packetbeat/index.html

#============================== Network device ================================

# Select the network interface to sniff the data. On Linux, you can use the
# "any" keyword to sniff on all connected interfaces.
packetbeat.interfaces.device: 0

#================================== Flows =====================================

# Set `enabled: false` or comment out all options to disable flows reporting.
packetbeat.flows:
  # Set network flow timeout. Flow is killed if no packet is received before being
  # timed out.
  timeout: 30s

  # Configure reporting period. If set to -1, only killed flows will be reported
  period: 10s

#========================== Transaction protocols =============================

packetbeat.protocols.icmp:
  # Enable ICMPv4 and ICMPv6 monitoring. Default: false
  enabled: true

packetbeat.protocols.amqp:
  # Configure the ports where to listen for AMQP traffic. You can disable
  # the AMQP protocol by commenting out the list of ports.
  ports: [5672]

packetbeat.protocols.cassandra:
  #Cassandra port for traffic monitoring.
  ports: [9042]

packetbeat.protocols.dns:
  # Configure the ports where to listen for DNS traffic. You can disable
  # the DNS protocol by commenting out the list of ports.
  ports: [53]

  # include_authorities controls whether or not the dns.authorities field
  # (authority resource records) is added to messages.
  include_authorities: true

  # include_additionals controls whether or not the dns.additionals field
  # (additional resource records) is added to messages.
  include_additionals: true

packetbeat.protocols.http:
  # Configure the ports where to listen for HTTP traffic. You can disable
  # the HTTP protocol by commenting out the list of ports.
  ports: [80, 8080, 8000, 5000, 8002]
  send_headers: ["User-Agent", "Cookie", "Set-Cookie"]
  send_all_headers: true
  send_response: true
  include_body_for: ["text/html"]
  split_cookie: true
  real_ip_header: "X-Forwarded-For"

packetbeat.protocols.memcache:
  # Configure the ports where to listen for memcache traffic. You can disable
  # the Memcache protocol by commenting out the list of ports.
  ports: [11211]

packetbeat.protocols.mysql:
  # Configure the ports where to listen for MySQL traffic. You can disable
  # the MySQL protocol by commenting out the list of ports.
  ports: [3306]

packetbeat.protocols.pgsql:
  # Configure the ports where to listen for Pgsql traffic. You can disable
  # the Pgsql protocol by commenting out the list of ports.
  ports: [5432]

packetbeat.protocols.redis:
  # Configure the ports where to listen for Redis traffic. You can disable
  # the Redis protocol by commenting out the list of ports.
  ports: [6379]

packetbeat.protocols.thrift:
  # Configure the ports where to listen for Thrift-RPC traffic. You can disable
  # the Thrift-RPC protocol by commenting out the list of ports.
  ports: [9090]

packetbeat.protocols.mongodb:
  # Configure the ports where to listen for MongoDB traffic. You can disable
  # the MongoDB protocol by commenting out the list of ports.
  ports: [27017]

packetbeat.protocols.nfs:
  # Configure the ports where to listen for NFS traffic. You can disable
  # the NFS protocol by commenting out the list of ports.
  ports: [2049]

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  pipeline: geoip-info

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]
4

Try disabling split_cookies. It looks like there is a logic bug in the code where it never stores the non-"Cookie" headers if "split_cookies" is enabled.

I just realized there is an open PR to fix the bug: https://github.com/elastic/beats/pull/3065

5

Andrew - thanks. This has worked!

For your next trick do you want to tell me an easy way to index the http hearders that are now being sent or am I going to be editing the packetbeat json file?

6

Remove the send_headers: ["User-Agent", "Cookie", "Set-Cookie"] and use send_all_headers: true.

7

Worked like a charm. Thank you.

Although I don't understand why specifying explicit headers in the YML prevented indices being created.

8

I opened another thread about this and other problems with cookie parser.
Handling cookies in HTTP parser

You will also notice that if the server response has more than one cookie, the cookie names are overwritten.

9

Well there's a gotcha I'm pleased you told me about.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.