Cannot make packetbeat work

Elastic Stack Beats
1

Hello, beats wizards.

I will greatly appreciate help in making packetbeats work.

I have the following configuration.

An Ubuntu-based sensor with two interfaces (management and sniffing) is running packetbeat version 1.1.2 (386). It is configured to send network capture data to the host with ELK stack, running logstash (version 2.2.2), over port 5044/tcp. Logstash is configured to forward the data to Elasticsearch (version 1.4.4). When I start packetbeat on the sensor, no "packetbeat-" index is being created in Elasticsearch. I have followed the recommendations on how to enable beats on ELK server.

Here is packetbeat configuration on the sensor (I took the lines with # sign out):

interfaces:
  device: eth4
  snaplen: 1514
protocols:
  dns:
    ports: [53]
    include_authorities: true
    include_additionals: true

  http:
    ports: [80, 800, 8080, 8000, 5000, 8002]

    send_all_headers: true
    split_cookie: true
  memcache:

  mysql:
  pgsql:
  redis:
  thrift:
  mongodb:
output:

  elasticsearch:
    hosts: ["10.1.5.199:5044"]
    compression_level: 0
    index: packetbeat

shipper:
logging:
  to_files: true
  files:
    path: /var/log/mybeat
    name: mybeat.log
    rotateeverybytes: 10485760 # = 10MB
    keepfiles: 7

Here is logstash configuration:

input {
  beats {
    port => 5044
    type => "packetbeat"
  }
}
output {
        if [type] == "packetbeat" {
        stdout { codec => rubydebug }
                elasticsearch {
                                hosts => "127.0.0.1:9200"
                                # sniffing => true
                                manage_template => false
                                index => "packetbeat-%{+YYYY.MM.dd}"
                                document_type => "packetbeat"
                }
        }
}

When I ran tcpdump on port 5044, I see the sensor connecting to the ELK server, but not sending the actual data, just the "HEAD" probes:

HEAD / HTTP/1.1

When I configure packetbeat to output into the file on the sensor, I see network packet data being logged. However, when logstash is configured as the output, the data is not being sent across. Could you let me know what I am doing wrong?

No errors are in logstash.err, logstash.log files. logstash.stdout file is empty.

Thanks a lot!

2

It looks like you intended to have Packetbeat -> LS -> ES, but you have Packetbeat -> ES. I made some changes:

interfaces:
  device: eth4
  snaplen: 1514
protocols:
  dns:
    ports: [53]
    include_authorities: true
    include_additionals: true
  http:
    ports: [80, 800, 8080, 8000, 5000, 8002]
    send_all_headers: true
    split_cookie: true
output:
  # You want to send packetbeat data through LS to ES? Are you planning
  # to add additional filters? You could send it straight to ES.
  logstash:
    hosts: ["10.1.5.199:5044"]
shipper:
  tags: [packetbeat]
logging:
  to_files: true
  files:
    path: /var/log/mybeat
    name: mybeat.log
    rotateeverybytes: 10485760
    keepfiles: 7

input {
  beats {
    port => 5044
  }
}

output {
  # For debugging, remove later.
  stdout { codec => rubydebug { metadata => true } }

  # If you need a conditional on the output you could use a tag. Don't use
  # type because it will be set to dns or http.
  if "packetbeat" in [tags] {
    elasticsearch {
      hosts => "127.0.0.1:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }   
  }
}
3

Thanks a lot, Andrew

This worked. I also place # sign on "elasticsearch:" as output in the config and made sure that "logstash:" was enabled.

Regards,

Alek