Not seeing Packetbeat info (no packetbeat data in Elasticsearch) for 5.0 Alpha 4

David_McClain · July 15, 2016, 2:04am

I went through the newest Docs to install the Alpha 4 stack, and was just trying to populate it with some data.

ES 5.0-4 works
Kibana 5.0-4 works
Filebeat 5.0-4 works

But when I tried to install Packetbeat, everything seems normal, until I do the validation step (where you hit a web page and then check ES for new data in packetbeat-* indices).

I have XPACK installed and Shield enabled with the default username/password.
I am using all default configs - no changes (except for a couple of extra paths in the filebeat config for log files)
So.. ES is on 9200, Kibana is on 5601... it's all bound to localhost.

I took a trace for Packetbeats, but none of it made much sense to me, and i'm not completely sure what to check for. I did doublecheck the packetbeat config, and port 80 and 8080 were both enabled.

I am running this on a Virtual Machine through VMWare Workstation running on a Windows 10 PC.

warkolm · July 15, 2016, 2:09am

So you aren't seeing data in ES from Packetbeat?

David_McClain · July 15, 2016, 2:15am

Correct.

No data in ES.
Also, no log files in /var/log/packetbeat

I installed it via the RPM, so that should be the location of the logs. The /var/log/packetbeat dir exists, but is empty.

And the process is definitely running.

David_McClain · July 15, 2016, 2:15am

Edited the title to hopefully be a little more clear about that. My apologies

warkolm · July 15, 2016, 2:18am

What's your config look like? Make sure you format it with the </> button

David_McClain · July 15, 2016, 2:25am

I did not change the config from the default. What I downloaded/installed is what I ran.
I just double checked and the documentation for quick setup (https://www.elastic.co/guide/en/beats/packetbeat/master/configuring-packetbeat.html) basically says you can use the default unless you are running any non-standard port configurations.

Here's the config (part 1, I'm being limited to 5k characters)

#################### Packetbeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The packetbeat.full.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/packetbeat/index.html

#============================== Network device ================================

# Select the network interface to sniff the data. You can use the "any"
# keyword to sniff on all connected interfaces.
packetbeat.interfaces.device: any

#================================== Flows =====================================

# Comment out all options to disable flows reporting.

# Set network flow timeout. Flow is killed if no packet is received before being
# timed out.
packetbeat.flows.timeout: 30s

# Configure reporting period. If set to -1, only killed flows will be reported
packetbeat.flows.period: 10s

#========================== Transaction protocols =============================

packetbeat.protocols.icmp:
  # Enable ICMPv4 and ICMPv6 monitoring. Default: false
  enabled: true

packetbeat.protocols.amqp:
  # Configure the ports where to listen for AMQP traffic. You can disable
  # the AMQP protocol by commenting out the list of ports.
  ports: [5672]

packetbeat.protocols.dns:
  # Configure the ports where to listen for DNS traffic. You can disable
  # the DNS protocol by commenting out the list of ports.
  ports: [53]

  # include_authorities controls whether or not the dns.authorities field
  # (authority resource records) is added to messages.
  include_authorities: true

  # include_additionals controls whether or not the dns.additionals field
  # (additional resource records) is added to messages.
  include_additionals: true

packetbeat.protocols.http:
  # Configure the ports where to listen for HTTP traffic. You can disable
  # the HTTP protocol by commenting out the list of ports.
  ports: [80, 8080, 8000, 5000, 8002]

packetbeat.protocols.memcache:
  # Configure the ports where to listen for memcache traffic. You can disable
  # the Memcache protocol by commenting out the list of ports.
  ports: [11211]

packetbeat.protocols.mysql:
  # Configure the ports where to listen for MySQL traffic. You can disable
  # the MySQL protocol by commenting out the list of ports.
  ports: [3306]

packetbeat.protocols.pgsql:
  # Configure the ports where to listen for Pgsql traffic. You can disable
  # the Pgsql protocol by commenting out the list of ports.
  ports: [5432]

packetbeat.protocols.redis:
  # Configure the ports where to listen for Redis traffic. You can disable
  # the Redis protocol by commenting out the list of ports.
  ports: [6379]

packetbeat.protocols.thrift:
  # Configure the ports where to listen for Thrift-RPC traffic. You can disable
  # the Thrift-RPC protocol by commenting out the list of ports.
  ports: [9090]

packetbeat.protocols.mongodb:
  # Configure the ports where to listen for MongoDB traffic. You can disable
  # the MongoDB protocol by commenting out the list of ports.
  ports: [27017]

packetbeat.protocols.nfs:
  # Configure the ports where to listen for NFS traffic. You can disable
  # the NFS protocol by commenting out the list of ports.
  ports: [2049]

David_McClain · July 15, 2016, 2:25am

And config part 2:

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  # Template name. By default the template name is packetbeat.
  template.name: "packetbeat"

  # Path to template file
  template.path: "packetbeat.template.json"

  # Overwrite existing template
  template.overwrite: false

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional TLS. By default is off.
  # List of root certificates for HTTPS server verifications
  #tls.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for TLS client authentication
  #tls.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #tls.certificate_key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is error.
# Available log levels are: critical, error, warning, info, debug
#logging.level: debug

Here's the output of netstat to show that ES is running on localhost:9200:

(19:23:04)->sudo netstat -tulpn | grep 9200
tcp        0      0 ::ffff:127.0.0.1:9200       :::*                        LISTEN      4395/java           
tcp        0      0 ::1:9200                    :::*                        LISTEN      4395/java

And... here's a listing of proc 4395 just for sanity:

(19:23:09)->ps -ef | grep 4395
494        4395      1  2 18:03 ?        00:02:19 /usr/bin/java -Xms256m -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.0.0-alpha4.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid -d -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch

David_McClain · July 15, 2016, 3:05am

Solved.

It seems once I disabled security in XPack, the beats were able to come through.

steffens · July 15, 2016, 10:27am

you can configure username and password in beats elasticsearch output + configure user in shield. No need to disable security.

system · August 5, 2016, 2:05am

This topic was automatically closed after 21 days. New replies are no longer allowed.