Packetbeat Conundrum

Elastic Stack Beats
1

Hi All, right now I just talked with some wonderful engineers at Elastic about a little problem I have been having with my deployment. I have a very bear bones setup of the Elastic Stack, and I am having an issue with my packet beats. I almost 80% of my destination IPs are going to 239.255.255.250. I am pretty new to the Elastic Stack, so the configurartion of my yml files are very very basic. I am wondering if anyone else has ran into this problem as well. (the other 20% of IPs are an APIPA address)

Thank you guys I could totally use the help.

2

Can you tell us a little bit more about your deployment. Is packetbeat running on a host, or inside a docker container or k8s?

Which interface have you configured Packetbeat to listen from?

Whis OS?

3

Right now I have the packetbeat running on a host. And I have my Logstash setup to listen for my packetbeat. Finally the OS I have all of my Elastic Stack running on is Ubuntu Live 18.04.

4

^ This answer is missing.

I mean the configuration setting for packetbeat.interfaces.device.

It should have the name of network interface (i.e. eth0, en1, enp0s3, etc.) where you want Packetbeat to capture traffic.

It seems that you are listening for a secondary interface that is only receiving broadcast traffic from your host. Check with the output of ifconfig

5

I'll check that!

6

I put that in the pipelines.yml file and it didnt seem to do anything. Did I do that right?

7

You mean packetbeat.yml ?

The setting is already there, at the start of the file, you need to overwrite it.

If you still having problems, please paste your full packetbeat.yml and the output of ifconfig

8

Ill try it again and get back to you.

9

Alright I found out it wasn't packet beat causing the problem it is logstash how should I have my pipelines.yml setup so it talks to packetbeat?

Right now it looks like this.

pipeline.id: main
pipeline.workers: 1
pipeline.batch.size: 1
config.string: "input { stdin {} } filter { sleep { time => 1 } } output { stdout { codec => dots } }"
10

It seems like logstash is just running and not talking to anyone.

11

Can you run Packetbeat with the -d '*' flag and generate some traffic for it to capture, so we can see if it's connecting to Logstash. Then share your log here.

Also share your full Logstash configuration, I will find someone from the team to look at it.

12

Alrighty, I did that and I got network cannot be found. Which now make sense since the dev network I am working on can communicate on any port outside of its self beside ones used for the internet. So, on my end we just have to change configuration, so we can talk on the desired ports. As for logstash, it seems to me that there is no reason why it shouldn't be able to talk to elastic search. I am able to run it, and it's configured to talk to my elasticsearch server. But when I go to elasticsearch and use somthing like this command curl 'localhost:9200/_cat/indices?v' to see it on elasticsearch and nothing is coming up. I am only getting kibana logs from a few days ago.

input {
 beats {
    port => 5044
 }
}

output {
   elasticsearch {
        hosts => "10.101.0.141:9200"
        manage_template => false
        index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
        document_type => "%{{@metadata][type]}"
       }
}

This is what my logstash.yml file looks like. Should it matter if I have that manage_template in there? I am not sure what is bugging all this up.

13

Can you share your Packetbeat logs? If there's a problem connecting to Logstash there should be an indication there.

How did you configure Packetbeat to talk to Logstash? Can you share also your packetbeat.yml?

14 
#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  hosts: ["10.101.0.140:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#============================== Network device ================================

# Select the network interface to sniff the data. On Linux, you can use the
# "any" keyword to sniff on all connected interfaces.
packetbeat.interfaces.device: ens160

#================================== Flows =====================================

# Set `enabled: false` or comment out all options to disable flows reporting.
packetbeat.flows:
  # Set network flow timeout. Flow is killed if no packet is received before being
  # timed out.
  timeout: 30s

  # Configure reporting period. If set to -1, only killed flows will be reported
  period: 10s

#========================== Transaction protocols =============================

packetbeat.protocols:
- type: icmp
  # Enable ICMPv4 and ICMPv6 monitoring. Default: false
  enabled: true

- type: amqp
  # Configure the ports where to listen for AMQP traffic. You can disable
  # the AMQP protocol by commenting out the list of ports.
  ports: [5672]

- type: cassandra
  #Cassandra port for traffic monitoring.
  ports: [9042]

- type: dhcpv4
  # Configure the DHCP for IPv4 ports.
  ports: [67, 68]

- type: dns
  # Configure the ports where to listen for DNS traffic. You can disable
  # the DNS protocol by commenting out the list of ports.
  ports: [53]

- type: http
  # Configure the ports where to listen for HTTP traffic. You can disable
  # the HTTP protocol by commenting out the list of ports.
  ports: [80, 8080, 8000, 5000, 8002]

- type: memcache
  # Configure the ports where to listen for memcache traffic. You can disable
  # the Memcache protocol by commenting out the list of ports.
  ports: [11211]

- type: mysql
  # Configure the ports where to listen for MySQL traffic. You can disable
  # the MySQL protocol by commenting out the list of ports.
  ports: [3306,3307]

- type: pgsql
  # Configure the ports where to listen for Pgsql traffic. You can disable
  # the Pgsql protocol by commenting out the list of ports.
  ports: [5432]

- type: redis
  # Configure the ports where to listen for Redis traffic. You can disable
  # the Redis protocol by commenting out the list of ports.
  ports: [6379]

- type: thrift
  # Configure the ports where to listen for Thrift-RPC traffic. You can disable
  # the Thrift-RPC protocol by commenting out the list of ports.
  ports: [9090]

- type: mongodb
  # Configure the ports where to listen for MongoDB traffic. You can disable
  # the MongoDB protocol by commenting out the list of ports.
  ports: [27017]

- type: nfs
  # Configure the ports where to listen for NFS traffic. You can disable
  # the NFS protocol by commenting out the list of ports.
  ports: [2049]

- type: tls
  # Configure the ports where to listen for TLS traffic. You can disable
  # the TLS protocol by commenting out the list of ports.
  ports:
    - 443   # HTTPS
    - 993   # IMAPS
    - 995   # POP3S
    - 5223  # XMPP over SSL
    - 8443
    - 8883  # Secure MQTT
    - 9243  # Elasticsearch
15 
C:\Users\alexk\OneDrive\Desktop\My Beats\packetbeat-7.1.1-windows-x86_64\packetbeat-7.1.1-windows-x86_64>.\packetbeat -e -c packetbeat.yml
2019-06-04T11:00:28.362-0500    INFO    instance/beat.go:571    Home path: [C:\Users\alexk\OneDrive\Desktop\My Beats\packetbeat-7.1.1-windows-x86_64\packetbeat-7.1.1-windows-x86_64] Config path: [C:\Users\alexk\OneDrive\Desktop\My Beats\packetbeat-7.1.1-windows-x86_64\packetbeat-7.1.1-windows-x86_64] Data path: [C:\Users\alexk\OneDrive\Desktop\My Beats\packetbeat-7.1.1-windows-x86_64\packetbeat-7.1.1-windows-x86_64\data] Logs path: [C:\Users\alexk\OneDrive\Desktop\My Beats\packetbeat-7.1.1-windows-x86_64\packetbeat-7.1.1-windows-x86_64\logs]
2019-06-04T11:00:28.381-0500    INFO    instance/beat.go:579    Beat ID: 92508ebb-7001-4a6b-8303-ef6e87b4b934
2019-06-04T11:00:28.382-0500    INFO    [index-management.ilm]  ilm/ilm.go:129  Policy name: packetbeat-7.1.1
2019-06-04T11:00:28.418-0500    INFO    [beat]  instance/beat.go:827    Beat info       {"system_info": {"beat": {"path": {"config": "C:\\Users\\alexk\\OneDrive\\Desktop\\My Beats\\packetbeat-7.1.1-windows-x86_64\\packetbeat-7.1.1-windows-x86_64", "data": "C:\\Users\\alexk\\OneDrive\\Desktop\\My Beats\\packetbeat-7.1.1-windows-x86_64\\packetbeat-7.1.1-windows-x86_64\\data", "home": "C:\\Users\\alexk\\OneDrive\\Desktop\\My Beats\\packetbeat-7.1.1-windows-x86_64\\packetbeat-7.1.1-windows-x86_64", "logs": "C:\\Users\\alexk\\OneDrive\\Desktop\\My Beats\\packetbeat-7.1.1-windows-x86_64\\packetbeat-7.1.1-windows-x86_64\\logs"}, "type": "packetbeat", "uuid": "92508ebb-7001-4a6b-8303-ef6e87b4b934"}}}
2019-06-04T11:00:28.418-0500    INFO    [beat]  instance/beat.go:836    Build info      {"system_info": {"build": {"commit": "3358d9a5a09e3c6709a2d3aaafde628ea34e8419", "libbeat": "7.1.1", "time": "2019-05-23T13:14:34.000Z", "version": "7.1.1"}}}
2019-06-04T11:00:28.420-0500    INFO    [beat]  instance/beat.go:839    Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.11.5"}}}
2019-06-04T11:00:28.452-0500    INFO    [beat]  instance/beat.go:843    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-05-31T14:42:34.46-05:00","name":"Alex-Kratobot","ip":["fe80::4949:dfc4:52b5:8aba/64","10.20.20.166/24","fe80::8888:b834:8158:bc68/64","169.254.188.104/16","fe80::d9b8:218c:93fb:db79/64","169.254.219.121/16","2600:1702:4340:48c0::29/128","fe80::c4fe:d2dc:a394:bbb1/64","10.20.20.151/24","fe80::7dfd:f9b2:bd:8357/64","169.254.131.87/16","::1/128","127.0.0.1/8"],"kernel_version":"10.0.17134.799 (WinBuild.160101.0800)","mac":["00:24:32:16:d7:79","2a:16:a8:47:10:4d","2a:16:a8:47:15:4d","28:16:a8:47:11:4c","28:16:a8:47:11:4d"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Home","version":"10.0","major":10,"minor":0,"patch":0,"build":"17134.799"},"timezone":"CDT","timezone_offset_sec":-18000,"id":"551943ec-272f-458d-8a58-bcda0501cd05"}}}
2019-06-04T11:00:28.458-0500    INFO    [beat]  instance/beat.go:872    Process info    {"system_info": {"process": {"cwd": "C:\\Users\\alexk\\OneDrive\\Desktop\\My Beats\\packetbeat-7.1.1-windows-x86_64\\packetbeat-7.1.1-windows-x86_64", "exe": "C:\\Users\\alexk\\OneDrive\\Desktop\\My Beats\\packetbeat-7.1.1-windows-x86_64\\packetbeat-7.1.1-windows-x86_64\\packetbeat.exe", "name": "packetbeat.exe", "pid": 15196, "ppid": 5200, "start_time": "2019-06-04T11:00:27.711-0500"}}}
2019-06-04T11:00:28.458-0500    INFO    instance/beat.go:280    Setup Beat: packetbeat; Version: 7.1.1
2019-06-04T11:00:28.459-0500    INFO    [index-management]      idxmgmt/std.go:165      Set output.elasticsearch.index to 'packetbeat-7.1.1' as ILM is enabled.
2019-06-04T11:00:28.459-0500    INFO    elasticsearch/client.go:165     Elasticsearch url: http://10.101.0.140:5044
2019-06-04T11:00:28.461-0500    INFO    [publisher]     pipeline/module.go:97   Beat name: Alex-Kratobot
2019-06-04T11:00:28.462-0500    INFO    procs/procs.go:105      Process watcher disabled
2019-06-04T11:00:28.483-0500    INFO    instance/beat.go:391    packetbeat start running.
2019-06-04T11:00:28.485-0500    INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s
2019-06-04T11:00:28.629-0500    INFO    flows/util.go:64        flows worker loop stopped
2019-06-04T11:00:28.963-0500    INFO    [monitoring]    log/log.go:152  Total non-zero metrics  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":359,"time":{"ms":390}},"total":{"ticks":671,"time":{"ms":718},"value":671},"user":{"ticks":312,"time":{"ms":328}}},"handles":{"open":260},"info":{"ephemeral_id":"14d302c0-ef00-4240-9988-838528a3e3b9","uptime":{"ms":392}},"memstats":{"gc_next":35726800,"memory_alloc":19272912,"memory_total":23916216,"rss":48041984}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":14,"events":{"active":0}}},"system":{"cpu":{"cores":8}}}}}
2019-06-04T11:00:28.964-0500    INFO    [monitoring]    log/log.go:153  Uptime: 727.0214ms
2019-06-04T11:00:28.984-0500    INFO    [monitoring]    log/log.go:130  Stopping metrics logging.
2019-06-04T11:00:29.000-0500    INFO    instance/beat.go:401    packetbeat stopped.
2019-06-04T11:00:29.020-0500    ERROR   instance/beat.go:802    Exiting: Sniffer main loop failed: Error starting sniffer: Error opening adapter: The system cannot find the device specified. (20)
Exiting: Sniffer main loop failed: Error starting sniffer: Error opening adapter: The system cannot find the device specified. (20)
16

Sorry this is all really messy.

17

It's okay; I'm going back to add code-fences around those blocks to make them a bit easier to read. You can do it yourself on future posts by adding a line with just three tildes (~~~) to start a code block and another to end it, "fencing" it off from the auto-formatting.

1 Like
18

The line with output.logstash in your packetbeat.yml needs to be uncommented; it should be:

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["10.101.0.140:5044"]
1 Like
19 
2019-06-04T12:12:13.785-0500    INFO    instance/beat.go:579    Beat ID: 92508ebb-7001-4a6b-8303-ef6e87b4b934
2019-06-04T12:12:13.803-0500    INFO    [index-management.ilm]  ilm/ilm.go:129  Policy name: packetbeat-7.1.1
2019-06-04T12:12:13.840-0500    INFO    [beat]  instance/beat.go:827    Beat info       {"system_info": {"beat": {"path": {"config": "C:\\Users\\alexk\\OneDrive\\Desktop\\My Beats\\packetbeat-7.1.1-windows-x86_64\\packetbeat-7.1.1-windows-x86_64", "data": "C:\\Users\\alexk\\OneDrive\\Desktop\\My Beats\\packetbeat-7.1.1-windows-x86_64\\packetbeat-7.1.1-windows-x86_64\\data", "home": "C:\\Users\\alexk\\OneDrive\\Desktop\\My Beats\\packetbeat-7.1.1-windows-x86_64\\packetbeat-7.1.1-windows-x86_64", "logs": "C:\\Users\\alexk\\OneDrive\\Desktop\\My Beats\\packetbeat-7.1.1-windows-x86_64\\packetbeat-7.1.1-windows-x86_64\\logs"}, "type": "packetbeat", "uuid": "92508ebb-7001-4a6b-8303-ef6e87b4b934"}}}
2019-06-04T12:12:13.840-0500    INFO    [beat]  instance/beat.go:836    Build info      {"system_info": {"build": {"commit": "3358d9a5a09e3c6709a2d3aaafde628ea34e8419", "libbeat": "7.1.1", "time": "2019-05-23T13:14:34.000Z", "version": "7.1.1"}}}
2019-06-04T12:12:13.856-0500    INFO    [beat]  instance/beat.go:839    Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.11.5"}}}
2019-06-04T12:12:13.899-0500    INFO    [beat]  instance/beat.go:843    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-05-31T14:42:34.46-05:00","name":"Alex-Kratobot","ip":["fe80::4949:dfc4:52b5:8aba/64","10.20.20.166/24","fe80::8888:b834:8158:bc68/64","169.254.188.104/16","fe80::d9b8:218c:93fb:db79/64","169.254.219.121/16","2600:1702:4340:48c0::29/128","fe80::c4fe:d2dc:a394:bbb1/64","10.20.20.151/24","fe80::7dfd:f9b2:bd:8357/64","169.254.131.87/16","::1/128","127.0.0.1/8"],"kernel_version":"10.0.17134.799 (WinBuild.160101.0800)","mac":["00:24:32:16:d7:79","2a:16:a8:47:10:4d","2a:16:a8:47:15:4d","28:16:a8:47:11:4c","28:16:a8:47:11:4d"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Home","version":"10.0","major":10,"minor":0,"patch":0,"build":"17134.799"},"timezone":"CDT","timezone_offset_sec":-18000,"id":"551943ec-272f-458d-8a58-bcda0501cd05"}}}
2019-06-04T12:12:13.905-0500    INFO    [beat]  instance/beat.go:872    Process info    {"system_info": {"process": {"cwd": "C:\\Users\\alexk\\OneDrive\\Desktop\\My Beats\\packetbeat-7.1.1-windows-x86_64\\packetbeat-7.1.1-windows-x86_64", "exe": "C:\\Users\\alexk\\OneDrive\\Desktop\\My Beats\\packetbeat-7.1.1-windows-x86_64\\packetbeat-7.1.1-windows-x86_64\\packetbeat.exe", "name": "packetbeat.exe", "pid": 17456, "ppid": 6172, "start_time": "2019-06-04T12:12:13.599-0500"}}}
2019-06-04T12:12:13.923-0500    INFO    instance/beat.go:280    Setup Beat: packetbeat; Version: 7.1.1
2019-06-04T12:12:13.932-0500    INFO    [publisher]     pipeline/module.go:97   Beat name: Alex-Kratobot
2019-06-04T12:12:13.934-0500    INFO    procs/procs.go:105      Process watcher disabled
2019-06-04T12:12:13.964-0500    INFO    instance/beat.go:391    packetbeat start running.
2019-06-04T12:12:13.964-0500    INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s
2019-06-04T12:12:14.078-0500    INFO    flows/util.go:64        flows worker loop stopped
2019-06-04T12:12:14.312-0500    INFO    [monitoring]    log/log.go:152  Total non-zero metrics  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":343,"time":{"ms":375}},"total":{"ticks":624,"time":{"ms":671},"value":624},"user":{"ticks":281,"time":{"ms":296}}},"handles":{"open":254},"info":{"ephemeral_id":"8fed38b6-71d7-4d49-a1f2-fcdd0bd2e297","uptime":{"ms":362}},"memstats":{"gc_next":35162928,"memory_alloc":20328840,"memory_total":23870848,"rss":47980544}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":14,"events":{"active":0}}},"system":{"cpu":{"cores":8}}}}}
2019-06-04T12:12:14.314-0500    INFO    [monitoring]    log/log.go:153  Uptime: 597.6368ms
2019-06-04T12:12:14.319-0500    INFO    [monitoring]    log/log.go:130  Stopping metrics logging.
2019-06-04T12:12:14.335-0500    INFO    instance/beat.go:401    packetbeat stopped.
2019-06-04T12:12:14.356-0500    ERROR   instance/beat.go:802    Exiting: Sniffer main loop failed: Error starting sniffer: Error opening adapter: The system cannot find the device specified. (20)
Exiting: Sniffer main loop failed: Error starting sniffer: Error opening adapter: The system cannot find the device specified. (20) ~~~

This is what happened after I changed the yml file
20

Does the ens160 device show up when you run ifconfig ens160 or ip link ens160?