Filebeat cisco modue "Error starting the server address already in use"

Elastic Stack Beats
1

i am trying to setup log server for network devices using ELK and filebeat with Ubuntu 18, but kibana doesn't display any output. when i run filebeat -e i get the following messages:
2020-02-20T14:53:10.891Z INFO udp/input.go:103 Starting UDP input
2020-02-20T14:53:10.891Z ERROR udp/input.go:106 Error running harvester: listen udp 127.0.0.1:9001: bind: address already in use
2020-02-20T14:53:10.891Z ERROR udp/input.go:106 Error running harvester: listen udp 127.0.0.1:9003: bind: address already in use
2020-02-20T14:53:13.785Z INFO add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2020-02-20T14:53:20.891Z INFO [syslog] syslog/input.go:155 Starting Syslog input {"protocol": "udp"}
2020-02-20T14:53:20.891Z ERROR [syslog] syslog/input.go:158 Error starting the servererrorlisten udp 127.0.0.1:9002: bind: address already in use
2020-02-20T14:53:30.892Z INFO [syslog] syslog/input.go:155 Starting Syslog input {"protocol": "udp"}
2020-02-20T14:53:30.892Z ERROR [syslog] syslog/input.go:158

i have checked that i am receiving logs from ASA:

15:06:23.156798 IP (tos 0x0, ttl 255, id 12006, offset 0, flags [none], proto UDP (17), length 223)
192.168.x.x.syslog > 192.168.x.x.9001: SYSLOG, length: 195
Facility local4 (20), Severity info (6)
Msg: Feb 20 2020 17:27:17 xxx-ASAFW : %ASA-6-302015: Built inbound UDP connection 3505019954 for OUTSIDE:31.13.x.x/29587 (31.13.x.x/29587) to HOSTNAME:x.x.5.9/53 (x.x.5.9/53)\0x0a

the server is listening on the same ports:

root@elk:~# netstat -ntlpu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 779/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1164/sshd
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 1033/node
tcp6 0 0 127.0.0.1:9200 :::* LISTEN 1027/java
tcp6 0 0 127.0.0.1:9300 :::* LISTEN 1027/java
tcp6 0 0 :::22 :::* LISTEN 1164/sshd
udp 0 0 127.0.0.1:9001 0.0.0.0:* 1940/filebeat
udp 0 0 127.0.0.1:9002 0.0.0.0:* 1940/filebeat
udp 0 0 127.0.0.1:9003 0.0.0.0:* 1940/filebeat
udp 0 0 127.0.0.53:53 0.0.0.0:* 779/systemd-resolve

any help?
thanks

2

You could stop the filebeat service first sudo service filebeat stop, and then run your command filebeat -e. I've seen that before while I was trying to run filebeat -e and the service was already running and using the same port.

Make sure before you run the command filebeat -e, there is no filebeat service or other filebeat process running on that system.

3

thank you but stopping the service caused the server to stop listening on those ports even though when running filebeat -e i get the following:
2020-02-20T16:20:35.328Z INFO [udp] udp/server.go:81 Started listening for UDP connection {"address": "localhost:9001"}
2020-02-20T16:20:35.328Z INFO [udp] udp/server.go:81 Started listening for UDP connection {"address": "localhost:9002"}
2020-02-20T16:20:35.328Z INFO [udp] udp/server.go:81 Started listening for UDP connection {"address": "localhost:9003"}
2020-02-20T16:20:38.237Z INFO add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.

and the same result, nothing on kibana.

4

After you run the command filebeat -e, do you see the filebeat processes with the 900x ports open? You can check this by running netstat -ntlpu

5

i checked they are not open any more:
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 779/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1164/sshd
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 1033/node
tcp6 0 0 127.0.0.1:9200 :::* LISTEN 1027/java
tcp6 0 0 127.0.0.1:9300 :::* LISTEN 1027/java
tcp6 0 0 :::22 :::* LISTEN 1164/sshd
udp 0 0 127.0.0.53:53 0.0.0.0:* 779/systemd-resolve

6

Run the filebeat -e command first, run the netstat -ntlpu to see if the process has started, is running or listening.

You can also check Kibana Logs (https://www.elastic.co/guide/en/kibana/current/xpack-logs.html) to see if data comes in, but that should be check only if the filebeat is running.

7

So maybe the filebeat can't start. What's the complete error? Right after you run the command.

8

kibana says:

There are no log messages to display

9

You can also check Kibana Logs (https://www.elastic.co/guide/en/kibana/current/xpack-logs.html) to see if data comes in, but that should be check only if the filebeat is running.

10

2020-02-20T16:46:20.969Z ERROR udp/input.go:106 Error running harvester: listen udp 127.0.0.1:9001: bind: address already in use
2020-02-20T16:46:20.969Z ERROR udp/input.go:106 Error running harvester: listen udp 127.0.0.1:9003: bind: address already in use
2020-02-20T16:46:20.969Z ERROR [syslog] syslog/input.go:158 Error starting the servererrorlisten udp 127.0.0.1:9002: bind: address already in use
2020-02-20T16:46:25.661Z INFO [udp] udp/server.go:140 Stopping UDP server {"address": "localhost:9001"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x25c3481]

11

This is a second process you're trying to start, again the same issue you had at the first place.

12

Can you make sure to check if you're running a filebeat before starting it? You can't bind the same port to two different processes.

13

i did the following:
root@elk:~# sudo service filebeat stop

root@elk:~# ps -aux |grep filebeat
root 13950 0.0 0.0 13136 1040 pts/1 S+ 16:53 0:00 grep --color=auto filebeat

root@elk:~# netstat -ntlpu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 779/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1164/sshd
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 1033/node
tcp6 0 0 127.0.0.1:9200 :::* LISTEN 1027/java
tcp6 0 0 127.0.0.1:9300 :::* LISTEN 1027/java
tcp6 0 0 :::22 :::* LISTEN 1164/sshd
udp 0 0 127.0.0.53:53 0.0.0.0:* 779/systemd-resolve

14

So this looks like you're not running it. And what happens when you run the command filebeat -e? Can you paste the complete error message here after you run the command?

15

root@elk:~# filebeat -e
2020-02-20T17:20:16.999Z INFO instance/beat.go:610 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2020-02-20T17:20:17.000Z INFO instance/beat.go:618 Beat ID: 18448d51-50ed-4281-9db2-d7aa9e816b1a
2020-02-20T17:20:17.004Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2020-02-20T17:20:17.004Z INFO [beat] instance/beat.go:941 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "18448d51-50ed-4281-9db2-d7aa9e816b1a"}}}
2020-02-20T17:20:17.004Z INFO [beat] instance/beat.go:950 Build info {"system_info": {"build": {"commit": "a9c141434cd6b25d7a74a9c770be6b70643dc767", "libbeat": "7.5.2", "time": "2020-01-15T11:13:22.000Z", "version": "7.5.2"}}}
2020-02-20T17:20:17.004Z INFO [beat] instance/beat.go:953 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.12.12"}}}
2020-02-20T17:20:17.005Z INFO [beat] instance/beat.go:957 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-02-20T17:17:12Z","containerized":false,"name":"elk","ip":["127.0.0.1/8","::1/128","192.168.6.85/24","fe80::250:56ff:fe94:54bf/64"],"kernel_version":"4.15.0-88-generic","mac":["00:50:56:94:54:bf"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"18.04.3 LTS (Bionic Beaver)","major":18,"minor":4,"patch":3,"codename":"bionic"},"timezone":"UTC","timezone_offset_sec":0,"id":"fc0893db1a894c06b7971587bc623544"}}}
2020-02-20T17:20:17.005Z INFO [beat] instance/beat.go:986 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/root", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 2023, "ppid": 1764, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-02-20T17:20:16.700Z"}}}
2020-02-20T17:20:17.005Z INFO instance/beat.go:297 Setup Beat: filebeat; Version: 7.5.2
2020-02-20T17:20:17.005Z INFO [index-management] idxmgmt/std.go:182 Set output.elasticsearch.index to 'filebeat-7.5.2' as ILM is enabled.
2020-02-20T17:20:17.006Z INFO elasticsearch/client.go:171 Elasticsearch url: http://localhost:9200
2020-02-20T17:20:17.006Z INFO [publisher] pipeline/module.go:97 Beat name: elk
2020-02-20T17:20:17.007Z INFO instance/beat.go:429 filebeat start running.
2020-02-20T17:20:17.008Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-02-20T17:20:17.008Z INFO registrar/registrar.go:145 Loading registrar data from /var/lib/filebeat/registry/filebeat/data.json
2020-02-20T17:20:17.008Z INFO registrar/registrar.go:152 States Loaded from registrar: 0
2020-02-20T17:20:17.008Z INFO crawler/crawler.go:72 Loading Inputs: 1
2020-02-20T17:20:17.014Z WARN [cfgwarn] syslog/input.go:111 EXPERIMENTAL: Syslog input type is used
2020-02-20T17:20:17.018Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 0
2020-02-20T17:20:17.018Z INFO cfgfile/reload.go:171 Config reloader started
2020-02-20T17:20:17.021Z WARN [cfgwarn] syslog/input.go:111 EXPERIMENTAL: Syslog input type is used
2020-02-20T17:20:17.025Z INFO elasticsearch/client.go:171 Elasticsearch url: http://localhost:9200
2020-02-20T17:20:17.066Z INFO elasticsearch/client.go:753 Attempting to connect to Elasticsearch version 7.5.2
2020-02-20T17:20:17.223Z INFO input/input.go:114 Starting input of type: syslog; ID: 3066581323769424982
2020-02-20T17:20:17.224Z INFO input/input.go:114 Starting input of type: udp; ID: 16558585575225267322
2020-02-20T17:20:17.224Z INFO input/input.go:114 Starting input of type: udp; ID: 4280410881191362430
2020-02-20T17:20:17.224Z INFO cfgfile/reload.go:226 Loading of config files completed.
2020-02-20T17:20:17.224Z INFO udp/input.go:103 Starting UDP input
2020-02-20T17:20:17.224Z INFO [udp] udp/server.go:81 Started listening for UDP connection {"address": "localhost:9003"}
2020-02-20T17:20:17.224Z INFO [syslog] syslog/input.go:155 Starting Syslog input {"protocol": "udp"}
2020-02-20T17:20:17.224Z INFO [udp] udp/server.go:81 Started listening for UDP connection {"address": "localhost:9002"}
2020-02-20T17:20:17.224Z INFO udp/input.go:103 Starting UDP input
2020-02-20T17:20:17.224Z INFO [udp] udp/server.go:81 Started listening for UDP connection {"address": "localhost:9001"}
2020-02-20T17:20:20.004Z INFO add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
^C2020-02-20T17:20:42.414Z INFO beater/filebeat.go:443 Stopping filebeat
2020-02-20T17:20:42.414Z INFO crawler/crawler.go:139 Stopping Crawler
2020-02-20T17:20:42.414Z INFO crawler/crawler.go:149 Stopping 0 inputs
2020-02-20T17:20:42.414Z INFO cfgfile/reload.go:229 Dynamic config reloader stopped
2020-02-20T17:20:42.414Z INFO [reload] cfgfile/list.go:118 Stopping 1 runners ...
2020-02-20T17:20:42.414Z INFO input/input.go:149 input ticker stopped
2020-02-20T17:20:42.414Z INFO input/input.go:167 Stopping Input: 3066581323769424982
2020-02-20T17:20:42.414Z INFO [syslog] syslog/input.go:175 Stopping Syslog input
2020-02-20T17:20:42.414Z INFO [udp] udp/server.go:140 Stopping UDP server {"address": "localhost:9002"}
2020-02-20T17:20:42.414Z INFO [udp] udp/server.go:118 Connection has been closed {"address": "localhost:9002"}
2020-02-20T17:20:42.414Z INFO [udp] udp/server.go:144 UDP server stopped {"address": "localhost:9002"}
2020-02-20T17:20:42.414Z INFO input/input.go:149 input ticker stopped
2020-02-20T17:20:42.414Z INFO input/input.go:167 Stopping Input: 16558585575225267322
2020-02-20T17:20:42.414Z INFO udp/input.go:118 Stopping UDP input
2020-02-20T17:20:42.414Z INFO [udp] udp/server.go:140 Stopping UDP server {"address": "localhost:9001"}
2020-02-20T17:20:42.414Z INFO [udp] udp/server.go:118 Connection has been closed {"address": "localhost:9001"}
2020-02-20T17:20:42.414Z INFO [udp] udp/server.go:144 UDP server stopped {"address": "localhost:9001"}
2020-02-20T17:20:42.414Z INFO input/input.go:149 input ticker stopped
2020-02-20T17:20:42.415Z INFO input/input.go:167 Stopping Input: 4280410881191362430
2020-02-20T17:20:42.415Z INFO udp/input.go:118 Stopping UDP input
2020-02-20T17:20:42.415Z INFO [udp] udp/server.go:140 Stopping UDP server {"address": "localhost:9003"}
2020-02-20T17:20:42.415Z INFO [udp] udp/server.go:118 Connection has been closed {"address": "localhost:9003"}
2020-02-20T17:20:42.415Z INFO [udp] udp/server.go:144 UDP server stopped {"address": "localhost:9003"}
2020-02-20T17:20:42.415Z INFO crawler/crawler.go:165 Crawler stopped
2020-02-20T17:20:42.415Z INFO registrar/registrar.go:367 Stopping Registrar
2020-02-20T17:20:42.415Z INFO registrar/registrar.go:293 Ending Registrar
2020-02-20T17:20:42.533Z INFO [monitoring] log/log.go:153 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":80,"time":{"ms":82}},"total":{"ticks":170,"time":{"ms":174},"value":170},"user":{"ticks":90,"time":{"ms":92}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":8},"info":{"ephemeral_id":"02280636-11ca-4ab8-b298-c542ad295a97","uptime":{"ms":25612}},"memstats":{"gc_next":9359568,"memory_alloc":7373384,"memory_total":17281456,"rss":48869376},"runtime":{"goroutines":14}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":4},"load":{"1":0.54,"15":0.39,"5":0.83,"norm":{"1":0.135,"15":0.0975,"5":0.2075}}}}}}
2020-02-20T17:20:42.533Z INFO [monitoring] log/log.go:154 Uptime: 25.613400553s
2020-02-20T17:20:42.533Z INFO [monitoring] log/log.go:131 Stopping metrics logging.
2020-02-20T17:20:42.533Z INFO instance/beat.go:435 filebeat stopped.
root@elk:~#

16

This might be a network issue, do you have a log file in your config to test the ingest?

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html

17

i added the following to filebeat.yaml:

paths:
- /var/log/*.log
- /root/fb_logs/*.log

but still the same no logs appear in kibana

18

Can you test the ingest of the log files first (disable any other modules), this way you'd see if the filebeat sends data to elasticsearch. And it that works, you have issue with the network or the configuration for the Cisco module.

19

i have reinstalled all packages from scratch and still the same issue, i tried with MS windows and it worked fine, i will try to install it under Centos.

Thank you

20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.