Filebeat cisco modue "Error starting the server address already in use"

hazem_Alhamwi · February 20, 2020, 3:31pm

i am trying to setup log server for network devices using ELK and filebeat with Ubuntu 18, but kibana doesn't display any output. when i run filebeat -e i get the following messages:
2020-02-20T14:53:10.891Z INFO udp/input.go:103 Starting UDP input
2020-02-20T14:53:10.891Z ERROR udp/input.go:106 Error running harvester: listen udp 127.0.0.1:9001: bind: address already in use
2020-02-20T14:53:10.891Z ERROR udp/input.go:106 Error running harvester: listen udp 127.0.0.1:9003: bind: address already in use
2020-02-20T14:53:13.785Z INFO add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2020-02-20T14:53:20.891Z INFO [syslog] syslog/input.go:155 Starting Syslog input {"protocol": "udp"}
2020-02-20T14:53:20.891Z ERROR [syslog] syslog/input.go:158 Error starting the servererrorlisten udp 127.0.0.1:9002: bind: address already in use
2020-02-20T14:53:30.892Z INFO [syslog] syslog/input.go:155 Starting Syslog input {"protocol": "udp"}
2020-02-20T14:53:30.892Z ERROR [syslog] syslog/input.go:158

i have checked that i am receiving logs from ASA:

15:06:23.156798 IP (tos 0x0, ttl 255, id 12006, offset 0, flags [none], proto UDP (17), length 223)
192.168.x.x.syslog > 192.168.x.x.9001: SYSLOG, length: 195
Facility local4 (20), Severity info (6)
Msg: Feb 20 2020 17:27:17 xxx-ASAFW : %ASA-6-302015: Built inbound UDP connection 3505019954 for OUTSIDE:31.13.x.x/29587 (31.13.x.x/29587) to HOSTNAME:x.x.5.9/53 (x.x.5.9/53)\0x0a

the server is listening on the same ports:

root@elk:~# netstat -ntlpu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 779/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1164/sshd
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 1033/node
tcp6 0 0 127.0.0.1:9200 :::* LISTEN 1027/java
tcp6 0 0 127.0.0.1:9300 :::* LISTEN 1027/java
tcp6 0 0 :::22 :::* LISTEN 1164/sshd
udp 0 0 127.0.0.1:9001 0.0.0.0:* 1940/filebeat
udp 0 0 127.0.0.1:9002 0.0.0.0:* 1940/filebeat
udp 0 0 127.0.0.1:9003 0.0.0.0:* 1940/filebeat
udp 0 0 127.0.0.53:53 0.0.0.0:* 779/systemd-resolve

any help?
thanks

daniel_a · February 20, 2020, 4:33pm

You could stop the filebeat service first sudo service filebeat stop, and then run your command filebeat -e. I've seen that before while I was trying to run filebeat -e and the service was already running and using the same port.

Make sure before you run the command filebeat -e, there is no filebeat service or other filebeat process running on that system.

hazem_Alhamwi · February 20, 2020, 4:42pm

thank you but stopping the service caused the server to stop listening on those ports even though when running filebeat -e i get the following:
2020-02-20T16:20:35.328Z INFO [udp] udp/server.go:81 Started listening for UDP connection {"address": "localhost:9001"}
2020-02-20T16:20:35.328Z INFO [udp] udp/server.go:81 Started listening for UDP connection {"address": "localhost:9002"}
2020-02-20T16:20:35.328Z INFO [udp] udp/server.go:81 Started listening for UDP connection {"address": "localhost:9003"}
2020-02-20T16:20:38.237Z INFO add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.

and the same result, nothing on kibana.

daniel_a · February 20, 2020, 4:51pm

After you run the command filebeat -e, do you see the filebeat processes with the 900x ports open? You can check this by running netstat -ntlpu

hazem_Alhamwi · February 20, 2020, 4:53pm

i checked they are not open any more:
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 779/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1164/sshd
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 1033/node
tcp6 0 0 127.0.0.1:9200 :::* LISTEN 1027/java
tcp6 0 0 127.0.0.1:9300 :::* LISTEN 1027/java
tcp6 0 0 :::22 :::* LISTEN 1164/sshd
udp 0 0 127.0.0.53:53 0.0.0.0:* 779/systemd-resolve

daniel_a · February 20, 2020, 4:53pm

Run the filebeat -e command first, run the netstat -ntlpu to see if the process has started, is running or listening.

You can also check Kibana Logs (https://www.elastic.co/guide/en/kibana/current/xpack-logs.html) to see if data comes in, but that should be check only if the filebeat is running.

daniel_a · February 20, 2020, 4:54pm

So maybe the filebeat can't start. What's the complete error? Right after you run the command.

hazem_Alhamwi · February 20, 2020, 5:07pm

kibana says:

There are no log messages to display

daniel_a · February 20, 2020, 5:07pm

You can also check Kibana Logs (https://www.elastic.co/guide/en/kibana/current/xpack-logs.html) to see if data comes in, but that should be check only if the filebeat is running.

hazem_Alhamwi · February 20, 2020, 5:07pm

2020-02-20T16:46:20.969Z ERROR udp/input.go:106 Error running harvester: listen udp 127.0.0.1:9001: bind: address already in use
2020-02-20T16:46:20.969Z ERROR udp/input.go:106 Error running harvester: listen udp 127.0.0.1:9003: bind: address already in use
2020-02-20T16:46:20.969Z ERROR [syslog] syslog/input.go:158 Error starting the servererrorlisten udp 127.0.0.1:9002: bind: address already in use
2020-02-20T16:46:25.661Z INFO [udp] udp/server.go:140 Stopping UDP server {"address": "localhost:9001"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x25c3481]

daniel_a · February 20, 2020, 5:10pm

This is a second process you're trying to start, again the same issue you had at the first place.

daniel_a · February 20, 2020, 5:11pm

Can you make sure to check if you're running a filebeat before starting it? You can't bind the same port to two different processes.

hazem_Alhamwi · February 20, 2020, 5:14pm

i did the following:
root@elk:~# sudo service filebeat stop

root@elk:~# ps -aux |grep filebeat
root 13950 0.0 0.0 13136 1040 pts/1 S+ 16:53 0:00 grep --color=auto filebeat

root@elk:~# netstat -ntlpu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 779/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1164/sshd
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 1033/node
tcp6 0 0 127.0.0.1:9200 :::* LISTEN 1027/java
tcp6 0 0 127.0.0.1:9300 :::* LISTEN 1027/java
tcp6 0 0 :::22 :::* LISTEN 1164/sshd
udp 0 0 127.0.0.53:53 0.0.0.0:* 779/systemd-resolve

daniel_a · February 20, 2020, 5:21pm

So this looks like you're not running it. And what happens when you run the command filebeat -e? Can you paste the complete error message here after you run the command?

hazem_Alhamwi · February 20, 2020, 5:41pm

root@elk:~# filebeat -e
2020-02-20T17:20:16.999Z 2020-02-20T17:20:17.000Z 2020-02-20T17:20:17.004Z 2020-02-20T17:20:17.004Z 2020-02-20T17:20:17.004Z 2020-02-20T17:20:17.004Z 2020-02-20T17:20:17.005Z 2020-02-20T17:20:17.005Z 2020-02-20T17:20:17.005Z 2020-02-20T17:20:17.005Z 2020-02-20T17:20:17.006Z 2020-02-20T17:20:17.006Z 2020-02-20T17:20:17.007Z 2020-02-20T17:20:17.008Z 2020-02-20T17:20:17.008Z 2020-02-20T17:20:17.008Z 2020-02-20T17:20:17.008Z 2020-02-20T17:20:17.014Z 2020-02-20T17:20:17.018Z 2020-02-20T17:20:17.018Z 2020-02-20T17:20:17.021Z 2020-02-20T17:20:17.025Z 2020-02-20T17:20:17.066Z 2020-02-20T17:20:17.223Z 2020-02-20T17:20:17.224Z 2020-02-20T17:20:17.224Z 2020-02-20T17:20:17.224Z 2020-02-20T17:20:17.224Z 2020-02-20T17:20:17.224Z 2020-02-20T17:20:17.224Z 2020-02-20T17:20:17.224Z 2020-02-20T17:20:17.224Z 2020-02-20T17:20:17.224Z 2020-02-20T17:20:20.004Z ^C2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.414Z 2020-02-20T17:20:42.415Z 2020-02-20T17:20:42.415Z 2020-02-20T17:20:42.415Z 2020-02-20T17:20:42.415Z 2020-02-20T17:20:42.415Z 2020-02-20T17:20:42.415Z 2020-02-20T17:20:42.415Z 2020-02-20T17:20:42.415Z 2020-02-20T17:20:42.533Z 2020-02-20T17:20:42.533Z 2020-02-20T17:20:42.533Z 2020-02-20T17:20:42.533Z root@elk:~# INFO instance/beat.go:610 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
INFO instance/beat.go:618 Beat ID: 18448d51-50ed-4281-9db2-d7aa9e816b1a
INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
INFO [beat] instance/beat.go:941 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "18448d51-50ed-4281-9db2-d7aa9e816b1a"}}}
INFO [beat] instance/beat.go:950 Build info {"system_info": {"build": {"commit": "a9c141434cd6b25d7a74a9c770be6b70643dc767", "libbeat": "7.5.2", "time": "2020-01-15T11:13:22.000Z", "version": "7.5.2"}}}
INFO [beat] instance/beat.go:953 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.12.12"}}}
INFO [beat] instance/beat.go:957 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-02-20T17:17:12Z","containerized":false,"name":"elk","ip":["127.0.0.1/8","::1/128","192.168.6.85/24","fe80::250:56ff:fe94:54bf/64"],"kernel_version":"4.15.0-88-generic","mac":["00:50:56:94:54:bf"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"18.04.3 LTS (Bionic Beaver)","major":18,"minor":4,"patch":3,"codename":"bionic"},"timezone":"UTC","timezone_offset_sec":0,"id":"fc0893db1a894c06b7971587bc623544"}}}
INFO [beat] instance/beat.go:986 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/root", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 2023, "ppid": 1764, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-02-20T17:20:16.700Z"}}}
INFO instance/beat.go:297 Setup Beat: filebeat; Version: 7.5.2
INFO [index-management] idxmgmt/std.go:182 Set output.elasticsearch.index to 'filebeat-7.5.2' as ILM is enabled.
INFO elasticsearch/client.go:171 Elasticsearch url: http://localhost:9200
INFO [publisher] pipeline/module.go:97 Beat name: elk
INFO instance/beat.go:429 filebeat start running.
INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
INFO registrar/registrar.go:145 Loading registrar data from /var/lib/filebeat/registry/filebeat/data.json
INFO registrar/registrar.go:152 States Loaded from registrar: 0
INFO crawler/crawler.go:72 Loading Inputs: 1
WARN [cfgwarn] syslog/input.go:111 EXPERIMENTAL: Syslog input type is used
INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 0
INFO cfgfile/reload.go:171 Config reloader started
WARN [cfgwarn] syslog/input.go:111 EXPERIMENTAL: Syslog input type is used
INFO elasticsearch/client.go:171 Elasticsearch url: http://localhost:9200
INFO elasticsearch/client.go:753 Attempting to connect to Elasticsearch version 7.5.2
INFO input/input.go:114 Starting input of type: syslog; ID: 3066581323769424982
INFO input/input.go:114 Starting input of type: udp; ID: 16558585575225267322
INFO input/input.go:114 Starting input of type: udp; ID: 4280410881191362430
INFO cfgfile/reload.go:226 Loading of config files completed.
INFO udp/input.go:103 Starting UDP input
INFO [udp] udp/server.go:81 Started listening for UDP connection {"address": "localhost:9003"}
INFO [syslog] syslog/input.go:155 Starting Syslog input {"protocol": "udp"}
INFO [udp] udp/server.go:81 Started listening for UDP connection {"address": "localhost:9002"}
INFO udp/input.go:103 Starting UDP input
INFO [udp] udp/server.go:81 Started listening for UDP connection {"address": "localhost:9001"}
INFO add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
INFO beater/filebeat.go:443 Stopping filebeat
INFO crawler/crawler.go:139 Stopping Crawler
INFO crawler/crawler.go:149 Stopping 0 inputs
INFO cfgfile/reload.go:229 Dynamic config reloader stopped
INFO [reload] cfgfile/list.go:118 Stopping 1 runners ...
INFO input/input.go:149 input ticker stopped
INFO input/input.go:167 Stopping Input: 3066581323769424982
INFO [syslog] syslog/input.go:175 Stopping Syslog input
INFO [udp] udp/server.go:140 Stopping UDP server {"address": "localhost:9002"}
INFO [udp] udp/server.go:118 Connection has been closed {"address": "localhost:9002"}
INFO [udp] udp/server.go:144 UDP server stopped {"address": "localhost:9002"}
INFO input/input.go:149 input ticker stopped
INFO input/input.go:167 Stopping Input: 16558585575225267322
INFO udp/input.go:118 Stopping UDP input
INFO [udp] udp/server.go:140 Stopping UDP server {"address": "localhost:9001"}
INFO [udp] udp/server.go:118 Connection has been closed {"address": "localhost:9001"}
INFO [udp] udp/server.go:144 UDP server stopped {"address": "localhost:9001"}
INFO input/input.go:149 input ticker stopped
INFO input/input.go:167 Stopping Input: 4280410881191362430
INFO udp/input.go:118 Stopping UDP input
INFO [udp] udp/server.go:140 Stopping UDP server {"address": "localhost:9003"}
INFO [udp] udp/server.go:118 Connection has been closed {"address": "localhost:9003"}
INFO [udp] udp/server.go:144 UDP server stopped {"address": "localhost:9003"}
INFO crawler/crawler.go:165 Crawler stopped
INFO registrar/registrar.go:367 Stopping Registrar
INFO registrar/registrar.go:293 Ending Registrar
INFO [monitoring] log/log.go:153 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":80,"time":{"ms":82}},"total":{"ticks":170,"time":{"ms":174},"value":170},"user":{"ticks":90,"time":{"ms":92}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":8},"info":{"ephemeral_id":"02280636-11ca-4ab8-b298-c542ad295a97","uptime":{"ms":25612}},"memstats":{"gc_next":9359568,"memory_alloc":7373384,"memory_total":17281456,"rss":48869376},"runtime":{"goroutines":14}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":4},"load":{"1":0.54,"15":0.39,"5":0.83,"norm":{"1":0.135,"15":0.0975,"5":0.2075}}}}}}
INFO [monitoring] log/log.go:154 Uptime: 25.613400553s
INFO [monitoring] log/log.go:131 Stopping metrics logging.
INFO instance/beat.go:435 filebeat stopped.

daniel_a · February 20, 2020, 6:31pm

This might be a network issue, do you have a log file in your config to test the ingest?

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html

hazem_Alhamwi · February 24, 2020, 4:05pm

i added the following to filebeat.yaml:

paths:
- /var/log/*.log
- /root/fb_logs/*.log

but still the same no logs appear in kibana

daniel_a · February 24, 2020, 5:10pm

Can you test the ingest of the log files first (disable any other modules), this way you'd see if the filebeat sends data to elasticsearch. And it that works, you have issue with the network or the configuration for the Cisco module.

hazem_Alhamwi · March 1, 2020, 5:38pm

i have reinstalled all packages from scratch and still the same issue, i tried with MS windows and it worked fine, i will try to install it under Centos.

Thank you

system · March 29, 2020, 5:38pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.