Filebeat not starting

Elastic Stack Beats
1

I have installed elk version 7.3.2 on an ubuntu machine running on a VM, but the filebeat is not starting.
Below is the error message.

× filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
     Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sat 2023-12-02 20:56:11 GMT; 2s ago
       Docs: https://www.elastic.co/products/beats/filebeat
    Process: 7168 ExecStart=/usr/share/filebeat/bin/filebeat $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=1/FAILURE)
   Main PID: 7168 (code=exited, status=1/FAILURE)
        CPU: 91ms
Dec 02 20:56:10 ubuntu1 systemd[1]: filebeat.service: Failed with result 'exit-code'.
Dec 02 20:56:11 ubuntu1 systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 5.
Dec 02 20:56:11 ubuntu1 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Dec 02 20:56:11 ubuntu1 systemd[1]: filebeat.service: Start request repeated too quickly.
Dec 02 20:56:11 ubuntu1 systemd[1]: filebeat.service: Failed with result 'exit-code'.
Dec 02 20:56:11 ubuntu1 systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..

The filebeat.yml file is:

filebeat.inputs:

- type: log
  enabled: false
  paths:
    - /var/log/*.log
 path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
#output.elasticsearch:
  # Array of hosts to connect to.
#  hosts: ["localhost:9200"]
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5070"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

I will really appreciate any help to get pass this stage.
Kind Regards

Peter

2

This configuration looks wrong, the path line specifically, you should have just the paths as show in the example in the documentation.

Also, since the systemd is failing to start, you need to check the system log, which will be /var/log/messages or /var/log/syslog depending on your system.

Try to start it again to generate fresh logs and check the system logs.

3

I made the modification but the problem is still there.
below is the output of my syslog

root@ubuntu1:/var/log# tail syslog
Dec  3 06:55:28 ubuntu1 filebeat[21098]: cs     0x33
Dec  3 06:55:28 ubuntu1 filebeat[21098]: fs     0x0
Dec  3 06:55:28 ubuntu1 filebeat[21098]: gs     0x0
Dec  3 06:55:28 ubuntu1 systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 5.
Dec  3 06:55:28 ubuntu1 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Dec  3 06:55:28 ubuntu1 systemd[1]: filebeat.service: Start request repeated too quickly.
Dec  3 06:55:28 ubuntu1 systemd[1]: filebeat.service: Failed with result 'exit-code'.
Dec  3 06:55:28 ubuntu1 systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..
Dec  3 06:55:35 ubuntu1 gnome-shell[2049]: libinput error: event6  - VirtualBox mouse integration: client bug: event processing lagging behind by 57ms, your system is too slow
Dec  3 06:55:35 ubuntu1 gnome-shell[2049]: libinput error: event6  - VirtualBox mouse integration: WARNING: log rate limit exceeded (5 msgs per 60min). Discarding future messages.
4

There are not enough lines on this to troubleshoot, it is basically the same lines you already share.

You need to get more lines, restar the service again, look for any hint about filebeat and share them.

For example, look for lines like this Dec 3 06:55:28 ubuntu1 filebeat[21098]

5

I have rebooted the server. below are the syslog

root@ubuntu1:/var/log# tail syslog
Dec  3 19:06:24 ubuntu1 gnome-shell[1991]: Window manager warning: W5 appears to be one of the offending windows with a timestamp of 784673.  Working around...
Dec  3 19:06:24 ubuntu1 gnome-shell[1991]: Window manager warning: last_user_time (784708) is greater than comparison timestamp (784641).  This most likely represents a buggy client sending inaccurate timestamps in messages such as _NET_ACTIVE_WINDOW.  Trying to work around...
Dec  3 19:06:24 ubuntu1 gnome-shell[1991]: Window manager warning: W5 appears to be one of the offending windows with a timestamp of 784708.  Working around...
Dec  3 19:06:24 ubuntu1 gnome-shell[1991]: Window manager warning: last_user_time (784773) is greater than comparison timestamp (784743).  This most likely represents a buggy client sending inaccurate timestamps in messages such as _NET_ACTIVE_WINDOW.  Trying to work around...
Dec  3 19:06:24 ubuntu1 gnome-shell[1991]: Window manager warning: W5 appears to be one of the offending windows with a timestamp of 784773.  Working around...
Dec  3 19:06:24 ubuntu1 gnome-shell[1991]: Window manager warning: last_user_time (784838) is greater than comparison timestamp (784807).  This most likely represents a buggy client sending inaccurate timestamps in messages such as _NET_ACTIVE_WINDOW.  Trying to work around...
Dec  3 19:06:24 ubuntu1 gnome-shell[1991]: Window manager warning: W5 appears to be one of the offending windows with a timestamp of 784838.  Working around...
Dec  3 19:06:24 ubuntu1 gnome-shell[1991]: Window manager warning: last_user_time (784902) is greater than comparison timestamp (784868).  This most likely represents a buggy client sending inaccurate timestamps in messages such as _NET_ACTIVE_WINDOW.  Trying to work around...
Dec  3 19:06:24 ubuntu1 gnome-shell[1991]: Window manager warning: W5 appears to be one of the offending windows with a timestamp of 784902.  Working around...
Dec  3 19:06:41 ubuntu1 systemd[1824]: Started VTE child process 5679 launched by gnome-terminal-server process 4630.
root@ubuntu1:/var/log# 

Dec  3 19:10:11 ubuntu1 filebeat[5933]: rip    0x7f22d8296a7c
Dec  3 19:10:11 ubuntu1 filebeat[5933]: rflags 0x246
Dec  3 19:10:11 ubuntu1 filebeat[5933]: cs     0x33
Dec  3 19:10:11 ubuntu1 filebeat[5933]: fs     0x0
Dec  3 19:10:11 ubuntu1 filebeat[5933]: gs     0x0
Dec  3 19:10:11 ubuntu1 systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 5.
Dec  3 19:10:11 ubuntu1 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Dec  3 19:10:11 ubuntu1 systemd[1]: filebeat.service: Start request repeated too quickly.
Dec  3 19:10:11 ubuntu1 systemd[1]: filebeat.service: Failed with result 'exit-code'.
Dec  3 19:10:11 ubuntu1 systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..

The logs are the same.
I will reinstall the filebeat and see if there is any change
Regards
Peter

6

As mentioned before, you need to get the full log of this error, get anything mentioned filebeat inthe syslog after you restart it, with incomplete logs is not possible to give any feedback.

Do not use tail, it will get just the last lines, edit the file or use grep filebeat /var/log/syslog for example.

7

Here are more logs form syslog

Dec  4 08:09:40 ubuntu1 systemd[1]: filebeat.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Dec  4 08:09:40 ubuntu1 systemd[1]: filebeat.service: Failed with result 'exit-code'.
Dec  4 08:09:40 ubuntu1 systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 3.
Dec  4 08:09:40 ubuntu1 filebeat[1179]: 2023-12-04T08:09:40.669Z#011INFO#011instance/beat.go:607#011Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
Dec  4 08:09:40 ubuntu1 filebeat[1179]: 2023-12-04T08:09:40.673Z#011INFO#011instance/beat.go:615#011Beat ID: f093aa05-6576-4c40-b3ff-d197c1e1dc0d
Dec  4 08:09:40 ubuntu1 filebeat[1179]: 2023-12-04T08:09:40.681Z#011INFO#011[seccomp]#011seccomp/seccomp.go:124#011Syscall filter successfully installed
Dec  4 08:09:40 ubuntu1 filebeat[1179]: 2023-12-04T08:09:40.681Z#011INFO#011[beat]#011instance/beat.go:903#011Beat info#011{"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "f093aa05-6576-4c40-b3ff-d197c1e1dc0d"}}}
Dec  4 08:09:40 ubuntu1 filebeat[1179]: 2023-12-04T08:09:40.681Z#011INFO#011[beat]#011instance/beat.go:912#011Build info#011{"system_info": {"build": {"commit": "5b046c5a97fe1e312f22d40a1f05365621aad621", "libbeat": "7.3.2", "time": "2019-09-06T13:49:32.000Z", "version": "7.3.2"}}}
Dec  4 08:09:40 ubuntu1 filebeat[1179]: 2023-12-04T08:09:40.681Z#011INFO#011[beat]#011instance/beat.go:915#011Go runtime info#011{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.12.4"}}}
Dec  4 08:09:40 ubuntu1 filebeat[1179]: 2023-12-04T08:09:40.695Z#011INFO#011[beat]#011instance/beat.go:919#011Host info#011{"system_info": {"host": {"architecture":"x86_64","boot_time":"2023-12-04T08:09:29Z","containerized":false,"name":"ubuntu1","ip":["127.0.0.1/8","::1/128","10.0.2.29/24","fe80::5e37:e569:fc3:d8a7/64"],"kernel_version":"6.2.0-37-generic","mac":["08:00:27:c4:54:51"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"22.04.2 LTS (Jammy Jellyfish)","major":22,"minor":4,"patch":2,"codename":"jammy"},"timezone":"GMT","timezone_offset_sec":0,"id":"34279fbfdbf346fdab8579c879d393b5"}}}
Dec  4 08:09:40 ubuntu1 filebeat[1179]: 2023-12-04T08:09:40.698Z#011INFO#011[beat]#011instance/beat.go:948#011Process info#011{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null}, "cwd": "/", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 1179, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2023-12-04T08:09:39.620Z"}}}
Dec  4 08:09:40 ubuntu1 filebeat[1179]: 2023-12-04T08:09:40.699Z#011INFO#011instance/beat.go:292#011Setup Beat: filebeat; Version: 7.3.2
Dec  4 08:09:40 ubuntu1 filebeat[1179]: 2023-12-04T08:09:40.699Z#011INFO#011[publisher]#011pipeline/module.go:97#011Beat name: ubuntu1
Dec  4 08:09:40 ubuntu1 filebeat[1179]: Fatal glibc error: rseq registration failed
Dec  4 08:09:41 ubuntu1 systemd[1]: filebeat.service: Main process exited, code=dumped, status=6/ABRT
Dec  4 08:09:41 ubuntu1 systemd[1]: filebeat.service: Failed with result 'core-dump'.
Dec  4 08:09:41 ubuntu1 systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 4.
Dec  4 08:09:41 ubuntu1 filebeat[1370]: 2023-12-04T08:09:41.986Z#011INFO#011instance/beat.go:607#011Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
Dec  4 08:09:41 ubuntu1 filebeat[1370]: 2023-12-04T08:09:41.987Z#011INFO#011instance/beat.go:615#011Beat ID: f093aa05-6576-4c40-b3ff-d197c1e1dc0d
Dec  4 08:09:41 ubuntu1 filebeat[1370]: 2023-12-04T08:09:41.997Z#011INFO#011[seccomp]#011seccomp/seccomp.go:124#011Syscall filter successfully installed
Dec  4 08:09:41 ubuntu1 filebeat[1370]: 2023-12-04T08:09:41.997Z#011INFO#011[beat]#011instance/beat.go:903#011Beat info#011{"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "f093aa05-6576-4c40-b3ff-d197c1e1dc0d"}}}
Dec  4 08:09:41 ubuntu1 filebeat[1370]: 2023-12-04T08:09:41.998Z#011INFO#011[beat]#011instance/beat.go:912#011Build info#011{"system_info": {"build": {"commit": "5b046c5a97fe1e312f22d40a1f05365621aad621", "libbeat": "7.3.2", "time": "2019-09-06T13:49:32.000Z", "version": "7.3.2"}}}
Dec  4 08:09:41 ubuntu1 filebeat[1370]: 2023-12-04T08:09:41.998Z#011INFO#011[beat]#011instance/beat.go:915#011Go runtime info#011{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.12.4"}}}
Dec  4 08:09:42 ubuntu1 filebeat[1370]: runtime/cgo: pthread_create failed: Operation not permitted
Dec  4 08:09:42 ubuntu1 filebeat[1370]: runtime/cgo: pthread_create failed: Operation not permitted
Dec  4 08:09:42 ubuntu1 filebeat[1370]: SIGABRT: abort
Dec  4 08:09:42 ubuntu1 filebeat[1370]: PC=0x7f7671496a7c m=0 sigcode=18446744073709551610
Dec  4 08:09:42 ubuntu1 filebeat[1370]: goroutine 0 [idle]:
Dec  4 08:09:42 ubuntu1 filebeat[1370]: runtime: unknown pc 0x7f7671496a7c
Dec  4 08:09:42 ubuntu1 filebeat[1370]: stack: frame={sp:0x7fffb374c550, fp:0x0} stack=[0x7fffb2f4db58,0x7fffb374cb80)
Dec  4 08:09:42 ubuntu1 filebeat[1370]: 00007fffb374c450:  0000000000000000  0000000000000000

Thanks for you quick reponse.

Regards

Peter

8

It is probably related to this: Filebeat and GLIBC Errors on Ubuntu 22.04 - #2 by cmacknz

You are running a pretty old and non-supported version of Filebeat.

Try to use the last version for 7.17, which would be 7.17.15.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.