Linux client data not visible on ELK server after

Elastic Stack Beats
1

Hi,

I recently installed the ELK stack on a Linux server running Ubuntu 22.04 using the following as a guide:

After the initial installation and setup, I've also been able to successfully send logs from 4 other linux clients to the server using the filebeats service with the system module enabled.

However one of my linux clients is no longer being detected on the ELK server. The issue occurred after applying a number of regular linux patches and rebooting the system.

I've at least checked the following:

  • TCP port 5044 is accessible from the linux client
  • The configuration files haven't changed as a result of the patching
  • The filebeat service is running on the linux client
  • There were no changes on the ELK server, i.e. all other linux clients are able to communicate with the server

Not sure where to go from here or which log files to examine (server and client end) to troubleshoot the issue.

Thanks,

2

What do the Filebeat logs show?

3

Hi Mark,

I'm pasting a snippet of the filebeat logfile on the affected Linux client

.. snip ..
May 18 14:45:46 gitlab filebeat[154013]: 2023-05-18T14:45:46.000+1000        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s       >
May 18 14:46:15 gitlab filebeat[154013]: 2023-05-18T14:46:15.999+1000        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s       >
May 18 14:46:36 gitlab filebeat[154013]: 2023-05-18T14:46:36.492+1000        INFO        [input.harvester]        log/harvester.go:341        File is inactive. Closing be>
May 18 14:46:46 gitlab filebeat[154013]: 2023-05-18T14:46:46.000+1000        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s       >
May 18 14:47:16 gitlab filebeat[154013]: 2023-05-18T14:47:16.000+1000        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s       >
May 18 14:47:45 gitlab filebeat[154013]: 2023-05-18T14:47:45.999+1000        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s       >
May 18 14:48:15 gitlab filebeat[154013]: 2023-05-18T14:48:15.999+1000        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s       >
May 18 14:48:46 gitlab filebeat[154013]: 2023-05-18T14:48:46.000+1000        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s       >
May 18 14:49:15 gitlab filebeat[154013]: 2023-05-18T14:49:15.999+1000        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s

One noteable entry is the one shown below

May 18 14:46:36 gitlab filebeat[154013]: 2023-05-18T14:46:36.492+1000        INFO        [input.harvester]        log/harvester.go:341        File is inactive. Closing because close_inactive of 5m0s reached.        {"input_id": "cd7c325a-e4c9-49d5-97c4-42044db652a6", "source": "/var/log/auth.log", "state_id": "native::132006-64768", "finished": false, "os_id": "132006-64768", "old_source": "/var/log/auth.log", "old_finished": true, "old_os_id": "132006-64768", "harvester_id": "e48b4ced-d160-43af-ad5a-12cad66a5e78"}

Other checks include:

  • Checking the filebeat service is running
root@gitlab:~# systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
     Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-05-11 17:03:12 AEST; 6 days ago
       Docs: https://www.elastic.co/beats/filebeat
   Main PID: 154013 (filebeat)
      Tasks: 10 (limit: 9402)
     Memory: 43.0M
        CPU: 3min 16.698s
     CGroup: /system.slice/filebeat.service
             └─154013 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --p>

May 18 14:49:15 gitlab filebeat[154013]: 2023-05-18T14:49:15.999+1000        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s
  • Checking that the elk server is accessible from the client on TCP port 5044
root@gitlab:~# telnet sys-elk 5044
.. snip ..
Connected to sys-elk.....
Escape character is '^]'.
^]
4

OK thanks, is there anything written to that log after that timestamp?

Can you share your Filebeat config?

5

Hi Mark,

Yes it looks like the log file is still reporting activity after restarting the filebeat service:

May 18 15:38:46 gitlab filebeat[154013]: 2023-05-18T15:38:46.000+1000        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s       >
May 18 15:39:14 gitlab filebeat[154013]: 2023-05-18T15:39:14.767+1000        INFO        [input.harvester]        log/harvester.go:310        Harvester started for paths:>
May 18 15:39:16 gitlab filebeat[154013]: 2023-05-18T15:39:15.999+1000        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s       >
May 18 15:39:45 gitlab filebeat[154013]: 2023-05-18T15:39:45.999+1000        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s      
.. snip ..
May 18 16:01:46 gitlab filebeat[154013]: 2023-05-18T16:01:45.999+1000        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s       >
May 18 16:02:16 gitlab filebeat[154013]: 2023-05-18T16:02:16.000+1000        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s       >
May 18 16:02:46 gitlab filebeat[154013]: 2023-05-18T16:02:46.001+1000        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s

I'm pasting the filebeat config file (filtering out all the empty lines and comments):

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: false
  paths:
    - /var/log/*.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.logstash:
  hosts: ["sys-elk:5044"]
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Much appreciated,

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.