Suddenly Filebeat agent is not sending the logs to Elastic

Elastic Stack Beats
1

After a Vulnerability remediation update to a DockerHost server, the filebeat agent stopped pushing the logs. Other beats are pushing the logs and is received by the elastic.

Error:
ERROR [registrar] registrar/registrar.go:205 Error writing registrar state to statestore: failed in store/get operation on store 'filebeat': write /var/lib/filebeat/registry/filebeat/checkpoint.new: cannot allocate memory
ERROR [registrar] registrar/registrar.go:205 Error writing registrar state to statestore: failed in store/get operation on store 'filebeat': write /var/lib/filebeat/registry/filebeat/checkpoint.new: input/output error

Not aware what is checkpoint.new file

Trouble shooting done

  1. pinged the elastic host where logs are getting pushed, its working
    2)checked the filebeat.yml is there any change, no change
  2. checked the space available, sufficient space is left
  3. Tried out the commands,
    sudo filebeat test config
    sudo filebeat test output, and resulting positive answers
2

What was changed? Please provide more context about what was changed.

3

Vulnerability Remediation update for RHEL 7.9 hosted server.

Kernal updated to - kernel-3.10.0-1160.119.1.el7.x86_64
Updated services
strong textglibc-2.17-326.el7_9.3.x86_64
glibc-common-2.17-326.el7_9.3.x86_64
12:dhcp-libs-4.2.5-83.el7_9.2.x86_64
32:bind-license-9.11.4-26.P2.el7_9.16.noarch
32:bind-libs-lite-9.11.4-26.P2.el7_9.16.x86_64
32:bind-libs-9.11.4-26.P2.el7_9.16.x86_64
12:dhcp-common-4.2.5-83.el7_9.2.x86_64
flatpak-1.0.9-13.el7_9.x86_64
flatpak-libs-1.0.9-13.el7_9.x86_64
glibc-headers-2.17-326.el7_9.3.x86_64
32:bind-export-libs-9.11.4-26.P2.el7_9.16.x86_64
12:dhclient-4.2.5-83.el7_9.2.x86_64
glibc-devel-2.17-326.el7_9.3.x86_64
32:bind-utils-9.11.4-26.P2.el7_9.16.x86_64
less-458-10.el7_9.x86_64
iwl6050-firmware-41.28.5.1-83.el7_9.noarch
iwl4965-firmware-228.61.2.24-83.el7_9.noarch
iwl7260-firmware-25.30.13.0-83.el7_9.noarch
iwl2000-firmware-18.168.6.1-83.el7_9.noarch
iwl100-firmware-39.31.5.1-83.el7_9.noarch
1:iwl1000-firmware-39.31.5.1-83.el7_9.noarch
iwl5150-firmware-8.24.2.2-83.el7_9.noarch
iwl105-firmware-18.168.6.1-83.el7_9.noarch
iwl6000-firmware-9.221.4.1-83.el7_9.noarch
iwl135-firmware-18.168.6.1-83.el7_9.noarch
iwl3160-firmware-25.30.13.0-83.el7_9.noarch
iwl6000g2a-firmware-18.168.6.1-83.el7_9.noarch
iwl5000-firmware-8.83.5.1_1-83.el7_9.noarch
iwl2030-firmware-18.168.6.1-83.el7_9.noarch
iwl6000g2b-firmware-18.168.6.1-83.el7_9.noarch
iwl3945-firmware-15.32.2.9-83.el7_9.noarch
linux-firmware-20200421-83.git78c0348.el7_9.noarch
glibc-2.17-326.el7_9.3.i686

4

Other than these updates nothing else was change.
can anybody help me troubleshooting

  1. Which can be done that can confirm issue is with Docker host or Elasticsearch host
  2. Is the filebeat agent sending the logs to the Elasticsearch host
  3. is the Elasticsearch is receiving it.
5

Is filebeat running on a docker container or directly on the host? Also, what is the version you are using?

Try to completely stop the filebeat service, start it again, and get fresh logs from this start, share the logs here.

6

@madhavsankarg have you tried
filebeat -e -d "*" -c {location of the configuration file}
this enters the debug mode. it does not connect to the the logstash or the elastic but it lets you see what is happening before sending logs out.
if you are not seeing any errors and logs are been processed fine try telnet to the output address in the config file with the port.

something to have in mind if you do the -e -d it will create a "data" folder in the location you are executing the command. so clean after ur self. you can also use --path.data to redirect the output of the command. or you can point it to exist data folder that will continue showing based on the progress of the existing filebeat. tho that may be desirable for troubleshooting purpose.

7

Hi @leandrojmp and @OneWhoKnowsNothing ,

The issue was resolved
Resolved by restarting the server
there was some issue with network connection establishing ,it was confirmed when we ran
sudo netstat -taupn | grep filebeat
When tried for metricbeat the connection is established and for filebeat connection was not. So logs where not sent to elastic cluster.