Losing log data when forwarding from filebeat nodes

Elastic Stack Beats
1

Our production environment is setup to have 6 nodes running. We have setup filebeat on all of the 6 nodes. We have an ELK Stack running on a different node and all the filebeats are forwarding the logs to the logstash on this node. But I can see the logs are coming from all the nodes, but sometimes i see some lines of the logs were lost in Kibana.
I noticed at once the last log line of a log has not been forwarded from filebeat or may be forwarded dropped between filebeat and logstash. I didn't see any errors on the logs too.

Also I have a local environment setup with 1 node for filebeat and another for ELK Stack. But when I copied the production logs to the local I see all the logs till the last line has been indexed and searchable in Kibana.

We use Filebeat 5.0 and ELK Stack 5.0

What could be the issue here? Please let me know if I need to share any configurations or any other.

2

This will be easier for us to help if you can post your sanitized filebeat configuration. Mainly we need the prospectors section.

3

Find the filebeat.yml config file below. I've removed all the commented parts of the file and also masked the IP address of the logstash host.

filebeat.prospectors:

- input_type: log

  paths:
    - /opt/ALLMODULESLOG/*.log

  document_type: mixlog


output.logstash:
  # The Logstash hosts
  hosts: ["*.*.*.*:5044"]

Also FYI I've simply drafted the server structure below (Just to get an idea).

4

Have you checked filebeat logs for un-encodable events?

filebeat->logstash is based on ACKs. Only after ACK from logstash, the offset counter is serialized, so filebeat can start where it last left-off (between restarts).

Have you checked your log file writer? Filebeat requires a newline symbol \n to identify the end of the current log-line. Some log writers (php is known for this), do not append the \n until a new log line is to be written.

5

I didn't notice any errors/issues in filebeat logs.[quote="steffens, post:4, topic:69304"]
Have you checked your log file writer? Filebeat requires a newline symbol \n to identify the end of the current log-line. Some log writers (php is known for this), do not append the \n until a new log line is to be written.
[/quote]

Yes, newline is automatically added after the current log line.

6

Did you try to run filebeat with debug logs (-d '*') enabled to check if/where filebeat stops processing?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.