Logs are not parsing properly

My architecture is

*Filebeat A*(remote) > Logstash A (2 pipelines) > Elasticsearch A > Kibana A
*Filebeat B*(remote) > Logstash A (2 pipelines) > Elasticsearch A > Kibana A

Its for logs analysis.

say my logs format is abc_logs-yyyy.mm.dd.log

My Filebeats are pushing logs to Logstash (I can see in data/registry file) but Logstash is NOT creating index for some of the log files.

say, abc_logs-2019.11.02.log is there in my log location and also Filebeat pushed it to Logstash. But I can't see any index created in elasticsearch.

Also, one extra problem.

Even if the indexes are created, NOT all valid logs are getting parsed.
Like if a log file has 100 correct log format (as in grok filter pattern of
logstash.yml file) only 60%-70% data is shown in elasticsearch..
around 40% data is getting dropped .. I don't know what is the exact reason..

If I check the unparsed logs in grok debugger with the specified grok pattern, it is parsing perfectly.

Any solution for this problem?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.