Some times data Loss in elasticsearch

Elastic Stack Elasticsearch
1

sometimes we are having a data loss issue in our elastic environment. when we debug the environment we have found out that all the data that is sent by the filebeat is going through the logstash filter and the out put plugin and also saved in the csv file. but some of them are not published to the elasticsearch and that data are not visible in the kibana dashboard. other data are successfully published to the elasticsearch.

In our environment we have used elasticsearch 7.0, kibana 7.0, logstash 6.3 and filebeat 6.3. we have using cloud elasticearch stack.

Please help

2

Hi @Rajitha,

Could you please tell the reason behind using different version. I think you should use same version of all components.

However, For data loss have you checked the elasticsearch and logstash logs???

Regards,
Harsh Bajaj

3

How do you know that?

5

yes i have checked in the logstash logs and its not giving any error. some of same data types are already publishing to the elasticsearch. data loss happens only in some period of time.
And i dont know how to read elasticsearch logs in the cloud ELK stack dashboard. so i couldn't read the elasticsearch logs.
thanks

6

that data are not available in the kibana dashboard.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.