sometimes we are having a data loss issue in our elastic environment. when we debug the environment we have found out that all the data that is sent by the filebeat is going through the logstash filter and the out put plugin and also saved in the csv file. but some of them are not published to the elasticsearch and that data are not visible in the kibana dashboard. other data are successfully published to the elasticsearch.
In our environment we have used elasticsearch 7.0, kibana 7.0, logstash 6.3 and filebeat 6.3. we have using cloud elasticearch stack.