# Missing logs **URL:** \ **Category:** Elasticsearch\ **Tags:** docker\ **Created:** [June 17, 2021, 8:38am UTC](https://discuss.elastic.co/t/missing-logs/276234 "2021-06-17T08:38:51Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Shaked](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@Shaked](https://discuss.elastic.co/u/Shaked)\ **Post date:** [June 17, 2021, 8:38am UTC](https://discuss.elastic.co/t/missing-logs/276234/1 "2021-06-17T08:38:51Z")
Hi everyone, I would be happy to get your help with a critical issue we have. We run AWS ElasticSearch service (v7.10) with Logstash as a Docker container on ECS Fargate. All of our environments are running the service (with the architecture above) and everything works there great, but in 1 specific env (with more traffic than the other environments) we miss some logs (we checked it with Python script that sends logs to Logstash but only 30% of the logs arrive to ElasticSearch). Do you know what could be the reason? where should we start our research? Thanks in advance. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 17, 2021, 9:00am UTC](https://discuss.elastic.co/t/missing-logs/276234/2 "2021-06-17T09:00:59Z")
What is sending data to Logstash? Could it be that the Elasticsearch cluster is unable to keep up and backpressure is causing logs to be lost/dropped somewhere? ---
**Author:** ![Shaked](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@Shaked](https://discuss.elastic.co/u/Shaked)\ **Post date:** [June 17, 2021, 9:13am UTC](https://discuss.elastic.co/t/missing-logs/276234/3 "2021-06-17T09:13:08Z")
Maybe, data=logs of multiple apps. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 17, 2021, 9:18am UTC](https://discuss.elastic.co/t/missing-logs/276234/4 "2021-06-17T09:18:00Z")
What is sending logs? Which input plugins are in use? ---
**Author:** ![Shaked](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@Shaked](https://discuss.elastic.co/u/Shaked)\ **Post date:** [June 17, 2021, 9:24am UTC](https://discuss.elastic.co/t/missing-logs/276234/5 "2021-06-17T09:24:05Z")
Syslog plugin, the apps send it on UDP. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 17, 2021, 9:27am UTC](https://discuss.elastic.co/t/missing-logs/276234/6 "2021-06-17T09:27:49Z")
UDP does not guarantee delivery nor handle back pressure so if Logstash or Elasticsearch can not keep up data will be lost. Only way to guarantee delivery is to move away from UDP. Note that it is not just high load that may cause data loss. Any kind of hiccup could result in data loss, although possibly at a smaller scale that is more difficult to notice. ---
**Author:** ![Shaked](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@Shaked](https://discuss.elastic.co/u/Shaked)\ **Post date:** [June 17, 2021, 9:50am UTC](https://discuss.elastic.co/t/missing-logs/276234/7 "2021-06-17T09:50:54Z")
Ok, thank you! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 15, 2021, 9:51am UTC](https://discuss.elastic.co/t/missing-logs/276234/8 "2021-07-15T09:51:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.