# Losing log data when forwarding from filebeat nodes **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [December 16, 2016, 3:48pm UTC](https://discuss.elastic.co/t/losing-log-data-when-forwarding-from-filebeat-nodes/69304 "2016-12-16T15:48:12Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![himaz.m](https://avatars.discourse-cdn.com/v4/letter/h/9de053/32.png) [@himaz.m](https://discuss.elastic.co/u/himaz.m)\ **Post date:** [December 16, 2016, 3:48pm UTC](https://discuss.elastic.co/t/losing-log-data-when-forwarding-from-filebeat-nodes/69304/1 "2016-12-16T15:48:12Z")
Our production environment is setup to have 6 nodes running. We have setup filebeat on all of the 6 nodes. We have an ELK Stack running on a different node and all the filebeats are forwarding the logs to the logstash on this node. But I can see the logs are coming from all the nodes, but sometimes i see some lines of the logs were lost in Kibana. I noticed at once the last log line of a log has not been forwarded from filebeat or may be forwarded dropped between filebeat and logstash. I didn't see any errors on the logs too. Also I have a local environment setup with 1 node for filebeat and another for ELK Stack. But when I copied the production logs to the local I see all the logs till the last line has been indexed and searchable in Kibana. We use Filebeat 5.0 and ELK Stack 5.0 What could be the issue here? Please let me know if I need to share any configurations or any other. ---
**Author:** ![Troy\_Axthelm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/troy_axthelm/32/18347_2.png) [@Troy\_Axthelm](https://discuss.elastic.co/u/Troy_Axthelm)\ **Post date:** [December 17, 2016, 6:43pm UTC](https://discuss.elastic.co/t/losing-log-data-when-forwarding-from-filebeat-nodes/69304/2 "2016-12-17T18:43:51Z")
This will be easier for us to help if you can post your sanitized filebeat configuration. Mainly we need the prospectors section. ---
**Author:** ![himaz.m](https://avatars.discourse-cdn.com/v4/letter/h/9de053/32.png) [@himaz.m](https://discuss.elastic.co/u/himaz.m)\ **Post date:** [December 19, 2016, 6:26am UTC](https://discuss.elastic.co/t/losing-log-data-when-forwarding-from-filebeat-nodes/69304/3 "2016-12-19T06:26:14Z")
Find the filebeat.yml config file below. I've removed all the commented parts of the file and also masked the IP address of the logstash host. ``` filebeat.prospectors: - input_type: log paths: - /opt/ALLMODULESLOG/*.log document_type: mixlog output.logstash: # The Logstash hosts hosts: ["*.*.*.*:5044"] ``` Also FYI I've simply drafted the server structure below (Just to get an idea). ![](https://us1.discourse-cdn.com/elastic/original/2X/d/d4e9265c3c1f7281d9a0fd53d56d9a7fe30cdd78.png) ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [December 19, 2016, 1:46pm UTC](https://discuss.elastic.co/t/losing-log-data-when-forwarding-from-filebeat-nodes/69304/4 "2016-12-19T13:46:47Z")
Have you checked filebeat logs for un-encodable events? filebeat-\>logstash is based on ACKs. Only after ACK from logstash, the offset counter is serialized, so filebeat can start where it last left-off (between restarts). Have you checked your log file writer? Filebeat requires a newline symbol `\n` to identify the end of the current log-line. Some log writers (php is known for this), do not append the `\n` until a new log line is to be written. ---
**Author:** ![himaz.m](https://avatars.discourse-cdn.com/v4/letter/h/9de053/32.png) [@himaz.m](https://discuss.elastic.co/u/himaz.m)\ **Post date:** [December 20, 2016, 3:20pm UTC](https://discuss.elastic.co/t/losing-log-data-when-forwarding-from-filebeat-nodes/69304/5 "2016-12-20T15:20:53Z")
> [@steffens](#): > > Have you checked filebeat logs for un-encodable events? I didn't notice any errors/issues in filebeat logs.[quote="steffens, post:4, topic:69304"] Have you checked your log file writer? Filebeat requires a newline symbol \n to identify the end of the current log-line. Some log writers (php is known for this), do not append the \n until a new log line is to be written. [/quote] Yes, newline is automatically added after the current log line. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [December 21, 2016, 12:55pm UTC](https://discuss.elastic.co/t/losing-log-data-when-forwarding-from-filebeat-nodes/69304/6 "2016-12-21T12:55:27Z")
Did you try to run filebeat with debug logs (`-d '*'`) enabled to check if/where filebeat stops processing? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 18, 2017, 12:55pm UTC](https://discuss.elastic.co/t/losing-log-data-when-forwarding-from-filebeat-nodes/69304/7 "2017-01-18T12:55:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.