# Need to understand logs loss

**URL:** <https://discuss.elastic.co/t/need-to-understand-logs-loss/164621>\
**Category:** Elasticsearch\
**Created:** [January 17, 2019, 11:55am UTC](https://discuss.elastic.co/t/need-to-understand-logs-loss/164621 "2019-01-17T11:55:13Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![abrejuin](https://avatars.discourse-cdn.com/v4/letter/a/cab0a1/32.png) [@abrejuin](https://discuss.elastic.co/u/abrejuin)\
**Post date:** [January 17, 2019, 11:55am UTC](https://discuss.elastic.co/t/need-to-understand-logs-loss/164621/1 "2019-01-17T11:55:13Z")

</div>

Hello,

I am creating this Topic because I really don't know what to do with a behaviour I just noticed.  
Actually, I have an Elastic stack which is working and one application server is sending me logs with rsyslog.  
Everything works fine but I noticed a loss of logs between 4am and 6am.  
I checked on application server and I can see logs in this time slot.  
Our Elastic stack is monitored by our Centreon and I did not notice any error on Logstash or Elastic.

Did you ever meet this behaviour ?  
Could you help me find the cause of this behaviour ?

Thanks a lot for your help  
Best regards  
Antoine

---

<div class="post-metadata">

**Author:** ![abrejuin](https://avatars.discourse-cdn.com/v4/letter/a/cab0a1/32.png) [@abrejuin](https://discuss.elastic.co/u/abrejuin)\
**Post date:** [January 25, 2019, 3:55pm UTC](https://discuss.elastic.co/t/need-to-understand-logs-loss/164621/2 "2019-01-25T15:55:02Z")

</div>

Hello

I have more information to share :  
I noticed some warn logs in logstash-plain.log during logs loss time slot:

[2019-01-25T13:19:19,998][WARN][logstash.inputs.udp] UDP listener died {:exception=\>#\<SocketError: recvfrom: name or service not known\>, :backtrace=\>["org/jruby/ext/socket/RubyUDPSocket.java:217:in `recvfrom_nonblock'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.0/lib/logstash/inputs/udp.rb:97:in`udp\_listener'", "org/jruby/RubyFixnum.java:275:in `times'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.0/lib/logstash/inputs/udp.rb:95:in`udp\_listener'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.0/lib/logstash/inputs/udp.rb:56:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:425:in`inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:419:in `start\_input'"]}  
[2019-01-25T13:19:27,535][INFO][logstash.inputs.udp] Starting UDP listener {:address=\>"0.0.0.0:5514"}

Also I am collecting some elapsed time and those times are increasing just before logs loss.

Thanks a lot for your help  
Antoine

---

<div class="post-metadata">

**Author:** ![abrejuin](https://avatars.discourse-cdn.com/v4/letter/a/cab0a1/32.png) [@abrejuin](https://discuss.elastic.co/u/abrejuin)\
**Post date:** [February 22, 2019, 9:06am UTC](https://discuss.elastic.co/t/need-to-understand-logs-loss/164621/3 "2019-02-22T09:06:07Z")

</div>

Hello,

I am just sharing some news about this behavior.  
In my Logstash configuration I am using Elapsed filter plugin several times because I want to calculate several elapsed time.  
One of those elapsed time could last more than a day. For this elapsed time I set timeout with 172800.  
Just before I loose some logs I noticed an increasing time for all elapsed time calculate in Logstash.  
I have just deactivated the longest elapsed time and my behavior did not reproduce last night.

Here is my elapsed time configuration:  
if [rcu\_logcode] == "RCU\_INUR\_001" {  
mutate { add\_tag =\> ["assignID"] }  
}  
if [rcu\_logcode] == "RCU\_INUR\_002" {  
mutate { add\_tag =\> ["successIDend"] }  
mutate { add\_tag =\> ["successIDstart"] }  
}  
if [rcu\_logcode] == "RCU\_INUR\_003" {  
mutate { add\_tag =\> ["rejectID"] }  
}  
if [rcu\_logcode] == "RCU\_INTG\_002" or [rcu\_logcode] == "RCU\_INTG\_001" {  
mutate { add\_tag =\> ["intgIDend"] }  
mutate { add\_tag =\> ["intgIDstart"] }  
}  
elapsed {  
start\_tag =\> "assignID"  
end\_tag =\> "successIDstart"  
timeout =\> 1200  
unique\_id\_field =\> "rcu\_logid"  
}  
if ([elapsed\_time]){  
mutate { add\_tag =\> ["timeParseUR"] }  
ruby {  
code =\> "event.set('timeParseUR', event.get('elapsed\_time'))"  
}  
mutate { remove\_field =\> ["elapsed\_time"] }  
}  
elapsed {  
start\_tag =\> "successIDend"  
end\_tag =\> "rejectID"  
timeout =\> 1200  
unique\_id\_field =\> "rcu\_logid"  
}  
if ([elapsed\_time]){  
mutate { add\_tag =\> ["timeParseToReject"] }  
ruby {  
code =\> "event.set('timeParseToReject', event.get('elapsed\_time'))"  
}  
mutate { remove\_field =\> ["elapsed\_time"] }  
}  
elapsed {  
start\_tag =\> "successIDend"  
end\_tag =\> "intgIDstart"  
timeout =\> 172800  
unique\_id\_field =\> "rcu\_logid"  
}  
if ([elapsed\_time]){  
mutate { add\_tag =\> ["timeParseToINTG"] }  
ruby {  
code =\> "event.set('timeParseToINTG', event.get('elapsed\_time'))"  
}  
mutate { remove\_field =\> ["elapsed\_time"] }  
}

Thanks for your help  
Antoine

---

<div class="post-metadata">

**Author:** ![abrejuin](https://avatars.discourse-cdn.com/v4/letter/a/cab0a1/32.png) [@abrejuin](https://discuss.elastic.co/u/abrejuin)\
**Post date:** [February 28, 2019, 5:16pm UTC](https://discuss.elastic.co/t/need-to-understand-logs-loss/164621/4 "2019-02-28T17:16:13Z")

</div>

Hi

Does anyone can help me on this please ?  
Here is a screen of what is happening right now, top/left graph is showing an increasing in response time, others are showing less and less messages generated.  
If I check directly on the server which is generating logs, I am still seeing the same amount of logs generated.

Thanks for your help.

 ![2019-02-28_KibanaLogsLoss](https://us1.discourse-cdn.com/elastic/original/3X/0/7/07c1b31bd48e27a67258d793092a6ec3a946f886.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 28, 2019, 5:16pm UTC](https://discuss.elastic.co/t/need-to-understand-logs-loss/164621/5 "2019-03-28T17:16:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.