# A strange behavior we've encountered on our ELK **URL:** \ **Category:** Elasticsearch\ **Created:** [February 10, 2015, 4:29pm UTC](https://discuss.elastic.co/t/a-strange-behavior-weve-encountered-on-our-elk/22069 "2015-02-10T16:29:00Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Yuval\_Khalifa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuval_khalifa/32/9604_2.png) [@Yuval\_Khalifa](https://discuss.elastic.co/u/Yuval_Khalifa)\ **Post date:** [February 10, 2015, 4:29pm UTC](https://discuss.elastic.co/t/a-strange-behavior-weve-encountered-on-our-elk/22069/1 "2015-02-10T16:29:00Z")
Hi, We just installed an ELK server and configured the logstash configuration to match the data that we send to it and until last month it seems to be working fine but since then we see very strange behavior in the Kibana, the event over time histogram shows the event rate at the normal level for about a half an hour, then drops to about 20% of the normal rate and then it continues to drop slowly for about two hours and then stops and after a minute or two it returns to normal for the next half an hour or so and the same behavior repeats. Needless to say that both the /var/log/logstash and /var/log/elasticsearch both show nothing since the service started and by using tcpdump we can verify that events keep coming in at the same rate all time. I attached our logstash configuration, the /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log and a screenshot of our Kibana with no filter applied so that you can see the weird behavior that we see. Is there someone/somewhere that we can turn to to get some help on the subject? Thanks a lot, Yuval. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [February 10, 2015, 4:34pm UTC](https://discuss.elastic.co/t/a-strange-behavior-weve-encountered-on-our-elk/22069/2 "2015-02-10T16:34:14Z")
Are you sure your logs are generated linearly without bursts? -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant [Lucene.NET](http://Lucene.NET) committer and PMC member On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) wrote: > Hi, > > We just installed an ELK server and configured the logstash configuration > to match the data that we send to it and until last month it seems to be > working fine but since then we see very strange behavior in the Kibana, the > event over time histogram shows the event rate at the normal level for > about a half an hour, then drops to about 20% of the normal rate and then > it continues to drop slowly for about two hours and then stops and after a > minute or two it returns to normal for the next half an hour or so and the > same behavior repeats. Needless to say that both the /var/log/logstash and > /var/log/elasticsearch both show nothing since the service started and by > using tcpdump we can verify that events keep coming in at the same rate all > time. I attached our logstash configuration, the > /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log and > a screenshot of our Kibana with no filter applied so that you can see the > weird behavior that we see. > > Is there someone/somewhere that we can turn to to get some help on the > subject? > > Thanks a lot, > Yuval. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yuval\_Khalifa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuval_khalifa/32/9604_2.png) [@Yuval\_Khalifa](https://discuss.elastic.co/u/Yuval_Khalifa)\ **Post date:** [February 10, 2015, 5:07pm UTC](https://discuss.elastic.co/t/a-strange-behavior-weve-encountered-on-our-elk/22069/3 "2015-02-10T17:07:52Z")
Hi. Absolutely (but since that in the past I also worked at the helpdesk dept. I certainly understand why it is important to ask those "Are you sure it's plugged in?" questions...). One of the logs is comming from SecurityOnion which logs (via bro-conn) all the connections so it must be sending data 24x7x365. Thanks for the quick reply, Yuval. On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) wrote: > Are you sure your logs are generated linearly without bursts? > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > [Lucene.NET](http://Lucene.NET) committer and PMC member > > On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa \<[iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > \\> wrote: > > > Hi, > > > > We just installed an ELK server and configured the logstash configuration > > to match the data that we send to it and until last month it seems to be > > working fine but since then we see very strange behavior in the Kibana, the > > event over time histogram shows the event rate at the normal level for > > about a half an hour, then drops to about 20% of the normal rate and then > > it continues to drop slowly for about two hours and then stops and after a > > minute or two it returns to normal for the next half an hour or so and the > > same behavior repeats. Needless to say that both the /var/log/logstash and > > /var/log/elasticsearch both show nothing since the service started and by > > using tcpdump we can verify that events keep coming in at the same rate all > > time. I attached our logstash configuration, the > > /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log and > > a screenshot of our Kibana with no filter applied so that you can see the > > weird behavior that we see. > > > > Is there someone/somewhere that we can turn to to get some help on the > > subject? > > > > Thanks a lot, > > Yuval. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > > \ > > . > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > \ > . > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- בברכה, _יובל כליפא_ CTO תחום מערכות מידע | מגדל סוכנויות. נייד: 052-3336098 משרד: 03-7966565 פקס: 03-7976565 בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [February 10, 2015, 5:37pm UTC](https://discuss.elastic.co/t/a-strange-behavior-weve-encountered-on-our-elk/22069/4 "2015-02-10T17:37:37Z")
The graphic you sent suggests the issue is with logstash - since the @timestamp field is being populated by logstash and is the one that is used to display the date histogram graphics in Kibana. I would start there. I.e. maybe SecurityOnion buffers writes etc, and then to check the logstash shipper process stats. -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant [Lucene.NET](http://Lucene.NET) committer and PMC member On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) wrote: > Hi. > > Absolutely (but since that in the past I also worked at the helpdesk dept. > I certainly understand why it is important to ask those "Are you sure it's > plugged in?" questions...). One of the logs is comming from SecurityOnion > which logs (via bro-conn) all the connections so it must be sending data > 24x7x365. > > Thanks for the quick reply, > Yuval. > > On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > wrote: > > > Are you sure your logs are generated linearly without bursts? > > > > -- > > > > Itamar Syn-Hershko > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > Freelance Developer & Consultant > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) wrote: > > > > > Hi, > > > > > > We just installed an ELK server and configured the logstash > > > configuration to match the data that we send to it and until last month it > > > seems to be working fine but since then we see very strange behavior in the > > > Kibana, the event over time histogram shows the event rate at the normal > > > level for about a half an hour, then drops to about 20% of the normal rate > > > and then it continues to drop slowly for about two hours and then stops and > > > after a minute or two it returns to normal for the next half an hour or so > > > and the same behavior repeats. Needless to say that both the > > > /var/log/logstash and /var/log/elasticsearch both show nothing since the > > > service started and by using tcpdump we can verify that events keep coming > > > in at the same rate all time. I attached our logstash configuration, the > > > /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log and > > > a screenshot of our Kibana with no filter applied so that you can see the > > > weird behavior that we see. > > > > > > Is there someone/somewhere that we can turn to to get some help on the > > > subject? > > > > > > Thanks a lot, > > > Yuval. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > > בברכה, > > _יובל כליפא_ > > CTO > תחום מערכות מידע | מגדל סוכנויות. > נייד: 052-3336098 > משרד: 03-7966565 > פקס: 03-7976565 > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yuval\_Khalifa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuval_khalifa/32/9604_2.png) [@Yuval\_Khalifa](https://discuss.elastic.co/u/Yuval_Khalifa)\ **Post date:** [February 10, 2015, 5:47pm UTC](https://discuss.elastic.co/t/a-strange-behavior-weve-encountered-on-our-elk/22069/5 "2015-02-10T17:47:55Z")
Great! How can I check that? On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) wrote: > The graphic you sent suggests the issue is with logstash - since the > @timestamp field is being populated by logstash and is the one that is used > to display the date histogram graphics in Kibana. I would start there. I.e. > maybe SecurityOnion buffers writes etc, and then to check the logstash > shipper process stats. > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > [Lucene.NET](http://Lucene.NET) committer and PMC member > > On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa \<[iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > \\> wrote: > > > Hi. > > > > Absolutely (but since that in the past I also worked at the helpdesk > > dept. I certainly understand why it is important to ask those "Are you sure > > it's plugged in?" questions...). One of the logs is comming from > > SecurityOnion which logs (via bro-conn) all the connections so it must be > > sending data 24x7x365. > > > > Thanks for the quick reply, > > Yuval. > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko \<[itamar@code972.com](mailto:itamar@code972.com) > > \\> wrote: > > > > > Are you sure your logs are generated linearly without bursts? > > > > > > -- > > > > > > Itamar Syn-Hershko > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > Freelance Developer & Consultant > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > wrote: > > > > > > > Hi, > > > > > > > > We just installed an ELK server and configured the logstash > > > > configuration to match the data that we send to it and until last month it > > > > seems to be working fine but since then we see very strange behavior in the > > > > Kibana, the event over time histogram shows the event rate at the normal > > > > level for about a half an hour, then drops to about 20% of the normal rate > > > > and then it continues to drop slowly for about two hours and then stops and > > > > after a minute or two it returns to normal for the next half an hour or so > > > > and the same behavior repeats. Needless to say that both the > > > > /var/log/logstash and /var/log/elasticsearch both show nothing since the > > > > service started and by using tcpdump we can verify that events keep coming > > > > in at the same rate all time. I attached our logstash configuration, the > > > > /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log and > > > > a screenshot of our Kibana with no filter applied so that you can see the > > > > weird behavior that we see. > > > > > > > > Is there someone/somewhere that we can turn to to get some help on the > > > > subject? > > > > > > > > Thanks a lot, > > > > Yuval. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > > > בברכה, > > > > _יובל כליפא_ > > > > CTO > > תחום מערכות מידע | מגדל סוכנויות. > > נייד: 052-3336098 > > משרד: 03-7966565 > > פקס: 03-7976565 > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > > \ > > . > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > \ > . > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- בברכה, _יובל כליפא_ CTO תחום מערכות מידע | מגדל סוכנויות. נייד: 052-3336098 משרד: 03-7966565 פקס: 03-7976565 בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [February 10, 2015, 5:52pm UTC](https://discuss.elastic.co/t/a-strange-behavior-weve-encountered-on-our-elk/22069/6 "2015-02-10T17:52:52Z")
I'd start by using logstash with input tcp and output fs and see how the file behaves. Same for the fs inputs - see how their files behave. And take it from there. -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant [Lucene.NET](http://Lucene.NET) committer and PMC member On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) wrote: > Great! How can I check that? > > On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > wrote: > > > The graphic you sent suggests the issue is with logstash - since the > > @timestamp field is being populated by logstash and is the one that is used > > to display the date histogram graphics in Kibana. I would start there. I.e. > > maybe SecurityOnion buffers writes etc, and then to check the logstash > > shipper process stats. > > > > -- > > > > Itamar Syn-Hershko > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > Freelance Developer & Consultant > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) wrote: > > > > > Hi. > > > > > > Absolutely (but since that in the past I also worked at the helpdesk > > > dept. I certainly understand why it is important to ask those "Are you sure > > > it's plugged in?" questions...). One of the logs is comming from > > > SecurityOnion which logs (via bro-conn) all the connections so it must be > > > sending data 24x7x365. > > > > > > Thanks for the quick reply, > > > Yuval. > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > > > wrote: > > > > > > > Are you sure your logs are generated linearly without bursts? > > > > > > > > -- > > > > > > > > Itamar Syn-Hershko > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > Freelance Developer & Consultant > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > We just installed an ELK server and configured the logstash > > > > > configuration to match the data that we send to it and until last month it > > > > > seems to be working fine but since then we see very strange behavior in the > > > > > Kibana, the event over time histogram shows the event rate at the normal > > > > > level for about a half an hour, then drops to about 20% of the normal rate > > > > > and then it continues to drop slowly for about two hours and then stops and > > > > > after a minute or two it returns to normal for the next half an hour or so > > > > > and the same behavior repeats. Needless to say that both the > > > > > /var/log/logstash and /var/log/elasticsearch both show nothing since the > > > > > service started and by using tcpdump we can verify that events keep coming > > > > > in at the same rate all time. I attached our logstash configuration, the > > > > > /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log and > > > > > a screenshot of our Kibana with no filter applied so that you can see the > > > > > weird behavior that we see. > > > > > > > > > > Is there someone/somewhere that we can turn to to get some help on the > > > > > subject? > > > > > > > > > > Thanks a lot, > > > > > Yuval. > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > > > > בברכה, > > > > > > _יובל כליפא_ > > > > > > CTO > > > תחום מערכות מידע | מגדל סוכנויות. > > > נייד: 052-3336098 > > > משרד: 03-7966565 > > > פקס: 03-7976565 > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > > בברכה, > > _יובל כליפא_ > > CTO > תחום מערכות מידע | מגדל סוכנויות. > נייד: 052-3336098 > משרד: 03-7966565 > פקס: 03-7976565 > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yuval\_Khalifa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuval_khalifa/32/9604_2.png) [@Yuval\_Khalifa](https://discuss.elastic.co/u/Yuval_Khalifa)\ **Post date:** [February 11, 2015, 4:02am UTC](https://discuss.elastic.co/t/a-strange-behavior-weve-encountered-on-our-elk/22069/7 "2015-02-11T04:02:14Z")
Hi. When you say "see how the file behaves" I'm not quite sure what you mean by that... As I mentioned earlier, it's not that events do not appear at all but instead, the RATE at which they come decreases, so how can I measure the events rate in a file? I thought that there's another way that I can test this: I'll write a quick-and-dirty program that will send an event to the ELK via TCP every 12ms which should result in events rate of about 5,000 events per minute and I'll let you know if the events rate continues to drop or not... Thanks, Yuval. On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) wrote: > I'd start by using logstash with input tcp and output fs and see how the > file behaves. Same for the fs inputs - see how their files behave. And take > it from there. > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > [Lucene.NET](http://Lucene.NET) committer and PMC member > > On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa \<[iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > \\> wrote: > > > Great! How can I check that? > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko \<[itamar@code972.com](mailto:itamar@code972.com) > > \\> wrote: > > > > > The graphic you sent suggests the issue is with logstash - since the > > > @timestamp field is being populated by logstash and is the one that is used > > > to display the date histogram graphics in Kibana. I would start there. I.e. > > > maybe SecurityOnion buffers writes etc, and then to check the logstash > > > shipper process stats. > > > > > > -- > > > > > > Itamar Syn-Hershko > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > Freelance Developer & Consultant > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > wrote: > > > > > > > Hi. > > > > > > > > Absolutely (but since that in the past I also worked at the helpdesk > > > > dept. I certainly understand why it is important to ask those "Are you sure > > > > it's plugged in?" questions...). One of the logs is comming from > > > > SecurityOnion which logs (via bro-conn) all the connections so it must be > > > > sending data 24x7x365. > > > > > > > > Thanks for the quick reply, > > > > Yuval. > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > > > > wrote: > > > > > > > > > Are you sure your logs are generated linearly without bursts? > > > > > > > > > > -- > > > > > > > > > > Itamar Syn-Hershko > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > Freelance Developer & Consultant > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > We just installed an ELK server and configured the logstash > > > > > > configuration to match the data that we send to it and until last month it > > > > > > seems to be working fine but since then we see very strange behavior in the > > > > > > Kibana, the event over time histogram shows the event rate at the normal > > > > > > level for about a half an hour, then drops to about 20% of the normal rate > > > > > > and then it continues to drop slowly for about two hours and then stops and > > > > > > after a minute or two it returns to normal for the next half an hour or so > > > > > > and the same behavior repeats. Needless to say that both the > > > > > > /var/log/logstash and /var/log/elasticsearch both show nothing since the > > > > > > service started and by using tcpdump we can verify that events keep coming > > > > > > in at the same rate all time. I attached our logstash configuration, the > > > > > > /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log and > > > > > > a screenshot of our Kibana with no filter applied so that you can see the > > > > > > weird behavior that we see. > > > > > > > > > > > > Is there someone/somewhere that we can turn to to get some help on > > > > > > the subject? > > > > > > > > > > > > Thanks a lot, > > > > > > Yuval. > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > > > > > בברכה, > > > > > > > > _יובל כליפא_ > > > > > > > > CTO > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > נייד: 052-3336098 > > > > משרד: 03-7966565 > > > > פקס: 03-7976565 > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > > > בברכה, > > > > _יובל כליפא_ > > > > CTO > > תחום מערכות מידע | מגדל סוכנויות. > > נייד: 052-3336098 > > משרד: 03-7966565 > > פקס: 03-7976565 > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > > \ > > . > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > \ > . > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- בברכה, _יובל כליפא_ CTO תחום מערכות מידע | מגדל סוכנויות. נייד: 052-3336098 משרד: 03-7966565 פקס: 03-7976565 בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9JB3FHGHqW%2BuPDzVmuW7vOeRNPb8Bgz%3De4aY%3DXZNDtwg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9JB3FHGHqW%2BuPDzVmuW7vOeRNPb8Bgz%3De4aY%3DXZNDtwg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yuval\_Khalifa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuval_khalifa/32/9604_2.png) [@Yuval\_Khalifa](https://discuss.elastic.co/u/Yuval_Khalifa)\ **Post date:** [February 12, 2015, 9:28am UTC](https://discuss.elastic.co/t/a-strange-behavior-weve-encountered-on-our-elk/22069/8 "2015-02-12T09:28:00Z")
Hi, I wrote that program and ran it and it did managed to keep a steady rate of about 1,000 events per minute even when the Kibana's total events per minute dropped from 60,000 to 6,000. However, when the Kibana's total events per minute dropped to zero, my program got a "connection refused" exception. I ran netstat -s and found out that every time the Kibana's line hit zero the number of RX-DRP increased. At that point I understood that I forgot to mention that this server has a 10GbE nic. Is it possible that the packets are being dropped because of some bufferis filling up? If so, how can I test it and verify that this is actually the case? If it is, how can I solve it? Thanks, Yuval. On Wednesday, February 11, 2015, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) wrote: > Hi. > > When you say "see how the file behaves" I'm not quite sure what you mean > by that... As I mentioned earlier, it's not that events do not appear at > all but instead, the RATE at which they come decreases, so how can I > measure the events rate in a file? I thought that there's another way that > I can test this: I'll write a quick-and-dirty program that will send an > event to the ELK via TCP every 12ms which should result in events rate of > about 5,000 events per minute and I'll let you know if the events rate > continues to drop or not... > > Thanks, > Yuval. > > On Tuesday, February 10, 2015, Itamar Syn-Hershko \<[itamar@code972.com](mailto:itamar@code972.com) > \\> wrote: > > > I'd start by using logstash with input tcp and output fs and see how the > > file behaves. Same for the fs inputs - see how their files behave. And take > > it from there. > > > > -- > > > > Itamar Syn-Hershko > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > Freelance Developer & Consultant > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) wrote: > > > > > Great! How can I check that? > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > > > wrote: > > > > > > > The graphic you sent suggests the issue is with logstash - since the > > > > @timestamp field is being populated by logstash and is the one that is used > > > > to display the date histogram graphics in Kibana. I would start there. I.e. > > > > maybe SecurityOnion buffers writes etc, and then to check the logstash > > > > shipper process stats. > > > > > > > > -- > > > > > > > > Itamar Syn-Hershko > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > Freelance Developer & Consultant > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > wrote: > > > > > > > > > Hi. > > > > > > > > > > Absolutely (but since that in the past I also worked at the helpdesk > > > > > dept. I certainly understand why it is important to ask those "Are you sure > > > > > it's plugged in?" questions...). One of the logs is comming from > > > > > SecurityOnion which logs (via bro-conn) all the connections so it must be > > > > > sending data 24x7x365. > > > > > > > > > > Thanks for the quick reply, > > > > > Yuval. > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > > > > > wrote: > > > > > > > > > > > Are you sure your logs are generated linearly without bursts? > > > > > > > > > > > > -- > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > Freelance Developer & Consultant > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > We just installed an ELK server and configured the logstash > > > > > > > configuration to match the data that we send to it and until last month it > > > > > > > seems to be working fine but since then we see very strange behavior in the > > > > > > > Kibana, the event over time histogram shows the event rate at the normal > > > > > > > level for about a half an hour, then drops to about 20% of the normal rate > > > > > > > and then it continues to drop slowly for about two hours and then stops and > > > > > > > after a minute or two it returns to normal for the next half an hour or so > > > > > > > and the same behavior repeats. Needless to say that both the > > > > > > > /var/log/logstash and /var/log/elasticsearch both show nothing since the > > > > > > > service started and by using tcpdump we can verify that events keep coming > > > > > > > in at the same rate all time. I attached our logstash configuration, the > > > > > > > /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log and > > > > > > > a screenshot of our Kibana with no filter applied so that you can see the > > > > > > > weird behavior that we see. > > > > > > > > > > > > > > Is there someone/somewhere that we can turn to to get some help on > > > > > > > the subject? > > > > > > > > > > > > > > Thanks a lot, > > > > > > > Yuval. > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com) > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to a topic in > > > > > > the Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this topic, visit > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > . > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > > > > > > בברכה, > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > CTO > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > נייד: 052-3336098 > > > > > משרד: 03-7966565 > > > > > פקס: 03-7976565 > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > > > > בברכה, > > > > > > _יובל כליפא_ > > > > > > CTO > > > תחום מערכות מידע | מגדל סוכנויות. > > > נייד: 052-3336098 > > > משרד: 03-7966565 > > > פקס: 03-7976565 > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > > בברכה, > > _יובל כליפא_ > > CTO > תחום מערכות מידע | מגדל סוכנויות. > נייד: 052-3336098 > משרד: 03-7966565 > פקס: 03-7976565 > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ -- בברכה, _יובל כליפא_ CTO תחום מערכות מידע | מגדל סוכנויות. נייד: 052-3336098 משרד: 03-7966565 פקס: 03-7976565 בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [February 12, 2015, 11:48am UTC](https://discuss.elastic.co/t/a-strange-behavior-weve-encountered-on-our-elk/22069/9 "2015-02-12T11:48:53Z")
Yes - can you try using the bulk API? Also, are you running on a cloud server? -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant [Lucene.NET](http://Lucene.NET) committer and PMC member On Thu, Feb 12, 2015 at 11:28 AM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) wrote: > Hi, > > I wrote that program and ran it and it did managed to keep a steady rate > of about 1,000 events per minute even when the Kibana's total events per > minute dropped from 60,000 to 6,000. However, when the > Kibana's total events per minute dropped to zero, my program got a > "connection refused" exception. I ran netstat -s and found out that every > time the Kibana's line hit zero the number of RX-DRP increased. At that > point I understood that I forgot to mention that this server has a 10GbE > nic. Is it possible that the packets are being dropped because of some > bufferis filling up? If so, how can I test it and verify that this is > actually the case? If it is, how can I solve it? > > Thanks, > Yuval. > On Wednesday, February 11, 2015, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) wrote: > > > Hi. > > > > When you say "see how the file behaves" I'm not quite sure what you mean > > by that... As I mentioned earlier, it's not that events do not appear at > > all but instead, the RATE at which they come decreases, so how can I > > measure the events rate in a file? I thought that there's another way that > > I can test this: I'll write a quick-and-dirty program that will send an > > event to the ELK via TCP every 12ms which should result in events rate of > > about 5,000 events per minute and I'll let you know if the events rate > > continues to drop or not... > > > > Thanks, > > Yuval. > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > > wrote: > > > > > I'd start by using logstash with input tcp and output fs and see how the > > > file behaves. Same for the fs inputs - see how their files behave. And take > > > it from there. > > > > > > -- > > > > > > Itamar Syn-Hershko > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > Freelance Developer & Consultant > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > wrote: > > > > > > > Great! How can I check that? > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > > > > wrote: > > > > > > > > > The graphic you sent suggests the issue is with logstash - since the > > > > > @timestamp field is being populated by logstash and is the one that is used > > > > > to display the date histogram graphics in Kibana. I would start there. I.e. > > > > > maybe SecurityOnion buffers writes etc, and then to check the logstash > > > > > shipper process stats. > > > > > > > > > > -- > > > > > > > > > > Itamar Syn-Hershko > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > Freelance Developer & Consultant > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > wrote: > > > > > > > > > > > Hi. > > > > > > > > > > > > Absolutely (but since that in the past I also worked at the helpdesk > > > > > > dept. I certainly understand why it is important to ask those "Are you sure > > > > > > it's plugged in?" questions...). One of the logs is comming from > > > > > > SecurityOnion which logs (via bro-conn) all the connections so it must be > > > > > > sending data 24x7x365. > > > > > > > > > > > > Thanks for the quick reply, > > > > > > Yuval. > > > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > > > > > > wrote: > > > > > > > > > > > > > Are you sure your logs are generated linearly without bursts? > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > > Freelance Developer & Consultant > > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > > > On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > We just installed an ELK server and configured the logstash > > > > > > > > configuration to match the data that we send to it and until last month it > > > > > > > > seems to be working fine but since then we see very strange behavior in the > > > > > > > > Kibana, the event over time histogram shows the event rate at the normal > > > > > > > > level for about a half an hour, then drops to about 20% of the normal rate > > > > > > > > and then it continues to drop slowly for about two hours and then stops and > > > > > > > > after a minute or two it returns to normal for the next half an hour or so > > > > > > > > and the same behavior repeats. Needless to say that both the > > > > > > > > /var/log/logstash and /var/log/elasticsearch both show nothing since the > > > > > > > > service started and by using tcpdump we can verify that events keep coming > > > > > > > > in at the same rate all time. I attached our logstash configuration, the > > > > > > > > /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log and > > > > > > > > a screenshot of our Kibana with no filter applied so that you can see the > > > > > > > > weird behavior that we see. > > > > > > > > > > > > > > > > Is there someone/somewhere that we can turn to to get some help on > > > > > > > > the subject? > > > > > > > > > > > > > > > > Thanks a lot, > > > > > > > > Yuval. > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > To view this discussion on the web visit > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com) > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > To unsubscribe from this topic, visit > > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > > . > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com) > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > > > > > > > בברכה, > > > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > > > CTO > > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > > נייד: 052-3336098 > > > > > > משרד: 03-7966565 > > > > > > פקס: 03-7976565 > > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > > > > > בברכה, > > > > > > > > _יובל כליפא_ > > > > > > > > CTO > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > נייד: 052-3336098 > > > > משרד: 03-7966565 > > > > פקס: 03-7976565 > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > > > בברכה, > > > > _יובל כליפא_ > > > > CTO > > תחום מערכות מידע | מגדל סוכנויות. > > נייד: 052-3336098 > > משרד: 03-7966565 > > פקס: 03-7976565 > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > -- > > בברכה, > > _יובל כליפא_ > > CTO > תחום מערכות מידע | מגדל סוכנויות. > נייד: 052-3336098 > משרד: 03-7966565 > פקס: 03-7976565 > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J\_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yuval\_Khalifa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuval_khalifa/32/9604_2.png) [@Yuval\_Khalifa](https://discuss.elastic.co/u/Yuval_Khalifa)\ **Post date:** [February 12, 2015, 1:28pm UTC](https://discuss.elastic.co/t/a-strange-behavior-weve-encountered-on-our-elk/22069/10 "2015-02-12T13:28:45Z")
Sort of... The ELK is running as a VM on a dedicated ESXi. Are there special configurations I should do in such a case? Thanks, Yuval. On Thursday, February 12, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) wrote: > Yes - can you try using the bulk API? Also, are you running on a cloud > server? > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > [Lucene.NET](http://Lucene.NET) committer and PMC member > > On Thu, Feb 12, 2015 at 11:28 AM, Yuval Khalifa \<[iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > \\> wrote: > > > Hi, > > > > I wrote that program and ran it and it did managed to keep a steady rate > > of about 1,000 events per minute even when the Kibana's total events per > > minute dropped from 60,000 to 6,000. However, when the > > Kibana's total events per minute dropped to zero, my program got a > > "connection refused" exception. I ran netstat -s and found out that every > > time the Kibana's line hit zero the number of RX-DRP increased. At that > > point I understood that I forgot to mention that this server has a 10GbE > > nic. Is it possible that the packets are being dropped because of some > > bufferis filling up? If so, how can I test it and verify that this is > > actually the case? If it is, how can I solve it? > > > > Thanks, > > Yuval. > > On Wednesday, February 11, 2015, Yuval Khalifa \<[iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > \\> wrote: > > > > > Hi. > > > > > > When you say "see how the file behaves" I'm not quite sure what you > > > mean by that... As I mentioned earlier, it's not that events do not appear > > > at all but instead, the RATE at which they come decreases, so how can I > > > measure the events rate in a file? I thought that there's another way that > > > I can test this: I'll write a quick-and-dirty program that will send an > > > event to the ELK via TCP every 12ms which should result in events rate of > > > about 5,000 events per minute and I'll let you know if the events rate > > > continues to drop or not... > > > > > > Thanks, > > > Yuval. > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > > > wrote: > > > > > > > I'd start by using logstash with input tcp and output fs and see how > > > > the file behaves. Same for the fs inputs - see how their files behave. And > > > > take it from there. > > > > > > > > -- > > > > > > > > Itamar Syn-Hershko > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > Freelance Developer & Consultant > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > wrote: > > > > > > > > > Great! How can I check that? > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > > > > > wrote: > > > > > > > > > > > The graphic you sent suggests the issue is with logstash - since the > > > > > > @timestamp field is being populated by logstash and is the one that is used > > > > > > to display the date histogram graphics in Kibana. I would start there. I.e. > > > > > > maybe SecurityOnion buffers writes etc, and then to check the logstash > > > > > > shipper process stats. > > > > > > > > > > > > -- > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > Freelance Developer & Consultant > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > > wrote: > > > > > > > > > > > > > Hi. > > > > > > > > > > > > > > Absolutely (but since that in the past I also worked at the helpdesk > > > > > > > dept. I certainly understand why it is important to ask those "Are you sure > > > > > > > it's plugged in?" questions...). One of the logs is comming from > > > > > > > SecurityOnion which logs (via bro-conn) all the connections so it must be > > > > > > > sending data 24x7x365. > > > > > > > > > > > > > > Thanks for the quick reply, > > > > > > > Yuval. > > > > > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko \< > > > > > > > [itamar@code972.com](mailto:itamar@code972.com)\> wrote: > > > > > > > > > > > > > > > Are you sure your logs are generated linearly without bursts? > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > > > Freelance Developer & Consultant > > > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > > > > > On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > We just installed an ELK server and configured the logstash > > > > > > > > > configuration to match the data that we send to it and until last month it > > > > > > > > > seems to be working fine but since then we see very strange behavior in the > > > > > > > > > Kibana, the event over time histogram shows the event rate at the normal > > > > > > > > > level for about a half an hour, then drops to about 20% of the normal rate > > > > > > > > > and then it continues to drop slowly for about two hours and then stops and > > > > > > > > > after a minute or two it returns to normal for the next half an hour or so > > > > > > > > > and the same behavior repeats. Needless to say that both the > > > > > > > > > /var/log/logstash and /var/log/elasticsearch both show nothing since the > > > > > > > > > service started and by using tcpdump we can verify that events keep coming > > > > > > > > > in at the same rate all time. I attached our logstash configuration, the > > > > > > > > > /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log and > > > > > > > > > a screenshot of our Kibana with no filter applied so that you can see the > > > > > > > > > weird behavior that we see. > > > > > > > > > > > > > > > > > > Is there someone/somewhere that we can turn to to get some help on > > > > > > > > > the subject? > > > > > > > > > > > > > > > > > > Thanks a lot, > > > > > > > > > Yuval. > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > > Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > To view this discussion on the web visit > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com) > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > . > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > > > . > > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > To view this discussion on the web visit > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com) > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > בברכה, > > > > > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > > > > > CTO > > > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > > > נייד: 052-3336098 > > > > > > > משרד: 03-7966565 > > > > > > > פקס: 03-7976565 > > > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com) > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to a topic in > > > > > > the Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this topic, visit > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > . > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > > > > > > בברכה, > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > CTO > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > נייד: 052-3336098 > > > > > משרד: 03-7966565 > > > > > פקס: 03-7976565 > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > > > > בברכה, > > > > > > _יובל כליפא_ > > > > > > CTO > > > תחום מערכות מידע | מגדל סוכנויות. > > > נייד: 052-3336098 > > > משרד: 03-7966565 > > > פקס: 03-7976565 > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > -- > > > > בברכה, > > > > _יובל כליפא_ > > > > CTO > > תחום מערכות מידע | מגדל סוכנויות. > > נייד: 052-3336098 > > משרד: 03-7966565 > > פקס: 03-7976565 > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > > \ > > . > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > \ > . > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J\_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J\_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- בברכה, _יובל כליפא_ CTO תחום מערכות מידע | מגדל סוכנויות. נייד: 052-3336098 משרד: 03-7966565 פקס: 03-7976565 בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm\_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [February 12, 2015, 1:30pm UTC](https://discuss.elastic.co/t/a-strange-behavior-weve-encountered-on-our-elk/22069/11 "2015-02-12T13:30:46Z")
Yes, make sure the disk is local and not low latency shared one (e.g. SAN). Also SSD will probably fix all your pains. -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant [Lucene.NET](http://Lucene.NET) committer and PMC member On Thu, Feb 12, 2015 at 3:28 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) wrote: > Sort of... The ELK is running as a VM on a dedicated ESXi. Are there > special configurations I should do in such a case? > > Thanks, > Yuval. > > On Thursday, February 12, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > wrote: > > > Yes - can you try using the bulk API? Also, are you running on a cloud > > server? > > > > -- > > > > Itamar Syn-Hershko > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > Freelance Developer & Consultant > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > On Thu, Feb 12, 2015 at 11:28 AM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > wrote: > > > > > Hi, > > > > > > I wrote that program and ran it and it did managed to keep a steady rate > > > of about 1,000 events per minute even when the Kibana's total events per > > > minute dropped from 60,000 to 6,000. However, when the > > > Kibana's total events per minute dropped to zero, my program got a > > > "connection refused" exception. I ran netstat -s and found out that every > > > time the Kibana's line hit zero the number of RX-DRP increased. At that > > > point I understood that I forgot to mention that this server has a 10GbE > > > nic. Is it possible that the packets are being dropped because of some > > > bufferis filling up? If so, how can I test it and verify that this is > > > actually the case? If it is, how can I solve it? > > > > > > Thanks, > > > Yuval. > > > On Wednesday, February 11, 2015, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > wrote: > > > > > > > Hi. > > > > > > > > When you say "see how the file behaves" I'm not quite sure what you > > > > mean by that... As I mentioned earlier, it's not that events do not appear > > > > at all but instead, the RATE at which they come decreases, so how can I > > > > measure the events rate in a file? I thought that there's another way that > > > > I can test this: I'll write a quick-and-dirty program that will send an > > > > event to the ELK via TCP every 12ms which should result in events rate of > > > > about 5,000 events per minute and I'll let you know if the events rate > > > > continues to drop or not... > > > > > > > > Thanks, > > > > Yuval. > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > > > > wrote: > > > > > > > > > I'd start by using logstash with input tcp and output fs and see how > > > > > the file behaves. Same for the fs inputs - see how their files behave. And > > > > > take it from there. > > > > > > > > > > -- > > > > > > > > > > Itamar Syn-Hershko > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > Freelance Developer & Consultant > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > wrote: > > > > > > > > > > > Great! How can I check that? > > > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > > > > > > wrote: > > > > > > > > > > > > > The graphic you sent suggests the issue is with logstash - since the > > > > > > > @timestamp field is being populated by logstash and is the one that is used > > > > > > > to display the date histogram graphics in Kibana. I would start there. I.e. > > > > > > > maybe SecurityOnion buffers writes etc, and then to check the logstash > > > > > > > shipper process stats. > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > > Freelance Developer & Consultant > > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > > > On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > > > wrote: > > > > > > > > > > > > > > > Hi. > > > > > > > > > > > > > > > > Absolutely (but since that in the past I also worked at > > > > > > > > the helpdesk dept. I certainly understand why it is important to ask those > > > > > > > > "Are you sure it's plugged in?" questions...). One of the logs is comming > > > > > > > > from SecurityOnion which logs (via bro-conn) all the connections so it must > > > > > > > > be sending data 24x7x365. > > > > > > > > > > > > > > > > Thanks for the quick reply, > > > > > > > > Yuval. > > > > > > > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko \< > > > > > > > > [itamar@code972.com](mailto:itamar@code972.com)\> wrote: > > > > > > > > > > > > > > > > > Are you sure your logs are generated linearly without bursts? > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > > > > Freelance Developer & Consultant > > > > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > > > > > > > On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > We just installed an ELK server and configured the logstash > > > > > > > > > > configuration to match the data that we send to it and until last month it > > > > > > > > > > seems to be working fine but since then we see very strange behavior in the > > > > > > > > > > Kibana, the event over time histogram shows the event rate at the normal > > > > > > > > > > level for about a half an hour, then drops to about 20% of the normal rate > > > > > > > > > > and then it continues to drop slowly for about two hours and then stops and > > > > > > > > > > after a minute or two it returns to normal for the next half an hour or so > > > > > > > > > > and the same behavior repeats. Needless to say that both the > > > > > > > > > > /var/log/logstash and /var/log/elasticsearch both show nothing since the > > > > > > > > > > service started and by using tcpdump we can verify that events keep coming > > > > > > > > > > in at the same rate all time. I attached our logstash configuration, the > > > > > > > > > > /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log and > > > > > > > > > > a screenshot of our Kibana with no filter applied so that you can see the > > > > > > > > > > weird behavior that we see. > > > > > > > > > > > > > > > > > > > > Is there someone/somewhere that we can turn to to get some help > > > > > > > > > > on the subject? > > > > > > > > > > > > > > > > > > > > Thanks a lot, > > > > > > > > > > Yuval. > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > > Google Groups "elasticsearch" group. > > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com) > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > > > > . > > > > > > > > > To unsubscribe from this group and all its topics, send an email > > > > > > > > > to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > To view this discussion on the web visit > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com) > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > . > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > בברכה, > > > > > > > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > > > > > > > CTO > > > > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > > > > נייד: 052-3336098 > > > > > > > > משרד: 03-7966565 > > > > > > > > פקס: 03-7976565 > > > > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > To view this discussion on the web visit > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com) > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > . > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > To unsubscribe from this topic, visit > > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > > . > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com) > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > > > > > > > בברכה, > > > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > > > CTO > > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > > נייד: 052-3336098 > > > > > > משרד: 03-7966565 > > > > > > פקס: 03-7976565 > > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > > > > > בברכה, > > > > > > > > _יובל כליפא_ > > > > > > > > CTO > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > נייד: 052-3336098 > > > > משרד: 03-7966565 > > > > פקס: 03-7976565 > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > -- > > > > > > בברכה, > > > > > > _יובל כליפא_ > > > > > > CTO > > > תחום מערכות מידע | מגדל סוכנויות. > > > נייד: 052-3336098 > > > משרד: 03-7966565 > > > פקס: 03-7976565 > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J\_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J\_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > > בברכה, > > _יובל כליפא_ > > CTO > תחום מערכות מידע | מגדל סוכנויות. > נייד: 052-3336098 > משרד: 03-7966565 > פקס: 03-7976565 > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm\_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm\_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWmDWszZ4gHx8Ubv9%3DsN7%2BwEM1kuu5gtXGCiLumZ%2Buzg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWmDWszZ4gHx8Ubv9%3DsN7%2BwEM1kuu5gtXGCiLumZ%2Buzg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [February 12, 2015, 1:31pm UTC](https://discuss.elastic.co/t/a-strange-behavior-weve-encountered-on-our-elk/22069/12 "2015-02-12T13:31:45Z")
There's a good writeup on the subject by Mike btw, you should read it > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant [Lucene.NET](http://Lucene.NET) committer and PMC member On Thu, Feb 12, 2015 at 3:30 PM, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) wrote: > Yes, make sure the disk is local and not low latency shared one (e.g. > SAN). Also SSD will probably fix all your pains. > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > [Lucene.NET](http://Lucene.NET) committer and PMC member > > On Thu, Feb 12, 2015 at 3:28 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) wrote: > > > Sort of... The ELK is running as a VM on a dedicated ESXi. Are there > > special configurations I should do in such a case? > > > > Thanks, > > Yuval. > > > > On Thursday, February 12, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > > wrote: > > > > > Yes - can you try using the bulk API? Also, are you running on a cloud > > > server? > > > > > > -- > > > > > > Itamar Syn-Hershko > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > Freelance Developer & Consultant > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > On Thu, Feb 12, 2015 at 11:28 AM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > wrote: > > > > > > > Hi, > > > > > > > > I wrote that program and ran it and it did managed to keep a steady > > > > rate of about 1,000 events per minute even when the Kibana's total events > > > > per minute dropped from 60,000 to 6,000. However, when the > > > > Kibana's total events per minute dropped to zero, my program got a > > > > "connection refused" exception. I ran netstat -s and found out that every > > > > time the Kibana's line hit zero the number of RX-DRP increased. At that > > > > point I understood that I forgot to mention that this server has a 10GbE > > > > nic. Is it possible that the packets are being dropped because of some > > > > bufferis filling up? If so, how can I test it and verify that this is > > > > actually the case? If it is, how can I solve it? > > > > > > > > Thanks, > > > > Yuval. > > > > On Wednesday, February 11, 2015, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > wrote: > > > > > > > > > Hi. > > > > > > > > > > When you say "see how the file behaves" I'm not quite sure what you > > > > > mean by that... As I mentioned earlier, it's not that events do not appear > > > > > at all but instead, the RATE at which they come decreases, so how can I > > > > > measure the events rate in a file? I thought that there's another way that > > > > > I can test this: I'll write a quick-and-dirty program that will send an > > > > > event to the ELK via TCP every 12ms which should result in events rate of > > > > > about 5,000 events per minute and I'll let you know if the events rate > > > > > continues to drop or not... > > > > > > > > > > Thanks, > > > > > Yuval. > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > > > > > wrote: > > > > > > > > > > > I'd start by using logstash with input tcp and output fs and see how > > > > > > the file behaves. Same for the fs inputs - see how their files behave. And > > > > > > take it from there. > > > > > > > > > > > > -- > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > Freelance Developer & Consultant > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > > wrote: > > > > > > > > > > > > > Great! How can I check that? > > > > > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko \< > > > > > > > [itamar@code972.com](mailto:itamar@code972.com)\> wrote: > > > > > > > > > > > > > > > The graphic you sent suggests the issue is with logstash - since > > > > > > > > the @timestamp field is being populated by logstash and is the one that is > > > > > > > > used to display the date histogram graphics in Kibana. I would start there. > > > > > > > > I.e. maybe SecurityOnion buffers writes etc, and then to check the logstash > > > > > > > > shipper process stats. > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > > > Freelance Developer & Consultant > > > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > > > > > On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi. > > > > > > > > > > > > > > > > > > Absolutely (but since that in the past I also worked at > > > > > > > > > the helpdesk dept. I certainly understand why it is important to ask those > > > > > > > > > "Are you sure it's plugged in?" questions...). One of the logs is comming > > > > > > > > > from SecurityOnion which logs (via bro-conn) all the connections so it must > > > > > > > > > be sending data 24x7x365. > > > > > > > > > > > > > > > > > > Thanks for the quick reply, > > > > > > > > > Yuval. > > > > > > > > > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko \< > > > > > > > > > [itamar@code972.com](mailto:itamar@code972.com)\> wrote: > > > > > > > > > > > > > > > > > > > Are you sure your logs are generated linearly without bursts? > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > > > > > Freelance Developer & Consultant > > > > > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > > > > > > > > > On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa \<[iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > We just installed an ELK server and configured the logstash > > > > > > > > > > > configuration to match the data that we send to it and until last month it > > > > > > > > > > > seems to be working fine but since then we see very strange behavior in the > > > > > > > > > > > Kibana, the event over time histogram shows the event rate at the normal > > > > > > > > > > > level for about a half an hour, then drops to about 20% of the normal rate > > > > > > > > > > > and then it continues to drop slowly for about two hours and then stops and > > > > > > > > > > > after a minute or two it returns to normal for the next half an hour or so > > > > > > > > > > > and the same behavior repeats. Needless to say that both the > > > > > > > > > > > /var/log/logstash and /var/log/elasticsearch both show nothing since the > > > > > > > > > > > service started and by using tcpdump we can verify that events keep coming > > > > > > > > > > > in at the same rate all time. I attached our logstash configuration, the > > > > > > > > > > > /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log and > > > > > > > > > > > a screenshot of our Kibana with no filter applied so that you can see the > > > > > > > > > > > weird behavior that we see. > > > > > > > > > > > > > > > > > > > > > > Is there someone/somewhere that we can turn to to get some help > > > > > > > > > > > on the subject? > > > > > > > > > > > > > > > > > > > > > > Thanks a lot, > > > > > > > > > > > Yuval. > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > > > Google Groups "elasticsearch" group. > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from > > > > > > > > > > > it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com) > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > > > . > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > You received this message because you are subscribed to a topic > > > > > > > > > > in the Google Groups "elasticsearch" group. > > > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > > > > > . > > > > > > > > > > To unsubscribe from this group and all its topics, send an email > > > > > > > > > > to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com) > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > בברכה, > > > > > > > > > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > > > > > > > > > CTO > > > > > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > > > > > נייד: 052-3336098 > > > > > > > > > משרד: 03-7966565 > > > > > > > > > פקס: 03-7976565 > > > > > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > > Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > To view this discussion on the web visit > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com) > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > . > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > > > . > > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > To view this discussion on the web visit > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com) > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > בברכה, > > > > > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > > > > > CTO > > > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > > > נייד: 052-3336098 > > > > > > > משרד: 03-7966565 > > > > > > > פקס: 03-7976565 > > > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com) > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to a topic in > > > > > > the Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this topic, visit > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > . > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > > > > > > בברכה, > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > CTO > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > נייד: 052-3336098 > > > > > משרד: 03-7966565 > > > > > פקס: 03-7976565 > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > -- > > > > > > > > בברכה, > > > > > > > > _יובל כליפא_ > > > > > > > > CTO > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > נייד: 052-3336098 > > > > משרד: 03-7966565 > > > > פקס: 03-7976565 > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J\_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J\_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > > > בברכה, > > > > _יובל כליפא_ > > > > CTO > > תחום מערכות מידע | מגדל סוכנויות. > > נייד: 052-3336098 > > משרד: 03-7966565 > > פקס: 03-7976565 > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm\_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm\_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zsyx-YV3huejCpygh%3DmcW%3D-bVgc-%3DSEXOVLz%3DkjMaF%2BKA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zsyx-YV3huejCpygh%3DmcW%3D-bVgc-%3DSEXOVLz%3DkjMaF%2BKA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yuval\_Khalifa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuval_khalifa/32/9604_2.png) [@Yuval\_Khalifa](https://discuss.elastic.co/u/Yuval_Khalifa)\ **Post date:** [February 12, 2015, 4:33pm UTC](https://discuss.elastic.co/t/a-strange-behavior-weve-encountered-on-our-elk/22069/13 "2015-02-12T16:33:11Z")
Well SSD would also fix all the pains for my bank too... (-; Are you sure it's caused by disk latency and not some sort of mis-tuned TCP driver? I've read some blogs that recommeded to increase some of the buffers at the sysctl.conf. Do you think so too? On Thursday, February 12, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) wrote: > Yes, make sure the disk is local and not low latency shared one (e.g. > SAN). Also SSD will probably fix all your pains. > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > [Lucene.NET](http://Lucene.NET) committer and PMC member > > On Thu, Feb 12, 2015 at 3:28 PM, Yuval Khalifa \<[iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > \\> wrote: > > > Sort of... The ELK is running as a VM on a dedicated ESXi. Are there > > special configurations I should do in such a case? > > > > Thanks, > > Yuval. > > > > On Thursday, February 12, 2015, Itamar Syn-Hershko \<[itamar@code972.com](mailto:itamar@code972.com) > > \\> wrote: > > > > > Yes - can you try using the bulk API? Also, are you running on a cloud > > > server? > > > > > > -- > > > > > > Itamar Syn-Hershko > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > Freelance Developer & Consultant > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > On Thu, Feb 12, 2015 at 11:28 AM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > wrote: > > > > > > > Hi, > > > > > > > > I wrote that program and ran it and it did managed to keep a steady > > > > rate of about 1,000 events per minute even when the Kibana's total events > > > > per minute dropped from 60,000 to 6,000. However, when the > > > > Kibana's total events per minute dropped to zero, my program got a > > > > "connection refused" exception. I ran netstat -s and found out that every > > > > time the Kibana's line hit zero the number of RX-DRP increased. At that > > > > point I understood that I forgot to mention that this server has a 10GbE > > > > nic. Is it possible that the packets are being dropped because of some > > > > bufferis filling up? If so, how can I test it and verify that this is > > > > actually the case? If it is, how can I solve it? > > > > > > > > Thanks, > > > > Yuval. > > > > On Wednesday, February 11, 2015, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > wrote: > > > > > > > > > Hi. > > > > > > > > > > When you say "see how the file behaves" I'm not quite sure what you > > > > > mean by that... As I mentioned earlier, it's not that events do not appear > > > > > at all but instead, the RATE at which they come decreases, so how can I > > > > > measure the events rate in a file? I thought that there's another way that > > > > > I can test this: I'll write a quick-and-dirty program that will send an > > > > > event to the ELK via TCP every 12ms which should result in events rate of > > > > > about 5,000 events per minute and I'll let you know if the events rate > > > > > continues to drop or not... > > > > > > > > > > Thanks, > > > > > Yuval. > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > > > > > wrote: > > > > > > > > > > > I'd start by using logstash with input tcp and output fs and see how > > > > > > the file behaves. Same for the fs inputs - see how their files behave. And > > > > > > take it from there. > > > > > > > > > > > > -- > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > Freelance Developer & Consultant > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > > wrote: > > > > > > > > > > > > > Great! How can I check that? > > > > > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko \< > > > > > > > [itamar@code972.com](mailto:itamar@code972.com)\> wrote: > > > > > > > > > > > > > > > The graphic you sent suggests the issue is with logstash - since > > > > > > > > the @timestamp field is being populated by logstash and is the one that is > > > > > > > > used to display the date histogram graphics in Kibana. I would start there. > > > > > > > > I.e. maybe SecurityOnion buffers writes etc, and then to check the logstash > > > > > > > > shipper process stats. > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > > > Freelance Developer & Consultant > > > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > > > > > On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi. > > > > > > > > > > > > > > > > > > Absolutely (but since that in the past I also worked at > > > > > > > > > the helpdesk dept. I certainly understand why it is important to ask those > > > > > > > > > "Are you sure it's plugged in?" questions...). One of the logs is comming > > > > > > > > > from SecurityOnion which logs (via bro-conn) all the connections so it must > > > > > > > > > be sending data 24x7x365. > > > > > > > > > > > > > > > > > > Thanks for the quick reply, > > > > > > > > > Yuval. > > > > > > > > > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko \< > > > > > > > > > [itamar@code972.com](mailto:itamar@code972.com)\> wrote: > > > > > > > > > > > > > > > > > > > Are you sure your logs are generated linearly without bursts? > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > > > > > Freelance Developer & Consultant > > > > > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > > > > > > > > > On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa \<[iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > We just installed an ELK server and configured the logstash > > > > > > > > > > > configuration to match the data that we send to it and until last month it > > > > > > > > > > > seems to be working fine but since then we see very strange behavior in the > > > > > > > > > > > Kibana, the event over time histogram shows the event rate at the normal > > > > > > > > > > > level for about a half an hour, then drops to about 20% of the normal rate > > > > > > > > > > > and then it continues to drop slowly for about two hours and then stops and > > > > > > > > > > > after a minute or two it returns to normal for the next half an hour or so > > > > > > > > > > > and the same behavior repeats. Needless to say that both the > > > > > > > > > > > /var/log/logstash and /var/log/elasticsearch both show nothing since the > > > > > > > > > > > service started and by using tcpdump we can verify that events keep coming > > > > > > > > > > > in at the same rate all time. I attached our logstash configuration, the > > > > > > > > > > > /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log and > > > > > > > > > > > a screenshot of our Kibana with no filter applied so that you can see the > > > > > > > > > > > weird behavior that we see. > > > > > > > > > > > > > > > > > > > > > > Is there someone/somewhere that we can turn to to get some help > > > > > > > > > > > on the subject? > > > > > > > > > > > > > > > > > > > > > > Thanks a lot, > > > > > > > > > > > Yuval. > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > > > Google Groups "elasticsearch" group. > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from > > > > > > > > > > > it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com) > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > > > . > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > You received this message because you are subscribed to a topic > > > > > > > > > > in the Google Groups "elasticsearch" group. > > > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > > > > > . > > > > > > > > > > To unsubscribe from this group and all its topics, send an email > > > > > > > > > > to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com) > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > בברכה, > > > > > > > > > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > > > > > > > > > CTO > > > > > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > > > > > נייד: 052-3336098 > > > > > > > > > משרד: 03-7966565 > > > > > > > > > פקס: 03-7976565 > > > > > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > > Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > To view this discussion on the web visit > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com) > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > . > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > > > . > > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > To view this discussion on the web visit > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com) > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > בברכה, > > > > > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > > > > > CTO > > > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > > > נייד: 052-3336098 > > > > > > > משרד: 03-7966565 > > > > > > > פקס: 03-7976565 > > > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com) > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to a topic in > > > > > > the Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this topic, visit > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > . > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > > > > > > בברכה, > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > CTO > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > נייד: 052-3336098 > > > > > משרד: 03-7966565 > > > > > פקס: 03-7976565 > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > -- > > > > > > > > בברכה, > > > > > > > > _יובל כליפא_ > > > > > > > > CTO > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > נייד: 052-3336098 > > > > משרד: 03-7966565 > > > > פקס: 03-7976565 > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J\_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J\_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > > > בברכה, > > > > _יובל כליפא_ > > > > CTO > > תחום מערכות מידע | מגדל סוכנויות. > > נייד: 052-3336098 > > משרד: 03-7966565 > > פקס: 03-7976565 > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > > \ > > . > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm\_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm\_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > \ > . > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWmDWszZ4gHx8Ubv9%3DsN7%2BwEM1kuu5gtXGCiLumZ%2Buzg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWmDWszZ4gHx8Ubv9%3DsN7%2BwEM1kuu5gtXGCiLumZ%2Buzg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWmDWszZ4gHx8Ubv9%3DsN7%2BwEM1kuu5gtXGCiLumZ%2Buzg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWmDWszZ4gHx8Ubv9%3DsN7%2BwEM1kuu5gtXGCiLumZ%2Buzg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- בברכה, _יובל כליפא_ CTO תחום מערכות מידע | מגדל סוכנויות. נייד: 052-3336098 משרד: 03-7966565 פקס: 03-7976565 בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CADtR2A\_v%2Bo\_XcF%2Bcb6ywhsLotSyQ%3DKPZRgRY9P8%2BJ7GSNeJCqw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A_v%2Bo_XcF%2Bcb6ywhsLotSyQ%3DKPZRgRY9P8%2BJ7GSNeJCqw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yuval\_Khalifa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuval_khalifa/32/9604_2.png) [@Yuval\_Khalifa](https://discuss.elastic.co/u/Yuval_Khalifa)\ **Post date:** [February 12, 2015, 9:12pm UTC](https://discuss.elastic.co/t/a-strange-behavior-weve-encountered-on-our-elk/22069/14 "2015-02-12T21:12:48Z")
Well SSD would also fix all the pains for my bank too... (-; Are you sure it's caused by disk latency and not some sort of mis-tuned TCP driver? I've read some blogs that recommeded to increase some of the buffers at the sysctl.conf. Do you think so too? On Thursday, February 12, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) wrote: > Yes, make sure the disk is local and not low latency shared one (e.g. > SAN). Also SSD will probably fix all your pains. > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > [Lucene.NET](http://Lucene.NET) committer and PMC member > > On Thu, Feb 12, 2015 at 3:28 PM, Yuval Khalifa \<[iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > \\> wrote: > > > Sort of... The ELK is running as a VM on a dedicated ESXi. Are there > > special configurations I should do in such a case? > > > > Thanks, > > Yuval. > > > > On Thursday, February 12, 2015, Itamar Syn-Hershko \<[itamar@code972.com](mailto:itamar@code972.com) > > \\> wrote: > > > > > Yes - can you try using the bulk API? Also, are you running on a cloud > > > server? > > > > > > -- > > > > > > Itamar Syn-Hershko > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > Freelance Developer & Consultant > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > On Thu, Feb 12, 2015 at 11:28 AM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > wrote: > > > > > > > Hi, > > > > > > > > I wrote that program and ran it and it did managed to keep a steady > > > > rate of about 1,000 events per minute even when the Kibana's total events > > > > per minute dropped from 60,000 to 6,000. However, when the > > > > Kibana's total events per minute dropped to zero, my program got a > > > > "connection refused" exception. I ran netstat -s and found out that every > > > > time the Kibana's line hit zero the number of RX-DRP increased. At that > > > > point I understood that I forgot to mention that this server has a 10GbE > > > > nic. Is it possible that the packets are being dropped because of some > > > > bufferis filling up? If so, how can I test it and verify that this is > > > > actually the case? If it is, how can I solve it? > > > > > > > > Thanks, > > > > Yuval. > > > > On Wednesday, February 11, 2015, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > wrote: > > > > > > > > > Hi. > > > > > > > > > > When you say "see how the file behaves" I'm not quite sure what you > > > > > mean by that... As I mentioned earlier, it's not that events do not appear > > > > > at all but instead, the RATE at which they come decreases, so how can I > > > > > measure the events rate in a file? I thought that there's another way that > > > > > I can test this: I'll write a quick-and-dirty program that will send an > > > > > event to the ELK via TCP every 12ms which should result in events rate of > > > > > about 5,000 events per minute and I'll let you know if the events rate > > > > > continues to drop or not... > > > > > > > > > > Thanks, > > > > > Yuval. > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > > > > > wrote: > > > > > > > > > > > I'd start by using logstash with input tcp and output fs and see how > > > > > > the file behaves. Same for the fs inputs - see how their files behave. And > > > > > > take it from there. > > > > > > > > > > > > -- > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > Freelance Developer & Consultant > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > > wrote: > > > > > > > > > > > > > Great! How can I check that? > > > > > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko \< > > > > > > > [itamar@code972.com](mailto:itamar@code972.com)\> wrote: > > > > > > > > > > > > > > > The graphic you sent suggests the issue is with logstash - since > > > > > > > > the @timestamp field is being populated by logstash and is the one that is > > > > > > > > used to display the date histogram graphics in Kibana. I would start there. > > > > > > > > I.e. maybe SecurityOnion buffers writes etc, and then to check the logstash > > > > > > > > shipper process stats. > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > > > Freelance Developer & Consultant > > > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > > > > > On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi. > > > > > > > > > > > > > > > > > > Absolutely (but since that in the past I also worked at > > > > > > > > > the helpdesk dept. I certainly understand why it is important to ask those > > > > > > > > > "Are you sure it's plugged in?" questions...). One of the logs is comming > > > > > > > > > from SecurityOnion which logs (via bro-conn) all the connections so it must > > > > > > > > > be sending data 24x7x365. > > > > > > > > > > > > > > > > > > Thanks for the quick reply, > > > > > > > > > Yuval. > > > > > > > > > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko \< > > > > > > > > > [itamar@code972.com](mailto:itamar@code972.com)\> wrote: > > > > > > > > > > > > > > > > > > > Are you sure your logs are generated linearly without bursts? > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > > > > > Freelance Developer & Consultant > > > > > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > > > > > > > > > On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa \<[iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > We just installed an ELK server and configured the logstash > > > > > > > > > > > configuration to match the data that we send to it and until last month it > > > > > > > > > > > seems to be working fine but since then we see very strange behavior in the > > > > > > > > > > > Kibana, the event over time histogram shows the event rate at the normal > > > > > > > > > > > level for about a half an hour, then drops to about 20% of the normal rate > > > > > > > > > > > and then it continues to drop slowly for about two hours and then stops and > > > > > > > > > > > after a minute or two it returns to normal for the next half an hour or so > > > > > > > > > > > and the same behavior repeats. Needless to say that both the > > > > > > > > > > > /var/log/logstash and /var/log/elasticsearch both show nothing since the > > > > > > > > > > > service started and by using tcpdump we can verify that events keep coming > > > > > > > > > > > in at the same rate all time. I attached our logstash configuration, the > > > > > > > > > > > /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log and > > > > > > > > > > > a screenshot of our Kibana with no filter applied so that you can see the > > > > > > > > > > > weird behavior that we see. > > > > > > > > > > > > > > > > > > > > > > Is there someone/somewhere that we can turn to to get some help > > > > > > > > > > > on the subject? > > > > > > > > > > > > > > > > > > > > > > Thanks a lot, > > > > > > > > > > > Yuval. > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > > > Google Groups "elasticsearch" group. > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from > > > > > > > > > > > it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com) > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > > > . > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > You received this message because you are subscribed to a topic > > > > > > > > > > in the Google Groups "elasticsearch" group. > > > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > > > > > . > > > > > > > > > > To unsubscribe from this group and all its topics, send an email > > > > > > > > > > to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com) > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > בברכה, > > > > > > > > > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > > > > > > > > > CTO > > > > > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > > > > > נייד: 052-3336098 > > > > > > > > > משרד: 03-7966565 > > > > > > > > > פקס: 03-7976565 > > > > > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > > Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > To view this discussion on the web visit > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com) > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > . > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > > > . > > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > To view this discussion on the web visit > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com) > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > בברכה, > > > > > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > > > > > CTO > > > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > > > נייד: 052-3336098 > > > > > > > משרד: 03-7966565 > > > > > > > פקס: 03-7976565 > > > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com) > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to a topic in > > > > > > the Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this topic, visit > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > . > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > > > > > > בברכה, > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > CTO > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > נייד: 052-3336098 > > > > > משרד: 03-7966565 > > > > > פקס: 03-7976565 > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > -- > > > > > > > > בברכה, > > > > > > > > _יובל כליפא_ > > > > > > > > CTO > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > נייד: 052-3336098 > > > > משרד: 03-7966565 > > > > פקס: 03-7976565 > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J\_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J\_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > > > בברכה, > > > > _יובל כליפא_ > > > > CTO > > תחום מערכות מידע | מגדל סוכנויות. > > נייד: 052-3336098 > > משרד: 03-7966565 > > פקס: 03-7976565 > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > > \ > > . > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm\_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm\_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > \ > . > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWmDWszZ4gHx8Ubv9%3DsN7%2BwEM1kuu5gtXGCiLumZ%2Buzg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWmDWszZ4gHx8Ubv9%3DsN7%2BwEM1kuu5gtXGCiLumZ%2Buzg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWmDWszZ4gHx8Ubv9%3DsN7%2BwEM1kuu5gtXGCiLumZ%2Buzg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWmDWszZ4gHx8Ubv9%3DsN7%2BwEM1kuu5gtXGCiLumZ%2Buzg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- בברכה, _יובל כליפא_ CTO תחום מערכות מידע | מגדל סוכנויות. נייד: 052-3336098 משרד: 03-7966565 פקס: 03-7976565 בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9yZHpf3eQ4v0OmGa7aAn2hrQ69OkzZmB9bYFO11qRCbA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9yZHpf3eQ4v0OmGa7aAn2hrQ69OkzZmB9bYFO11qRCbA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yuval\_Khalifa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuval_khalifa/32/9604_2.png) [@Yuval\_Khalifa](https://discuss.elastic.co/u/Yuval_Khalifa)\ **Post date:** [February 15, 2015, 4:37am UTC](https://discuss.elastic.co/t/a-strange-behavior-weve-encountered-on-our-elk/22069/15 "2015-02-15T04:37:27Z")
Hi, I just wanted to let you all know that I think that I solved it... I found out that one of the programs that we built that sent logs to the ELK created new Tcp connection for each event which exausted the Tcp buffers on the server (just like a DoS attack). When I modified that program to re-use the same connection things started to return to norm. Thanks for all your help, Yuval. On Thursday, February 12, 2015, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) wrote: > Well SSD would also fix all the pains for my bank too... (-; > > Are you sure it's caused by disk latency and not some sort of mis-tuned > TCP driver? I've read some blogs that recommeded to increase some of the > buffers at the sysctl.conf. Do you think so too? > > On Thursday, February 12, 2015, Itamar Syn-Hershko \<[itamar@code972.com](mailto:itamar@code972.com) > \\> wrote: > > > Yes, make sure the disk is local and not low latency shared one (e.g. > > SAN). Also SSD will probably fix all your pains. > > > > -- > > > > Itamar Syn-Hershko > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > Freelance Developer & Consultant > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > On Thu, Feb 12, 2015 at 3:28 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) wrote: > > > > > Sort of... The ELK is running as a VM on a dedicated ESXi. Are there > > > special configurations I should do in such a case? > > > > > > Thanks, > > > Yuval. > > > > > > On Thursday, February 12, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > > > wrote: > > > > > > > Yes - can you try using the bulk API? Also, are you running on a cloud > > > > server? > > > > > > > > -- > > > > > > > > Itamar Syn-Hershko > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > Freelance Developer & Consultant > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > On Thu, Feb 12, 2015 at 11:28 AM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > I wrote that program and ran it and it did managed to keep a steady > > > > > rate of about 1,000 events per minute even when the Kibana's total events > > > > > per minute dropped from 60,000 to 6,000. However, when the > > > > > Kibana's total events per minute dropped to zero, my program got a > > > > > "connection refused" exception. I ran netstat -s and found out that every > > > > > time the Kibana's line hit zero the number of RX-DRP increased. At that > > > > > point I understood that I forgot to mention that this server has a 10GbE > > > > > nic. Is it possible that the packets are being dropped because of some > > > > > bufferis filling up? If so, how can I test it and verify that this is > > > > > actually the case? If it is, how can I solve it? > > > > > > > > > > Thanks, > > > > > Yuval. > > > > > On Wednesday, February 11, 2015, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > wrote: > > > > > > > > > > > Hi. > > > > > > > > > > > > When you say "see how the file behaves" I'm not quite sure what you > > > > > > mean by that... As I mentioned earlier, it's not that events do not appear > > > > > > at all but instead, the RATE at which they come decreases, so how can I > > > > > > measure the events rate in a file? I thought that there's another way that > > > > > > I can test this: I'll write a quick-and-dirty program that will send an > > > > > > event to the ELK via TCP every 12ms which should result in events rate of > > > > > > about 5,000 events per minute and I'll let you know if the events rate > > > > > > continues to drop or not... > > > > > > > > > > > > Thanks, > > > > > > Yuval. > > > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com) > > > > > > wrote: > > > > > > > > > > > > > I'd start by using logstash with input tcp and output fs and see how > > > > > > > the file behaves. Same for the fs inputs - see how their files behave. And > > > > > > > take it from there. > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > > Freelance Developer & Consultant > > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > > > On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > > > wrote: > > > > > > > > > > > > > > > Great! How can I check that? > > > > > > > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko \< > > > > > > > > [itamar@code972.com](mailto:itamar@code972.com)\> wrote: > > > > > > > > > > > > > > > > > The graphic you sent suggests the issue is with logstash - since > > > > > > > > > the @timestamp field is being populated by logstash and is the one that is > > > > > > > > > used to display the date histogram graphics in Kibana. I would start there. > > > > > > > > > I.e. maybe SecurityOnion buffers writes etc, and then to check the logstash > > > > > > > > > shipper process stats. > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > > > > Freelance Developer & Consultant > > > > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > > > > > > > On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com) > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi. > > > > > > > > > > > > > > > > > > > > Absolutely (but since that in the past I also worked at > > > > > > > > > > the helpdesk dept. I certainly understand why it is important to ask those > > > > > > > > > > "Are you sure it's plugged in?" questions...). One of the logs is comming > > > > > > > > > > from SecurityOnion which logs (via bro-conn) all the connections so it must > > > > > > > > > > be sending data 24x7x365. > > > > > > > > > > > > > > > > > > > > Thanks for the quick reply, > > > > > > > > > > Yuval. > > > > > > > > > > > > > > > > > > > > On Tuesday, February 10, 2015, Itamar Syn-Hershko \< > > > > > > > > > > [itamar@code972.com](mailto:itamar@code972.com)\> wrote: > > > > > > > > > > > > > > > > > > > > > Are you sure your logs are generated linearly without bursts? > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > Itamar Syn-Hershko > > > > > > > > > > > [http://code972.com](http://code972.com) | @synhershko > > > > > > > > > > > [https://twitter.com/synhershko](https://twitter.com/synhershko) > > > > > > > > > > > Freelance Developer & Consultant > > > > > > > > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa \< > > > > > > > > > > > [iyuvalk@gmail.com](mailto:iyuvalk@gmail.com)\> wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > We just installed an ELK server and configured the logstash > > > > > > > > > > > > configuration to match the data that we send to it and until last month it > > > > > > > > > > > > seems to be working fine but since then we see very strange behavior in the > > > > > > > > > > > > Kibana, the event over time histogram shows the event rate at the normal > > > > > > > > > > > > level for about a half an hour, then drops to about 20% of the normal rate > > > > > > > > > > > > and then it continues to drop slowly for about two hours and then stops and > > > > > > > > > > > > after a minute or two it returns to normal for the next half an hour or so > > > > > > > > > > > > and the same behavior repeats. Needless to say that both the > > > > > > > > > > > > /var/log/logstash and /var/log/elasticsearch both show nothing since the > > > > > > > > > > > > service started and by using tcpdump we can verify that events keep coming > > > > > > > > > > > > in at the same rate all time. I attached our logstash configuration, the > > > > > > > > > > > > /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log and > > > > > > > > > > > > a screenshot of our Kibana with no filter applied so that you can see the > > > > > > > > > > > > weird behavior that we see. > > > > > > > > > > > > > > > > > > > > > > > > Is there someone/somewhere that we can turn to to get some help > > > > > > > > > > > > on the subject? > > > > > > > > > > > > > > > > > > > > > > > > Thanks a lot, > > > > > > > > > > > > Yuval. > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > > > > Google Groups "elasticsearch" group. > > > > > > > > > > > > To unsubscribe from this group and stop receiving emails from > > > > > > > > > > > > it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > > > > > > > > > > > > . > > > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com) > > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > > > > > > > . > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > You received this message because you are subscribed to a topic > > > > > > > > > > > in the Google Groups "elasticsearch" group. > > > > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > > > > > > . > > > > > > > > > > > To unsubscribe from this group and all its topics, send an email > > > > > > > > > > > to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com) > > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ\_\_QdLnB6NYLhoDVaD9CR1RNkC\_9\_c%2Boaqccqww%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > > > . > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > בברכה, > > > > > > > > > > > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > > > > > > > > > > > CTO > > > > > > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > > > > > > נייד: 052-3336098 > > > > > > > > > > משרד: 03-7966565 > > > > > > > > > > פקס: 03-7976565 > > > > > > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > You received this message because you are subscribed to the > > > > > > > > > > Google Groups "elasticsearch" group. > > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > > To view this discussion on the web visit > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com) > > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > > . > > > > > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this topic, visit > > > > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > > > > . > > > > > > > > > To unsubscribe from this group and all its topics, send an email > > > > > > > > > to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > > To view this discussion on the web visit > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com) > > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB\_aNP3hhaWmuFrL1Po\_OAZw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > > . > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > בברכה, > > > > > > > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > > > > > > > CTO > > > > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > > > > נייד: 052-3336098 > > > > > > > > משרד: 03-7966565 > > > > > > > > פקס: 03-7976565 > > > > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > To view this discussion on the web visit > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com) > > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > > . > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to a topic in > > > > > > > the Google Groups "elasticsearch" group. > > > > > > > To unsubscribe from this topic, visit > > > > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > > > > . > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > To view this discussion on the web visit > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com) > > > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq\_9BsWrbmPTg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > > > . > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > > > -- > > > > > > > > > > > > בברכה, > > > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > > > CTO > > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > > נייד: 052-3336098 > > > > > > משרד: 03-7966565 > > > > > > פקס: 03-7976565 > > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > > > -- > > > > > > > > > > בברכה, > > > > > > > > > > _יובל כליפא_ > > > > > > > > > > CTO > > > > > תחום מערכות מידע | מגדל סוכנויות. > > > > > נייד: 052-3336098 > > > > > משרד: 03-7966565 > > > > > פקס: 03-7976565 > > > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A98h2wBzBxaub9A6Af-W7bgCf3-qPQCiquYAY9EiBAOXQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J\_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J\_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtKtXUZ7rnsib4tfqF0J_pd37j5D-7iLQoenovEtpibOg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > > > > בברכה, > > > > > > _יובל כליפא_ > > > > > > CTO > > > תחום מערכות מידע | מגדל סוכנויות. > > > נייד: 052-3336098 > > > משרד: 03-7966565 > > > פקס: 03-7976565 > > > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > > > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > > > > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm\_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm\_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CADtR2A8E6qtLgGVA2ihrvm_mXxTRjDRRP-pjqfM1heEM3EqvPQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWmDWszZ4gHx8Ubv9%3DsN7%2BwEM1kuu5gtXGCiLumZ%2Buzg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWmDWszZ4gHx8Ubv9%3DsN7%2BwEM1kuu5gtXGCiLumZ%2Buzg%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWmDWszZ4gHx8Ubv9%3DsN7%2BwEM1kuu5gtXGCiLumZ%2Buzg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsWmDWszZ4gHx8Ubv9%3DsN7%2BwEM1kuu5gtXGCiLumZ%2Buzg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > > בברכה, > > _יובל כליפא_ > > CTO > תחום מערכות מידע | מגדל סוכנויות. > נייד: 052-3336098 > משרד: 03-7966565 > פקס: 03-7976565 > בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) > [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) > > _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ -- בברכה, _יובל כליפא_ CTO תחום מערכות מידע | מגדל סוכנויות. נייד: 052-3336098 משרד: 03-7966565 פקס: 03-7976565 בלוג: [http://www.artifex.co.il](http://www.artifex.co.il) [https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3A%2F%2Fwww.artifex.co.il%2F](https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f) _[image: תיאור: תיאור: cid:image003.png@01CBB583.C49AE5A0]_ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CADtR2A975QLfQmsAKaXxr%3DVUUe9JetTY5Gp%2B4B28JZmCYrfr0A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADtR2A975QLfQmsAKaXxr%3DVUUe9JetTY5Gp%2B4B28JZmCYrfr0A%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:32am UTC](https://discuss.elastic.co/t/a-strange-behavior-weve-encountered-on-our-elk/22069/16 "2017-07-06T00:32:45Z")