# Spikes in usage and Logstash connection issues **URL:** \ **Category:** Elasticsearch\ **Created:** [February 13, 2015, 9:56pm UTC](https://discuss.elastic.co/t/spikes-in-usage-and-logstash-connection-issues/22163 "2015-02-13T21:56:17Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![rhea\_ghosh](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rhea\_ghosh](https://discuss.elastic.co/u/rhea_ghosh)\ **Post date:** [February 13, 2015, 9:56pm UTC](https://discuss.elastic.co/t/spikes-in-usage-and-logstash-connection-issues/22163/1 "2015-02-13T21:56:17Z")
Hi, I have a 5 node cluster that I'm using as part of an ELK system. Most of the time it works great but today we saw a spike in writes for one of the nodes and around the same time we saw indexing on that node spike up too -- which makes sense, if you have more writes you'll need to do more indexing. But none of the other servers were all that taxed, we don't have anything else writing to Elasticsearch other than Logstash and normally it does a pretty good job of load balancing. Any idea where I could start looking for clues? I was looking through the logs but there doesn't seem to be much information in there, most of it are just debug errors that say something like this: [logstash-2015.02.13][3] failed to execute bulk item (index) index {[logstash-2015.02.13] but they show up pretty consistently so it doesn't seem like anything to be worried about. Where else could I look to see why only one server is getting all the writes and how do I mitigate it if this is happening because it ended up making Elasticsearch unresponsive to information being sent from Logstash. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2570e228-1d3e-48e0-af8d-0946b7c58197%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2570e228-1d3e-48e0-af8d-0946b7c58197%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 16, 2015, 7:43am UTC](https://discuss.elastic.co/t/spikes-in-usage-and-logstash-connection-issues/22163/2 "2015-02-16T07:43:29Z")
Can you link us to your LS config? On 14 February 2015 at 08:56, rhea ghosh [rhea.ghosh@gmail.com](mailto:rhea.ghosh@gmail.com) wrote: > Hi, I have a 5 node cluster that I'm using as part of an ELK system. Most > of the time it works great but today we saw a spike in writes for one of > the nodes and around the same time we saw indexing on that node spike up > too -- which makes sense, if you have more writes you'll need to do more > indexing. But none of the other servers were all that taxed, we don't have > anything else writing to Elasticsearch other than Logstash and normally it > does a pretty good job of load balancing. Any idea where I could start > looking for clues? I was looking through the logs but there doesn't seem to > be much information in there, most of it are just debug errors that say > something like this: [logstash-2015.02.13][3] failed to execute bulk item > (index) index {[logstash-2015.02.13] but they show up pretty consistently > so it doesn't seem like anything to be worried about. Where else could I > look to see why only one server is getting all the writes and how do I > mitigate it if this is happening because it ended up making Elasticsearch > unresponsive to information being sent from Logstash. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/2570e228-1d3e-48e0-af8d-0946b7c58197%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2570e228-1d3e-48e0-af8d-0946b7c58197%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/2570e228-1d3e-48e0-af8d-0946b7c58197%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2570e228-1d3e-48e0-af8d-0946b7c58197%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X91b\_aWeNfKvohezQ0gBJniFm-dG9a%2BHdcWb1JSz61BRg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X91b_aWeNfKvohezQ0gBJniFm-dG9a%2BHdcWb1JSz61BRg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![rhea\_ghosh](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rhea\_ghosh](https://discuss.elastic.co/u/rhea_ghosh)\ **Post date:** [February 17, 2015, 8:42pm UTC](https://discuss.elastic.co/t/spikes-in-usage-and-logstash-connection-issues/22163/3 "2015-02-17T20:42:06Z")
I'm assuming you meant my elasticsearch config. These are the options i have specified: cluster.name: elasticEnvironmentName node.name: Nodename path.data: /path/to/data bootstrap.mlockall: true gateway.recover\_after\_nodes: 3 cluster.routing.allocation.node\_concurrent\_recoveries: 15 indices.recovery.max\_bytes\_per\_sec: 500mb discovery.zen.minimum\_master\_nodes: 3 discovery.zen.fd.ping\_interval: 15s discovery.zen.fd.ping\_timeout: 60s discovery.zen.fd.ping\_retries: 5 index.search.slowlog.threshold.query.warn: 10s index.search.slowlog.threshold.fetch.warn: 1s index.indexing.slowlog.threshold.index.warn: 10s indices.fielddata.cache.size: 50% indices.breaker.fielddata.limit: 60% cloud.aws.access\_key: AWSAccessKey cloud.aws.secret\_key: AWSSecretKey On Monday, February 16, 2015 at 1:43:59 AM UTC-6, Mark Walkom wrote: > Can you link us to your LS config? > > On 14 February 2015 at 08:56, rhea ghosh \<[rhea....@gmail.com](mailto:rhea....@gmail.com) \ > > > wrote: > > > Hi, I have a 5 node cluster that I'm using as part of an ELK system. Most > > of the time it works great but today we saw a spike in writes for one of > > the nodes and around the same time we saw indexing on that node spike up > > too -- which makes sense, if you have more writes you'll need to do more > > indexing. But none of the other servers were all that taxed, we don't have > > anything else writing to Elasticsearch other than Logstash and normally it > > does a pretty good job of load balancing. Any idea where I could start > > looking for clues? I was looking through the logs but there doesn't seem to > > be much information in there, most of it are just debug errors that say > > something like this: [logstash-2015.02.13][3] failed to execute bulk item > > (index) index {[logstash-2015.02.13] but they show up pretty consistently > > so it doesn't seem like anything to be worried about. Where else could I > > look to see why only one server is getting all the writes and how do I > > mitigate it if this is happening because it ended up making Elasticsearch > > unresponsive to information being sent from Logstash. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/2570e228-1d3e-48e0-af8d-0946b7c58197%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2570e228-1d3e-48e0-af8d-0946b7c58197%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/2570e228-1d3e-48e0-af8d-0946b7c58197%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2570e228-1d3e-48e0-af8d-0946b7c58197%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/46fd8986-6f0b-4573-b4af-07e8a04c1eb0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/46fd8986-6f0b-4573-b4af-07e8a04c1eb0%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 17, 2015, 9:41pm UTC](https://discuss.elastic.co/t/spikes-in-usage-and-logstash-connection-issues/22163/4 "2015-02-17T21:41:25Z")
I did not, I mean LS (Logstash) On 18 February 2015 at 07:42, rhea ghosh [rhea.ghosh@gmail.com](mailto:rhea.ghosh@gmail.com) wrote: > I'm assuming you meant my elasticsearch config. > > These are the options i have specified: > cluster.name: elasticEnvironmentName > > node.name: Nodename > > path.data: /path/to/data > > bootstrap.mlockall: true > > gateway.recover\_after\_nodes: 3 > > cluster.routing.allocation.node\_concurrent\_recoveries: 15 > > indices.recovery.max\_bytes\_per\_sec: 500mb > > discovery.zen.minimum\_master\_nodes: 3 > > discovery.zen.fd.ping\_interval: 15s > discovery.zen.fd.ping\_timeout: 60s > discovery.zen.fd.ping\_retries: 5 > > index.search.slowlog.threshold.query.warn: 10s > > index.search.slowlog.threshold.fetch.warn: 1s > > index.indexing.slowlog.threshold.index.warn: 10s > > indices.fielddata.cache.size: 50% > indices.breaker.fielddata.limit: 60% > > cloud.aws.access\_key: AWSAccessKey > cloud.aws.secret\_key: AWSSecretKey > > On Monday, February 16, 2015 at 1:43:59 AM UTC-6, Mark Walkom wrote: > > > Can you link us to your LS config? > > > > On 14 February 2015 at 08:56, rhea ghosh [rhea....@gmail.com](mailto:rhea....@gmail.com) wrote: > > > > > Hi, I have a 5 node cluster that I'm using as part of an ELK system. > > > Most of the time it works great but today we saw a spike in writes for one > > > of the nodes and around the same time we saw indexing on that node spike up > > > too -- which makes sense, if you have more writes you'll need to do more > > > indexing. But none of the other servers were all that taxed, we don't have > > > anything else writing to Elasticsearch other than Logstash and normally it > > > does a pretty good job of load balancing. Any idea where I could start > > > looking for clues? I was looking through the logs but there doesn't seem to > > > be much information in there, most of it are just debug errors that say > > > something like this: [logstash-2015.02.13][3] failed to execute bulk item > > > (index) index {[logstash-2015.02.13] but they show up pretty consistently > > > so it doesn't seem like anything to be worried about. Where else could I > > > look to see why only one server is getting all the writes and how do I > > > mitigate it if this is happening because it ended up making Elasticsearch > > > unresponsive to information being sent from Logstash. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/2570e228-1d3e-48e0-af8d-0946b7c58197% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/2570e228-1d3e-48e0-af8d-0946b7c58197%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/2570e228-1d3e-48e0-af8d-0946b7c58197%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/46fd8986-6f0b-4573-b4af-07e8a04c1eb0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/46fd8986-6f0b-4573-b4af-07e8a04c1eb0%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/46fd8986-6f0b-4573-b4af-07e8a04c1eb0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/46fd8986-6f0b-4573-b4af-07e8a04c1eb0%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9nn3B\_aTgrRdh3o6a2knRVqkun%3DDyfZUcy38vVzOvpFg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9nn3B_aTgrRdh3o6a2knRVqkun%3DDyfZUcy38vVzOvpFg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:32am UTC](https://discuss.elastic.co/t/spikes-in-usage-and-logstash-connection-issues/22163/5 "2017-07-06T00:32:04Z")