# Overloading by indexing? **URL:** \ **Category:** Elasticsearch\ **Created:** [June 24, 2014, 2:33pm UTC](https://discuss.elastic.co/t/overloading-by-indexing/18294 "2014-06-24T14:33:13Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Julius\_K](https://avatars.discourse-cdn.com/v4/letter/j/3d9bf3/32.png) [@Julius\_K](https://discuss.elastic.co/u/Julius_K)\ **Post date:** [June 24, 2014, 2:33pm UTC](https://discuss.elastic.co/t/overloading-by-indexing/18294/1 "2014-06-24T14:33:13Z")
Hi, I have a strange problem with ES. It's running on a cluster with 64 cores etc, so I don't think the power of the hardware is the issue. I want to index a lot of documents with elasticsearch-hadoop. After some problems I now have everything into place and it seems to work fine. So I wrote a simple pig script which loads all the files (~500) and stores them into an ES index. However, after ~22h the job failed, because of connection problems between the nodes. But during that time, there wasn't any heavy usage of network bandwidth or other ressources. After that I tried to run the pig script only for one document so I know what is indexed and what is missing. After about 3 documents indexed well doing this, the jobs started to fail again, due to network problems although there wasn't any significant load. I observed that even after the indexing jobs stopped, there was stuff happening with the index. The number of documents kept growing for quite some time and the translog operations went up and down being mostly at about half a million. For me this looks like the index takes more time indexing than the pig script takes for writing into the index and after some time somewhere a buffer gets too full. Is this possible? I would expect, that in this case elasticsearch-hadoop should get throttled. The only documentation about the translog is what I found here: [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-translog.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-translog.html) which I find a bit little. I still don't know what implications the number of translog operations has. On the linked page it says, I could increase the numbers when doing bulk indexing but I don't understand how this would help. Also what's TPS? Best regards Julius -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d17e1231-da99-4bc2-b019-806046ffd34e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d17e1231-da99-4bc2-b019-806046ffd34e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 24, 2014, 11:23pm UTC](https://discuss.elastic.co/t/overloading-by-indexing/18294/2 "2014-06-24T23:23:19Z")
TPS is usually transactions per second. Are you monitoring your cluster, and your memory/heap usage? How are you coming to the conclusion that it's a networking issue? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 25 June 2014 00:33, Julius K [fooliuskoolius@gmail.com](mailto:fooliuskoolius@gmail.com) wrote: > Hi, > > I have a strange problem with ES. It's running on a cluster with 64 cores > etc, so I don't think the power of the hardware is the issue. > > I want to index a lot of documents with elasticsearch-hadoop. > After some problems I now have everything into place and it seems to work > fine. > > So I wrote a simple pig script which loads all the files (~500) and stores > them into an ES index. > However, after ~22h the job failed, because of connection problems between > the nodes. > But during that time, there wasn't any heavy usage of network bandwidth or > other ressources. > > After that I tried to run the pig script only for one document so I know > what is indexed and what is missing. > After about 3 documents indexed well doing this, the jobs started to fail > again, due to network problems although there wasn't any significant load. > > I observed that even after the indexing jobs stopped, there was stuff > happening with the index. The number of documents kept growing for quite > some time and the translog operations went up and down being mostly at > about half a million. > > For me this looks like the index takes more time indexing than the pig > script takes for writing into the index and after some time somewhere a > buffer gets too full. > > Is this possible? I would expect, that in this case elasticsearch-hadoop > should get throttled. > > The only documentation about the translog is what I found here: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-translog.html) > which I find a bit little. I still don't know what implications the number > of translog operations has. > > On the linked page it says, I could increase the numbers when doing bulk > indexing but I don't understand how this would help. > Also what's TPS? > > Best regards > Julius > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/d17e1231-da99-4bc2-b019-806046ffd34e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d17e1231-da99-4bc2-b019-806046ffd34e%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/d17e1231-da99-4bc2-b019-806046ffd34e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d17e1231-da99-4bc2-b019-806046ffd34e%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624bSM3t2j%3DqJwBe4%3DjhSkd5BSJskoJtCf0Qj7amD3pUa0w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624bSM3t2j%3DqJwBe4%3DjhSkd5BSJskoJtCf0Qj7amD3pUa0w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Julius\_K](https://avatars.discourse-cdn.com/v4/letter/j/3d9bf3/32.png) [@Julius\_K](https://discuss.elastic.co/u/Julius_K)\ **Post date:** [June 25, 2014, 4:51pm UTC](https://discuss.elastic.co/t/overloading-by-indexing/18294/3 "2014-06-25T16:51:49Z")
2014-06-25 1:23 GMT+02:00 Mark Walkom [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com): > TPS is usually transactions per second. > Ok, thanks > > Are you monitoring your cluster, and your memory/heap usage? > Yes, that is what I meant with no significant load and no heavy usage > of ressources. > How are you coming to the conclusion that it's a networking issue? > The error messages say something like connection timed out and similar. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 25 June 2014 00:33, Julius K [fooliuskoolius@gmail.com](mailto:fooliuskoolius@gmail.com) wrote: > > > Hi, > > > > I have a strange problem with ES. It's running on a cluster with 64 cores > > etc, so I don't think the power of the hardware is the issue. > > > > I want to index a lot of documents with elasticsearch-hadoop. > > After some problems I now have everything into place and it seems to work > > fine. > > > > So I wrote a simple pig script which loads all the files (~500) and stores > > them into an ES index. > > However, after ~22h the job failed, because of connection problems between > > the nodes. > > But during that time, there wasn't any heavy usage of network bandwidth or > > other ressources. > > > > After that I tried to run the pig script only for one document so I know > > what is indexed and what is missing. > > After about 3 documents indexed well doing this, the jobs started to fail > > again, due to network problems although there wasn't any significant load. > > > > I observed that even after the indexing jobs stopped, there was stuff > > happening with the index. The number of documents kept growing for quite > > some time and the translog operations went up and down being mostly at about > > half a million. > > > > For me this looks like the index takes more time indexing than the pig > > script takes for writing into the index and after some time somewhere a > > buffer gets too full. > > > > Is this possible? I would expect, that in this case elasticsearch-hadoop > > should get throttled. > > > > The only documentation about the translog is what I found here: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-translog.html) > > which I find a bit little. I still don't know what implications the number > > of translog operations has. > > > > On the linked page it says, I could increase the numbers when doing bulk > > indexing but I don't understand how this would help. > > Also what's TPS? > > > > Best regards > > Julius > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/d17e1231-da99-4bc2-b019-806046ffd34e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d17e1231-da99-4bc2-b019-806046ffd34e%40googlegroups.com). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/ipm7UQpQO88/unsubscribe](https://groups.google.com/d/topic/elasticsearch/ipm7UQpQO88/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEM624bSM3t2j%3DqJwBe4%3DjhSkd5BSJskoJtCf0Qj7amD3pUa0w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624bSM3t2j%3DqJwBe4%3DjhSkd5BSJskoJtCf0Qj7amD3pUa0w%40mail.gmail.com). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAN%3Dtz-j-SgU7ZCJbHYF%2B9CcWPiRTYVGk\_qCLmCm\_wBMA6n7AQQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAN%3Dtz-j-SgU7ZCJbHYF%2B9CcWPiRTYVGk_qCLmCm_wBMA6n7AQQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:19am UTC](https://discuss.elastic.co/t/overloading-by-indexing/18294/4 "2017-07-06T01:19:44Z")
Closed on July 6, 2017, 1:19am UTC