# Upsert and Script on large index cause the cluster to timeout

**URL:** <https://discuss.elastic.co/t/upsert-and-script-on-large-index-cause-the-cluster-to-timeout/21154>\
**Category:** Elasticsearch\
**Created:** [December 8, 2014, 7:49pm UTC](https://discuss.elastic.co/t/upsert-and-script-on-large-index-cause-the-cluster-to-timeout/21154 "2014-12-08T19:49:25Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Christophe\_Verbinnen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christophe_verbinnen/32/1054_2.png) [@Christophe\_Verbinnen](https://discuss.elastic.co/u/Christophe_Verbinnen)\
**Post date:** [December 8, 2014, 7:49pm UTC](https://discuss.elastic.co/t/upsert-and-script-on-large-index-cause-the-cluster-to-timeout/21154/1 "2014-12-08T19:49:25Z")

</div>

Hello,

We have a small cluster with 3 nodes running 1.3.6.

I have an index setup with only two fields.

```
      {
        index: index_name,
        body: {
          settings: {
              number_of_shards: 3,
              store: {
                type: :mmapfs
              }
          },
          mappings: {
            mapping_name => {
              properties: {
                :value => {type: 'string', analyzer: 'keyword'},
                :post_ids => {type: 'long', index: 'not_analyzed'}
              }
            }
          }
        }
      }

```

We are basically storing strings and all the post they are related to.

The problem is that this data is not stored this way in the database so I  
don't have an id to represent each string nor do I have all the post\_ids  
from the start.

So I use the sha1 of the string value as id and I use and script to append  
to the post\_ids.

Here is my code that I use to index using the bulk api end point.

def index!  
posts\_ids = Post.where...  
bulk\_data = []  
strings.uniq.each do |string|  
string\_id = Digest::SHA1.hexdigest string  
bulk\_data \<\<  
{  
update:  
{  
\_index: 'post\_strings',  
\_type: 'post\_string',  
\_id: string\_id,  
data: {  
script: "ctx.\_source.post\_ids += additional\_post\_ids",  
params: {  
additional\_post\_ids: post\_ids  
},  
upsert: {  
value: string,  
post\_ids: post\_ids  
}  
}  
}  
}  
if bulk\_data.count == 100  
$elasticsearch.bulk :body =\> bulk\_data  
bulk\_data = []  
end  
end  
$elasticsearch.bulk :body =\> bulk\_data if bulk\_data.any?  
end

So this worked fine for the first 75 Million strings but It was getting  
slower and slower until it reached an indexing rate of only 50 doc per sec.

After that the cluster just killed itself because the nodes couldn't take  
to each other.

I'm gessing all the threads were blocked trying to index and nodes had no  
available threads to respond.

At first I tought it would be related to the sha1 id being not very  
efficient but with my test with sequencial ids it was not getting better.

I'm out of ideas right now. Any help would be greatly appreciated.

Cheers.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/82c27f2c-bf56-4064-80bc-b348203edcb5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/82c27f2c-bf56-4064-80bc-b348203edcb5%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [December 8, 2014, 8:03pm UTC](https://discuss.elastic.co/t/upsert-and-script-on-large-index-cause-the-cluster-to-timeout/21154/2 "2014-12-08T20:03:47Z")

</div>

I'm not sure what is up but remember that post\_ids in the script is a list  
not a set. You might be growing it without bounds.  
On Dec 8, 2014 2:49 PM, "Christophe Verbinnen" [djpate@gmail.com](mailto:djpate@gmail.com) wrote:

> Hello,
> 
> We have a small cluster with 3 nodes running 1.3.6.
> 
> I have an index setup with only two fields.
> 
> ```
> {
> index: index_name,
> body: {
> settings: {
> number_of_shards: 3,
> store: {
> type: :mmapfs
> }
> },
> mappings: {
> mapping_name => {
> properties: {
> :value => {type: 'string', analyzer: 'keyword'},
> :post_ids => {type: 'long', index: 'not_analyzed'}
> }
> }
> }
> }
> }
> 
> ```
> 
> We are basically storing strings and all the post they are related to.
> 
> The problem is that this data is not stored this way in the database so I  
> don't have an id to represent each string nor do I have all the post\_ids  
> from the start.
> 
> So I use the sha1 of the string value as id and I use and script to append  
> to the post\_ids.
> 
> Here is my code that I use to index using the bulk api end point.
> 
> def index!  
> posts\_ids = Post.where...  
> bulk\_data =   
> strings.uniq.each do |string|  
> string\_id = Digest::SHA1.hexdigest string  
> bulk\_data \<\<  
> {  
> update:  
> {  
> \_index: 'post\_strings',  
> \_type: 'post\_string',  
> \_id: string\_id,  
> data: {  
> script: "ctx.\_source.post\_ids += additional\_post\_ids",  
> params: {  
> additional\_post\_ids: post\_ids  
> },  
> upsert: {  
> value: string,  
> post\_ids: post\_ids  
> }  
> }  
> }  
> }  
> if bulk\_data.count == 100  
> $elasticsearch.bulk :body =\> bulk\_data  
> bulk\_data =   
> end  
> end  
> $elasticsearch.bulk :body =\> bulk\_data if bulk\_data.any?  
> end
> 
> So this worked fine for the first 75 Million strings but It was getting  
> slower and slower until it reached an indexing rate of only 50 doc per sec.
> 
> After that the cluster just killed itself because the nodes couldn't take  
> to each other.
> 
> I'm gessing all the threads were blocked trying to index and nodes had no  
> available threads to respond.
> 
> At first I tought it would be related to the sha1 id being not very  
> efficient but with my test with sequencial ids it was not getting better.
> 
> I'm out of ideas right now. Any help would be greatly appreciated.
> 
> Cheers.
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/82c27f2c-bf56-4064-80bc-b348203edcb5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/82c27f2c-bf56-4064-80bc-b348203edcb5%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/82c27f2c-bf56-4064-80bc-b348203edcb5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/82c27f2c-bf56-4064-80bc-b348203edcb5%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0\_qr%2B-jU%2BYgPiN-hA283aGgoy-UtH3j5-0wEJBCuP2Mg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0_qr%2B-jU%2BYgPiN-hA283aGgoy-UtH3j5-0wEJBCuP2Mg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Christophe\_Verbinnen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christophe_verbinnen/32/1054_2.png) [@Christophe\_Verbinnen](https://discuss.elastic.co/u/Christophe_Verbinnen)\
**Post date:** [December 8, 2014, 10:08pm UTC](https://discuss.elastic.co/t/upsert-and-script-on-large-index-cause-the-cluster-to-timeout/21154/3 "2014-12-08T22:08:28Z")

</div>

I see what you mean but the way my records are it cannot happen unless I  
reindex it.

Le lundi 8 décembre 2014 12:05:13 UTC-8, Nikolas Everett a écrit :

> I'm not sure what is up but remember that post\_ids in the script is a list  
> not a set. You might be growing it without bounds.  
> On Dec 8, 2014 2:49 PM, "Christophe Verbinnen" \<[djp...@gmail.com](mailto:djp...@gmail.com)  
> \<javascript:\>\> wrote:
> 
> > Hello,
> > 
> > We have a small cluster with 3 nodes running 1.3.6.
> > 
> > I have an index setup with only two fields.
> > 
> > ```
> > {
> > index: index_name,
> > body: {
> > settings: {
> > number_of_shards: 3,
> > store: {
> > type: :mmapfs
> > }
> > },
> > mappings: {
> > mapping_name => {
> > properties: {
> > :value => {type: 'string', analyzer: 'keyword'},
> > :post_ids => {type: 'long', index: 'not_analyzed'}
> > }
> > }
> > }
> > }
> > }
> > 
> > ```
> > 
> > We are basically storing strings and all the post they are related to.
> > 
> > The problem is that this data is not stored this way in the database so I  
> > don't have an id to represent each string nor do I have all the post\_ids  
> > from the start.
> > 
> > So I use the sha1 of the string value as id and I use and script to  
> > append to the post\_ids.
> > 
> > Here is my code that I use to index using the bulk api end point.
> > 
> > def index!  
> > posts\_ids = Post.where...  
> > bulk\_data =   
> > strings.uniq.each do |string|  
> > string\_id = Digest::SHA1.hexdigest string  
> > bulk\_data \<\<  
> > {  
> > update:  
> > {  
> > \_index: 'post\_strings',  
> > \_type: 'post\_string',  
> > \_id: string\_id,  
> > data: {  
> > script: "ctx.\_source.post\_ids += additional\_post\_ids",  
> > params: {  
> > additional\_post\_ids: post\_ids  
> > },  
> > upsert: {  
> > value: string,  
> > post\_ids: post\_ids  
> > }  
> > }  
> > }  
> > }  
> > if bulk\_data.count == 100  
> > $elasticsearch.bulk :body =\> bulk\_data  
> > bulk\_data =   
> > end  
> > end  
> > $elasticsearch.bulk :body =\> bulk\_data if bulk\_data.any?  
> > end
> > 
> > So this worked fine for the first 75 Million strings but It was getting  
> > slower and slower until it reached an indexing rate of only 50 doc per sec.
> > 
> > After that the cluster just killed itself because the nodes couldn't take  
> > to each other.
> > 
> > I'm gessing all the threads were blocked trying to index and nodes had no  
> > available threads to respond.
> > 
> > At first I tought it would be related to the sha1 id being not very  
> > efficient but with my test with sequencial ids it was not getting better.
> > 
> > I'm out of ideas right now. Any help would be greatly appreciated.
> > 
> > Cheers.
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/82c27f2c-bf56-4064-80bc-b348203edcb5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/82c27f2c-bf56-4064-80bc-b348203edcb5%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/82c27f2c-bf56-4064-80bc-b348203edcb5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/82c27f2c-bf56-4064-80bc-b348203edcb5%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/740ab5ce-eaef-4ae4-9a00-f50be5aa45c3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/740ab5ce-eaef-4ae4-9a00-f50be5aa45c3%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:44am UTC](https://discuss.elastic.co/t/upsert-and-script-on-large-index-cause-the-cluster-to-timeout/21154/4 "2017-07-06T00:44:51Z")

</div>