# After update to 1.1 version, delay become ridiculous high
**URL:** https://discuss.elastic.co/t/after-update-to-1-1-version-delay-become-ridiculous-high/17106
**Category:** Elasticsearch
**Created:** [April 21, 2014, 7:44am UTC](https://discuss.elastic.co/t/after-update-to-1-1-version-delay-become-ridiculous-high/17106 "2014-04-21T07:44:34Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@mega](https://discuss.elastic.co/u/mega)
#### Post date: [April 21, 2014, 7:44am UTC](https://discuss.elastic.co/t/after-update-to-1-1-version-delay-become-ridiculous-high/17106/1 "2014-04-21T07:44:34Z")
I'm using 0.90.3 before, have 1 master node, 1 search balance node and 2
data nodes, including 5 indexes, the data directory in each data server is
about 101g, and 24.3% is deleted documents.
The data servers have 24 cores and 32g memory and only running one
elasticsearch process, the jvm arguments are -Xms12g -Xmx12g -Xss256k
-Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError.
We got hundreds request per second, average delay time was about 40ms.
And last friday we update elasticsearch to 1.1.0 version, and we didn't
reindex the documents, just switch to 1.1 and restart all the elasticsearch
processes, seems ok, but today i found out the dalay time become much
higher than before, the average delay time is beyond 500ms.
I guess maybe because lucene's data structure has changed and we didn't
reindex the documents, so there are some extra consume?
Is there anyone know the reason and solution or has the same problem, it's
very urgent.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0fdd1420-0110-4a28-85d0-ec6b26e13ad9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0fdd1420-0110-4a28-85d0-ec6b26e13ad9%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [April 21, 2014, 7:49am UTC](https://discuss.elastic.co/t/after-update-to-1-1-version-delay-become-ridiculous-high/17106/2 "2014-04-21T07:49:55Z")
Yes, Lucene version changed.
What JVM version do you use?
Jörg
On Mon, Apr 21, 2014 at 9:44 AM, mega [yuhg2310@gmail.com](mailto:yuhg2310@gmail.com) wrote:
> I'm using 0.90.3 before, have 1 master node, 1 search balance node and 2
> data nodes, including 5 indexes, the data directory in each data server is
> about 101g, and 24.3% is deleted documents.
>
> The data servers have 24 cores and 32g memory and only running one
> elasticsearch process, the jvm arguments are -Xms12g -Xmx12g -Xss256k
> -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+HeapDumpOnOutOfMemoryError.
>
> We got hundreds request per second, average delay time was about 40ms.
>
> And last friday we update elasticsearch to 1.1.0 version, and we didn't
> reindex the documents, just switch to 1.1 and restart all the elasticsearch
> processes, seems ok, but today i found out the dalay time become much
> higher than before, the average delay time is beyond 500ms.
>
> I guess maybe because lucene's data structure has changed and we didn't
> reindex the documents, so there are some extra consume?
>
> Is there anyone know the reason and solution or has the same problem, it's
> very urgent.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/0fdd1420-0110-4a28-85d0-ec6b26e13ad9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0fdd1420-0110-4a28-85d0-ec6b26e13ad9%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/0fdd1420-0110-4a28-85d0-ec6b26e13ad9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0fdd1420-0110-4a28-85d0-ec6b26e13ad9%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFRk2u%3D%3DjPoF-pkb%3DiL5%3DBPAaSa90zpQW6T2%2BRCLY5xTQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFRk2u%3D%3DjPoF-pkb%3DiL5%3DBPAaSa90zpQW6T2%2BRCLY5xTQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@mega](https://discuss.elastic.co/u/mega)
#### Post date: [April 21, 2014, 7:55am UTC](https://discuss.elastic.co/t/after-update-to-1-1-version-delay-become-ridiculous-high/17106/3 "2014-04-21T07:55:10Z")
1.7.0\_21, everything is the same, except update to 1.1 now, i'm really
confusing right now.
On Monday, April 21, 2014 3:49:55 PM UTC+8, Jörg Prante wrote:
> Yes, Lucene version changed.
>
> What JVM version do you use?
>
> Jörg
>
> On Mon, Apr 21, 2014 at 9:44 AM, mega \<[yuhg...@gmail.com](mailto:yuhg...@gmail.com) \\>wrote:
>
> > I'm using 0.90.3 before, have 1 master node, 1 search balance node and 2
> > data nodes, including 5 indexes, the data directory in each data server is
> > about 101g, and 24.3% is deleted documents.
> >
> > The data servers have 24 cores and 32g memory and only running one
> > elasticsearch process, the jvm arguments are -Xms12g -Xmx12g -Xss256k
> > -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> > -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> > -XX:+HeapDumpOnOutOfMemoryError.
> >
> > We got hundreds request per second, average delay time was about 40ms.
> >
> > And last friday we update elasticsearch to 1.1.0 version, and we didn't
> > reindex the documents, just switch to 1.1 and restart all the elasticsearch
> > processes, seems ok, but today i found out the dalay time become much
> > higher than before, the average delay time is beyond 500ms.
> >
> > I guess maybe because lucene's data structure has changed and we didn't
> > reindex the documents, so there are some extra consume?
> >
> > Is there anyone know the reason and solution or has the same problem,
> > it's very urgent.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/0fdd1420-0110-4a28-85d0-ec6b26e13ad9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0fdd1420-0110-4a28-85d0-ec6b26e13ad9%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/0fdd1420-0110-4a28-85d0-ec6b26e13ad9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0fdd1420-0110-4a28-85d0-ec6b26e13ad9%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9af8b9f8-4604-4bb3-80c0-a4ae6dc75b28%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9af8b9f8-4604-4bb3-80c0-a4ae6dc75b28%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [April 21, 2014, 8:05am UTC](https://discuss.elastic.co/t/after-update-to-1-1-version-delay-become-ridiculous-high/17106/4 "2014-04-21T08:05:05Z")
Search times may increase when segment number is getting high, do you
create new documents since upgrading?
Maybe you can post a \_stats output of a slow index? Or is something in the
slow query log?
For myself, I decided to always reindex everything after a Lucene version
change, to get a clean index structure from ground up.
Jörg
On Mon, Apr 21, 2014 at 9:55 AM, mega [yuhg2310@gmail.com](mailto:yuhg2310@gmail.com) wrote:
> 1.7.0\_21, everything is the same, except update to 1.1 now, i'm really
> confusing right now.
>
> On Monday, April 21, 2014 3:49:55 PM UTC+8, Jörg Prante wrote:
>
> > Yes, Lucene version changed.
> >
> > What JVM version do you use?
> >
> > Jörg
> >
> > On Mon, Apr 21, 2014 at 9:44 AM, mega [yuhg...@gmail.com](mailto:yuhg...@gmail.com) wrote:
> >
> > > I'm using 0.90.3 before, have 1 master node, 1 search balance node and 2
> > > data nodes, including 5 indexes, the data directory in each data server is
> > > about 101g, and 24.3% is deleted documents.
> > >
> > > The data servers have 24 cores and 32g memory and only running one
> > > elasticsearch process, the jvm arguments are -Xms12g -Xmx12g -Xss256k
> > > -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:
> > > CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> > > -XX:+HeapDumpOnOutOfMemoryError.
> > >
> > > We got hundreds request per second, average delay time was about 40ms.
> > >
> > > And last friday we update elasticsearch to 1.1.0 version, and we didn't
> > > reindex the documents, just switch to 1.1 and restart all the elasticsearch
> > > processes, seems ok, but today i found out the dalay time become much
> > > higher than before, the average delay time is beyond 500ms.
> > >
> > > I guess maybe because lucene's data structure has changed and we didn't
> > > reindex the documents, so there are some extra consume?
> > >
> > > Is there anyone know the reason and solution or has the same problem,
> > > it's very urgent.
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > >
> > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > msgid/elasticsearch/0fdd1420-0110-4a28-85d0-ec6b26e13ad9%
> > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/0fdd1420-0110-4a28-85d0-ec6b26e13ad9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0fdd1420-0110-4a28-85d0-ec6b26e13ad9%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/9af8b9f8-4604-4bb3-80c0-a4ae6dc75b28%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9af8b9f8-4604-4bb3-80c0-a4ae6dc75b28%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/9af8b9f8-4604-4bb3-80c0-a4ae6dc75b28%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9af8b9f8-4604-4bb3-80c0-a4ae6dc75b28%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHNDFLReMnL\_g4Yv2Je-do2MV52CBO00qbh1OvQcKCm4Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHNDFLReMnL_g4Yv2Je-do2MV52CBO00qbh1OvQcKCm4Q%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@mega](https://discuss.elastic.co/u/mega)
#### Post date: [April 21, 2014, 8:21am UTC](https://discuss.elastic.co/t/after-update-to-1-1-version-delay-become-ridiculous-high/17106/5 "2014-04-21T08:21:45Z")
that's indices stats, [https://gist.github.com/megayu/11135838](https://gist.github.com/megayu/11135838)
slow log is like:
[2014-04-21 16:07:00,221][TRACE][index.search.slowlog.query] [Data-2]
[main][6] took[529.3ms], took\_millis[529], types[main], stats,
search\_type[QUERY\_THEN\_FETCH], total\_shards[8],
source[{"fields":"id","query":{"function\_score":{"boost\_mode":"replace","query":{"filtered":{"query":{"match":{"text":{"query":"test","operator":"and"}}},"filter":{"bool":{"must":{"exists":{"field":"category"}},"must\_not":{"terms":{"category":["a","b"]}}}}}},"script\_score":{"script":"simple"}}}}],
extra\_source[{"from":220,"size":20}]
"script":"simple" is some mathematic calculation, like cos, kind complex,
but before update it's the same algorithm.
Right now almost every request is in slow log.
On Monday, April 21, 2014 4:05:05 PM UTC+8, Jörg Prante wrote:
> Search times may increase when segment number is getting high, do you
> create new documents since upgrading?
>
> Maybe you can post a \_stats output of a slow index? Or is something in the
> slow query log?
>
> For myself, I decided to always reindex everything after a Lucene version
> change, to get a clean index structure from ground up.
>
> Jörg
>
> On Mon, Apr 21, 2014 at 9:55 AM, mega \<[yuhg...@gmail.com](mailto:yuhg...@gmail.com) \\>wrote:
>
> > 1.7.0\_21, everything is the same, except update to 1.1 now, i'm really
> > confusing right now.
> >
> > On Monday, April 21, 2014 3:49:55 PM UTC+8, Jörg Prante wrote:
> >
> > > Yes, Lucene version changed.
> > >
> > > What JVM version do you use?
> > >
> > > Jörg
> > >
> > > On Mon, Apr 21, 2014 at 9:44 AM, mega [yuhg...@gmail.com](mailto:yuhg...@gmail.com) wrote:
> > >
> > > > I'm using 0.90.3 before, have 1 master node, 1 search balance node and
> > > > 2 data nodes, including 5 indexes, the data directory in each data server
> > > > is about 101g, and 24.3% is deleted documents.
> > > >
> > > > The data servers have 24 cores and 32g memory and only running one
> > > > elasticsearch process, the jvm arguments are -Xms12g -Xmx12g -Xss256k
> > > > -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:
> > > > CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> > > > -XX:+HeapDumpOnOutOfMemoryError.
> > > >
> > > > We got hundreds request per second, average delay time was about 40ms.
> > > >
> > > > And last friday we update elasticsearch to 1.1.0 version, and we didn't
> > > > reindex the documents, just switch to 1.1 and restart all the elasticsearch
> > > > processes, seems ok, but today i found out the dalay time become much
> > > > higher than before, the average delay time is beyond 500ms.
> > > >
> > > > I guess maybe because lucene's data structure has changed and we didn't
> > > > reindex the documents, so there are some extra consume?
> > > >
> > > > Is there anyone know the reason and solution or has the same problem,
> > > > it's very urgent.
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > >
> > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > msgid/elasticsearch/0fdd1420-0110-4a28-85d0-ec6b26e13ad9%
> > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/0fdd1420-0110-4a28-85d0-ec6b26e13ad9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0fdd1420-0110-4a28-85d0-ec6b26e13ad9%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > .
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/9af8b9f8-4604-4bb3-80c0-a4ae6dc75b28%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9af8b9f8-4604-4bb3-80c0-a4ae6dc75b28%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/9af8b9f8-4604-4bb3-80c0-a4ae6dc75b28%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9af8b9f8-4604-4bb3-80c0-a4ae6dc75b28%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/443e7fe1-b5b7-4716-a8f5-081c08496333%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/443e7fe1-b5b7-4716-a8f5-081c08496333%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:34am UTC](https://discuss.elastic.co/t/after-update-to-1-1-version-delay-become-ridiculous-high/17106/6 "2017-07-06T01:34:42Z")