Elasticsearch 1.1.0 - Optimize broken?

Elastic Stack Elasticsearch
21

Hi,

We also have this issue in 1.0.2. Per index we have over 1000 segments.
Optimize doesn't reduce them.

Would love to see a fix for this.

Thanks,
Thibaut

On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett nik9000@gmail.com wrote:

For reference I'm also on 1.1.0 but I'm not seeing more segments then I
expect. I see an average of ~28 per shard on an index I write to
constantly. I don't write all that quickly, < 50 updates a second.

Nik

On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand <
adrien.grand@elasticsearch.com> wrote:

I managed to reproduce the issue locally, I'm looking into it.

On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin walro467@gmail.com wrote:

Hello,

We are experiencing a related problem with 1.1.0. Segments do not seem
to merge as they should during indexing. The optimize API does practically
nothing in terms of lowering the segments count either. The problem
persists through a cluster restart. The vast amount of segments seem to be
greatly impact the performance of the cluster, in a very negative way.

We currently have 414 million documents across 3 nodes, each shard has
in average 1200 segments(!).

With 1.0.1 we had even more documents, ~650 million, without any segment
problems. Looking in Marvel we were hovering at around 30-40 segments per
shard back then.

Best Regards,
Robin

On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:

Thanks for reporting this, the behavior is definitely unexpected. I'll
test _optimize on very large numbers of shards to see if I can reproduce
the issue.

On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw ebrad...@gmail.comwrote:

Adrien,

Just an FYI, after resetting the cluster, things seem to have
improved. Optimize calls now lead to CPU/IO activity over their duration.
Max_num_segments=1 does not seem to be working for me on any given call, as
each call would only reduce the segment count by about 600-700. I ran 10
calls in sequence overnight, and actually got down to 4 segments (1/shard)!

I'm glad I got the index optimized, searches are literally 10-20 times
faster without 1500/segments per shard to deal with. It's awesome.

That said, any thoughts on why the index wasn't merging on its own, or
why optimize was returning prematurely?

On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw wrote:

Hi Adrien,

I kept the logs up over the last optimize call, and I did see an
exception. I Ctrl-C'd a curl optimize call before making another one, but
I don't think that that caused this exception. The error is essentially as
follows:

netty - Caught exception while handling client http traffic, closing
connection [id: 0x4d8f1a90, /127.0.0.1:33480 :> /127.0.0.1:9200]

java.nio.channels.ClosedChannelException at AbstractNioWorker.
cleanUpWriteBuffer(AbstractNioWorker.java:433)
at AbstractNioWorker.writeFromUserCode
at NioServerSocketPipelineSink.handleAcceptedSocket
at NioServerSocketPipelineSink.eventSunk
at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDownstream
at Channels.write
at OneToOneEncoder.doEncode
at OneToOneEncoder.handleDownstream
at DefaultChannelPipeline.sendDownstream
at DefaultChannelPipeline.sendDownstream
at Channels.write
at AbstractChannel.write
at NettyHttpChannel.sendResponse
at RestOptimizeAction$1.onResponse(95)
at RestOptimizeAction$1.onResponse(85)
at TransportBroadcastOperationAction$AsyncBroadcastAction.finishHim
at TransportBroadcastOperationAction$AsyncBroadcastAction.onOperation
at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run

Sorry about the crappy stack trace. Still, looks like this might
point to a problem! The exception fired about an hour after I kicked off
the optimize. Any thoughts?

On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw wrote:

Hi Adrien,

I did customize my merge policy, although I did so only because I
was so surprised by the number of segments left over after the load. I'm
pretty sure the optimize problem was happening before I made this change,
but either way here are my settings:

"index" : {
"merge" : {
"policy" : {
"max_merged_segment" : "20gb",
"segments_per_tier" : 5,
"floor_segment" : "10mb"
},
"scheduler" : "concurrentmergescheduler"
}
}

Not sure whether this set up could be a contributing factor or not.
Nothing really jumps out at me in the logs. In fact, when i kick off the
optimize, I don't see any logging at all. Should I?

I'm running the following command: curl -XPOST
http://localhost:9200/index/_optimize

Thanks!

On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote:

Hi Elliott,

1500 segments per shard is certainly way too much, and it is not
normal that optimize doesn't manage to reduce the number of segments.

  • Is there anything suspicious in the logs?
  • Have you customized the merge policy or scheduler?[1]
  • Does the issue still reproduce if you restart your cluster?

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
e/current/index-modules-merge.html

On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any other thoughts on this? Would 1500 segments per shard be
significantly impacting performance? Have you guys noticed this behavior
elsewhere?

Thanks.

On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw wrote:

Adrian,

I ran the following command:

curl -XPUT http://localhost:9200/_settings -d
'{"indices.store.throttle.max_bytes_per_sec" : "10gb"}'

and received a { "acknowledged" : "true" } response. The logs
showed "cluster state updated".

I did have to close my index prior to changing the setting and
reopen afterward.

I've since began another optimize, but again it doesn't look like
much is happening. The optimize isn't returning and the total CPU usage on
every node is holding at about 2% of a single core. I would copy a
hot_threads stack trace, but I'm unfortunately on a closed network and this
isn't possible. I can tell you that refreshes of hot_threads show vary
little happening. The occasional [merge] thread (always in a
LinkedTransferQueue.awaitMatch() state) or [optimize] (doing
nothing on a waitForMerge() call) thread shows up, but it's always
consuming 0-1% CPU. It sure feels like something isn't right. Any
thoughts?

On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand <
adrien...@elasticsearch.com> wrote:

Did you see a message in the logs confirming that the setting
has been updated? It would be interesting to see the output of hot
threads[1] to see what your node is doing.

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
e/current/cluster-nodes-hot-threads.html

On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Yes. I have run max_num_segments=1 every time.

On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick <
michae...@serenesoftware.com> wrote:

Have you tried max_num_segments=1 on your optimize?

On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any thoughts on this? I've run optimize several more times,
and the number of segments falls each time, but I'm still over 1000
segments per shard. Has anyone else run into something similar?

On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott
Bradshaw wrote:

OK. Optimize finally returned, so I suppose something was
happening in the background, but I'm still seeing over 6500 segments. Even
after setting max_num_segments=5. Does this seem right? Queries are a
little faster (350-400ms) but still not great. Bigdesk is still showing a
fair amount of file IO.

On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott
Bradshaw wrote:

Hi All,

I've recently upgraded to Elasticsearch 1.1.0. I've got a
4 node cluster, each with 64G of ram, with 24G allocated to Elasticsearch
on each. I've batch loaded approximately 86 million documents into a
single index (4 shards) and have started benchmarking
cross_field/multi_match queries on them. The index has one replica and
takes up a total of 111G. I've run several batches of warming queries, but
queries are not as fast as I had hoped, approximately 400-500ms each.
Given that *top *(on Centos) shows 5-8 GB of free memory
on each server, I would assume that the entire index has been paged into
memory (I had worried about disk performance previously, as we are working
in a virtualized environment).

A stats query on the index in questions shows that the
index is composed of > 7000 segments. This seemed high to me, but maybe
it's appropriate. Regardless, I dispatched an optimize command, but I am
not seeing any progress and the command has not returned. Current merges
remains at zero, and the segment count is not changing. Checking out hot
threads in ElasticHQ, I initially saw an optimize call in the stack that
was blocked on a waitForMerge call. This however has disappeared, and I'm
seeing no evidence that the optimize is occuring.

Does any of this seem out of the norm or unusual? Has
anyone else had similar issues. This is the second time I have tried to
optimize an index since upgrading. I've gotten the same result both time.

Thanks in advance for any help/tips!

  • Elliott

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5
e-4088-a1f2-93272beef0bb%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a
topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/
unsubscribe.
To unsubscribe from this group and all its topics, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUz
iGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS
QTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gma
il.comhttps://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to a topic
in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/
unsubscribe.
To unsubscribe from this group and all its topics, send an email
to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP
jijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8742280e-922
f-4e91-bcb2-6096ca0165e6%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

22

I see also exploding numbers in a quick local test on 1.1.0, thousands of
segments are created, but I'm not sure if this behavior is expected or not.
But that does not look like it should. It slows down bulk indexing
significantly.

Issuing an optimize just stalls...

curl '0:9200/test/_optimize?max_num_segments=1'

Does anyone see something similar?

Stack trace excerpt:

"elasticsearch[Doctor Strange][optimize][T#1]" #81 daemon prio=5 os_prio=31
tid=0x00007f85cdd9f800 nid=0xc507 in Object.wait() [0x000000012c140000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4339)

  • locked <0x00000006cee0fad0> (a org.apache.lucene.index.IndexWriter)
    at org.apache.lucene.index.IndexWriter.waitForMerges(IndexWriter.java:2262)
  • locked <0x00000006cee0fad0> (a org.apache.lucene.index.IndexWriter)
    at
    org.elasticsearch.index.engine.internal.InternalEngine.optimize(InternalEngine.java:998)
    at
    org.elasticsearch.index.shard.service.InternalIndexShard.optimize(InternalIndexShard.java:599)
    at
    org.elasticsearch.action.admin.indices.optimize.TransportOptimizeAction.shardOperation(TransportOptimizeAction.java:115)
    at
    org.elasticsearch.action.admin.indices.optimize.TransportOptimizeAction.shardOperation(TransportOptimizeAction.java:49)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$2.run(TransportBroadcastOperationAction.java:226)
    at
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)

(node name "Doctor Strange" was not my selection btw)

Jörg

On Sat, Apr 12, 2014 at 8:09 PM, Thibaut Britz t.britz@trendiction.comwrote:

Hi,

We also have this issue in 1.0.2. Per index we have over 1000 segments.
Optimize doesn't reduce them.

Would love to see a fix for this.

Thanks,
Thibaut

On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett nik9000@gmail.comwrote:

For reference I'm also on 1.1.0 but I'm not seeing more segments then I
expect. I see an average of ~28 per shard on an index I write to
constantly. I don't write all that quickly, < 50 updates a second.

Nik

On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand <
adrien.grand@elasticsearch.com> wrote:

I managed to reproduce the issue locally, I'm looking into it.

On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin walro467@gmail.comwrote:

Hello,

We are experiencing a related problem with 1.1.0. Segments do not seem
to merge as they should during indexing. The optimize API does practically
nothing in terms of lowering the segments count either. The problem
persists through a cluster restart. The vast amount of segments seem to be
greatly impact the performance of the cluster, in a very negative way.

We currently have 414 million documents across 3 nodes, each shard has
in average 1200 segments(!).

With 1.0.1 we had even more documents, ~650 million, without any
segment problems. Looking in Marvel we were hovering at around 30-40
segments per shard back then.

Best Regards,
Robin

On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:

Thanks for reporting this, the behavior is definitely unexpected. I'll
test _optimize on very large numbers of shards to see if I can reproduce
the issue.

On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw <ebrad...@gmail.com

wrote:

Adrien,

Just an FYI, after resetting the cluster, things seem to have
improved. Optimize calls now lead to CPU/IO activity over their duration.
Max_num_segments=1 does not seem to be working for me on any given call, as
each call would only reduce the segment count by about 600-700. I ran 10
calls in sequence overnight, and actually got down to 4 segments (1/shard)!

I'm glad I got the index optimized, searches are literally 10-20
times faster without 1500/segments per shard to deal with. It's awesome.

That said, any thoughts on why the index wasn't merging on its own,
or why optimize was returning prematurely?

On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw wrote:

Hi Adrien,

I kept the logs up over the last optimize call, and I did see an
exception. I Ctrl-C'd a curl optimize call before making another one, but
I don't think that that caused this exception. The error is essentially as
follows:

netty - Caught exception while handling client http traffic, closing
connection [id: 0x4d8f1a90, /127.0.0.1:33480 :> /127.0.0.1:9200]

java.nio.channels.ClosedChannelException at AbstractNioWorker.
cleanUpWriteBuffer(AbstractNioWorker.java:433)
at AbstractNioWorker.writeFromUserCode
at NioServerSocketPipelineSink.handleAcceptedSocket
at NioServerSocketPipelineSink.eventSunk
at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDown
stream
at Channels.write
at OneToOneEncoder.doEncode
at OneToOneEncoder.handleDownstream
at DefaultChannelPipeline.sendDownstream
at DefaultChannelPipeline.sendDownstream
at Channels.write
at AbstractChannel.write
at NettyHttpChannel.sendResponse
at RestOptimizeAction$1.onResponse(95)
at RestOptimizeAction$1.onResponse(85)
at TransportBroadcastOperationAction$AsyncBroadcastAction.finishHim
at TransportBroadcastOperationAction$AsyncBroadcastAction.onOpe
ration
at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run

Sorry about the crappy stack trace. Still, looks like this might
point to a problem! The exception fired about an hour after I kicked off
the optimize. Any thoughts?

On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw
wrote:

Hi Adrien,

I did customize my merge policy, although I did so only because I
was so surprised by the number of segments left over after the load. I'm
pretty sure the optimize problem was happening before I made this change,
but either way here are my settings:

"index" : {
"merge" : {
"policy" : {
"max_merged_segment" : "20gb",
"segments_per_tier" : 5,
"floor_segment" : "10mb"
},
"scheduler" : "concurrentmergescheduler"
}
}

Not sure whether this set up could be a contributing factor or
not. Nothing really jumps out at me in the logs. In fact, when i kick off
the optimize, I don't see any logging at all. Should I?

I'm running the following command: curl -XPOST
http://localhost:9200/index/_optimize

Thanks!

On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote:

Hi Elliott,

1500 segments per shard is certainly way too much, and it is not
normal that optimize doesn't manage to reduce the number of segments.

  • Is there anything suspicious in the logs?
  • Have you customized the merge policy or scheduler?[1]
  • Does the issue still reproduce if you restart your cluster?

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
e/current/index-modules-merge.html

On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any other thoughts on this? Would 1500 segments per shard be
significantly impacting performance? Have you guys noticed this behavior
elsewhere?

Thanks.

On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw wrote:

Adrian,

I ran the following command:

curl -XPUT http://localhost:9200/_settings -d
'{"indices.store.throttle.max_bytes_per_sec" : "10gb"}'

and received a { "acknowledged" : "true" } response. The logs
showed "cluster state updated".

I did have to close my index prior to changing the setting and
reopen afterward.

I've since began another optimize, but again it doesn't look
like much is happening. The optimize isn't returning and the total CPU
usage on every node is holding at about 2% of a single core. I would copy
a hot_threads stack trace, but I'm unfortunately on a closed network and
this isn't possible. I can tell you that refreshes of hot_threads show
vary little happening. The occasional [merge] thread (always in a
LinkedTransferQueue.awaitMatch() state) or [optimize] (doing
nothing on a waitForMerge() call) thread shows up, but it's always
consuming 0-1% CPU. It sure feels like something isn't right. Any
thoughts?

On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand <
adrien...@elasticsearch.com> wrote:

Did you see a message in the logs confirming that the setting
has been updated? It would be interesting to see the output of hot
threads[1] to see what your node is doing.

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/cluster-nodes-hot-threads.html

On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Yes. I have run max_num_segments=1 every time.

On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick <
michae...@serenesoftware.com> wrote:

Have you tried max_num_segments=1 on your optimize?

On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any thoughts on this? I've run optimize several more times,
and the number of segments falls each time, but I'm still over 1000
segments per shard. Has anyone else run into something similar?

On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott
Bradshaw wrote:

OK. Optimize finally returned, so I suppose something was
happening in the background, but I'm still seeing over 6500 segments. Even
after setting max_num_segments=5. Does this seem right? Queries are a
little faster (350-400ms) but still not great. Bigdesk is still showing a
fair amount of file IO.

On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott
Bradshaw wrote:

Hi All,

I've recently upgraded to Elasticsearch 1.1.0. I've got a
4 node cluster, each with 64G of ram, with 24G allocated to Elasticsearch
on each. I've batch loaded approximately 86 million documents into a
single index (4 shards) and have started benchmarking
cross_field/multi_match queries on them. The index has one replica and
takes up a total of 111G. I've run several batches of warming queries, but
queries are not as fast as I had hoped, approximately 400-500ms each.
Given that *top *(on Centos) shows 5-8 GB of free memory
on each server, I would assume that the entire index has been paged into
memory (I had worried about disk performance previously, as we are working
in a virtualized environment).

A stats query on the index in questions shows that the
index is composed of > 7000 segments. This seemed high to me, but maybe
it's appropriate. Regardless, I dispatched an optimize command, but I am
not seeing any progress and the command has not returned. Current merges
remains at zero, and the segment count is not changing. Checking out hot
threads in ElasticHQ, I initially saw an optimize call in the stack that
was blocked on a waitForMerge call. This however has disappeared, and I'm
seeing no evidence that the optimize is occuring.

Does any of this seem out of the norm or unusual? Has
anyone else had similar issues. This is the second time I have tried to
optimize an index since upgrading. I've gotten the same result both time.

Thanks in advance for any help/tips!

  • Elliott

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5391291f-5c5
e-4088-a1f2-93272beef0bb%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a
topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/
unsubscribe.
To unsubscribe from this group and all its topics, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUz
iGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS
QTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gma
il.comhttps://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to a topic
in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/
unsubscribe.
To unsubscribe from this group and all its topics, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP
jijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8742280e-922
f-4e91-bcb2-6096ca0165e6%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFQSMMGVkb%3DhQhTFTGJiAVcbeig%2B0rEfJutscoMyXtgtQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

23

Yes, that 's the exact same behavior we're seeing. Bulk indexing gradually
slows down to about 1/5th of its usual pace.

I noted something that might be another clue. During an optimize call:
every 30th minute there are some I/O operations, according to Marvel, and
the total segment count, which currently is around 280 000, gets lowered by
about 120-150 segments, then after another 30 minutes it's lowered again
with about the same amount of segments. This has been going on ever since
we first issued the optimize API call after the cluster restart. The
cluster is not serving any search requests or doing any indexing at the
time, so can't be that.

Robin

On Sunday, April 13, 2014 1:06:21 AM UTC+2, Jörg Prante wrote:

I see also exploding numbers in a quick local test on 1.1.0, thousands of
segments are created, but I'm not sure if this behavior is expected or not.
But that does not look like it should. It slows down bulk indexing
significantly.

Issuing an optimize just stalls...

curl '0:9200/test/_optimize?max_num_segments=1'

Does anyone see something similar?

Stack trace excerpt:

"elasticsearch[Doctor Strange][optimize][T#1]" #81 daemon prio=5
os_prio=31 tid=0x00007f85cdd9f800 nid=0xc507 in Object.wait()
[0x000000012c140000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4339)

  • locked <0x00000006cee0fad0> (a org.apache.lucene.index.IndexWriter)
    at org.apache.lucene.index.IndexWriter.waitForMerges(IndexWriter.java:2262)
  • locked <0x00000006cee0fad0> (a org.apache.lucene.index.IndexWriter)
    at
    org.elasticsearch.index.engine.internal.InternalEngine.optimize(InternalEngine.java:998)
    at
    org.elasticsearch.index.shard.service.InternalIndexShard.optimize(InternalIndexShard.java:599)
    at
    org.elasticsearch.action.admin.indices.optimize.TransportOptimizeAction.shardOperation(TransportOptimizeAction.java:115)
    at
    org.elasticsearch.action.admin.indices.optimize.TransportOptimizeAction.shardOperation(TransportOptimizeAction.java:49)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$2.run(TransportBroadcastOperationAction.java:226)
    at
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)

(node name "Doctor Strange" was not my selection btw)

Jörg

On Sat, Apr 12, 2014 at 8:09 PM, Thibaut Britz <t.b...@trendiction.com<javascript:>

wrote:

Hi,

We also have this issue in 1.0.2. Per index we have over 1000 segments.
Optimize doesn't reduce them.

Would love to see a fix for this.

Thanks,
Thibaut

On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett <nik...@gmail.com<javascript:>

wrote:

For reference I'm also on 1.1.0 but I'm not seeing more segments then I
expect. I see an average of ~28 per shard on an index I write to
constantly. I don't write all that quickly, < 50 updates a second.

Nik

On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand <
adrien...@elasticsearch.com <javascript:>> wrote:

I managed to reproduce the issue locally, I'm looking into it.

On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin <walr...@gmail.com<javascript:>

wrote:

Hello,

We are experiencing a related problem with 1.1.0. Segments do not seem
to merge as they should during indexing. The optimize API does practically
nothing in terms of lowering the segments count either. The problem
persists through a cluster restart. The vast amount of segments seem to be
greatly impact the performance of the cluster, in a very negative way.

We currently have 414 million documents across 3 nodes, each shard has
in average 1200 segments(!).

With 1.0.1 we had even more documents, ~650 million, without any
segment problems. Looking in Marvel we were hovering at around 30-40
segments per shard back then.

Best Regards,
Robin

On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:

Thanks for reporting this, the behavior is definitely unexpected.
I'll test _optimize on very large numbers of shards to see if I can
reproduce the issue.

On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Adrien,

Just an FYI, after resetting the cluster, things seem to have
improved. Optimize calls now lead to CPU/IO activity over their duration.
Max_num_segments=1 does not seem to be working for me on any given call, as
each call would only reduce the segment count by about 600-700. I ran 10
calls in sequence overnight, and actually got down to 4 segments (1/shard)!

I'm glad I got the index optimized, searches are literally 10-20
times faster without 1500/segments per shard to deal with. It's awesome.

That said, any thoughts on why the index wasn't merging on its own,
or why optimize was returning prematurely?

On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw
wrote:

Hi Adrien,

I kept the logs up over the last optimize call, and I did see an
exception. I Ctrl-C'd a curl optimize call before making another one, but
I don't think that that caused this exception. The error is essentially as
follows:

netty - Caught exception while handling client http traffic,
closing connection [id: 0x4d8f1a90, /127.0.0.1:33480 :> /
127.0.0.1:9200]

java.nio.channels.ClosedChannelException at AbstractNioWorker.
cleanUpWriteBuffer(AbstractNioWorker.java:433)
at AbstractNioWorker.writeFromUserCode
at NioServerSocketPipelineSink.handleAcceptedSocket
at NioServerSocketPipelineSink.eventSunk
at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDown
stream
at Channels.write
at OneToOneEncoder.doEncode
at OneToOneEncoder.handleDownstream
at DefaultChannelPipeline.sendDownstream
at DefaultChannelPipeline.sendDownstream
at Channels.write
at AbstractChannel.write
at NettyHttpChannel.sendResponse
at RestOptimizeAction$1.onResponse(95)
at RestOptimizeAction$1.onResponse(85)
at TransportBroadcastOperationAction$AsyncBroadcastAction.finishHim
at TransportBroadcastOperationAction$AsyncBroadcastAction.onOpe
ration
at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run

Sorry about the crappy stack trace. Still, looks like this might
point to a problem! The exception fired about an hour after I kicked off
the optimize. Any thoughts?

On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw
wrote:

Hi Adrien,

I did customize my merge policy, although I did so only because I
was so surprised by the number of segments left over after the load. I'm
pretty sure the optimize problem was happening before I made this change,
but either way here are my settings:

"index" : {
"merge" : {
"policy" : {
"max_merged_segment" : "20gb",
"segments_per_tier" : 5,
"floor_segment" : "10mb"
},
"scheduler" : "concurrentmergescheduler"
}
}

Not sure whether this set up could be a contributing factor or
not. Nothing really jumps out at me in the logs. In fact, when i kick off
the optimize, I don't see any logging at all. Should I?

I'm running the following command: curl -XPOST
http://localhost:9200/index/_optimize

Thanks!

On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote:

Hi Elliott,

1500 segments per shard is certainly way too much, and it is not
normal that optimize doesn't manage to reduce the number of segments.

  • Is there anything suspicious in the logs?
  • Have you customized the merge policy or scheduler?[1]
  • Does the issue still reproduce if you restart your cluster?

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
e/current/index-modules-merge.html

On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any other thoughts on this? Would 1500 segments per shard be
significantly impacting performance? Have you guys noticed this behavior
elsewhere?

Thanks.

On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw
wrote:

Adrian,

I ran the following command:

curl -XPUT http://localhost:9200/_settings -d
'{"indices.store.throttle.max_bytes_per_sec" : "10gb"}'

and received a { "acknowledged" : "true" } response. The logs
showed "cluster state updated".

I did have to close my index prior to changing the setting and
reopen afterward.

I've since began another optimize, but again it doesn't look
like much is happening. The optimize isn't returning and the total CPU
usage on every node is holding at about 2% of a single core. I would copy
a hot_threads stack trace, but I'm unfortunately on a closed network and
this isn't possible. I can tell you that refreshes of hot_threads show
vary little happening. The occasional [merge] thread (always in a
LinkedTransferQueue.awaitMatch() state) or [optimize] (doing
nothing on a waitForMerge() call) thread shows up, but it's always
consuming 0-1% CPU. It sure feels like something isn't right. Any
thoughts?

On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand <
adrien...@elasticsearch.com> wrote:

Did you see a message in the logs confirming that the setting
has been updated? It would be interesting to see the output of hot
threads[1] to see what your node is doing.

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/cluster-nodes-hot-threads.html

On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Yes. I have run max_num_segments=1 every time.

On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick <
michae...@serenesoftware.com> wrote:

Have you tried max_num_segments=1 on your optimize?

On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any thoughts on this? I've run optimize several more
times, and the number of segments falls each time, but I'm still over 1000
segments per shard. Has anyone else run into something similar?

On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott
Bradshaw wrote:

OK. Optimize finally returned, so I suppose something was
happening in the background, but I'm still seeing over 6500 segments. Even
after setting max_num_segments=5. Does this seem right? Queries are a
little faster (350-400ms) but still not great. Bigdesk is still showing a
fair amount of file IO.

On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott
Bradshaw wrote:

Hi All,

I've recently upgraded to Elasticsearch 1.1.0. I've got
a 4 node cluster, each with 64G of ram, with 24G allocated to Elasticsearch
on each. I've batch loaded approximately 86 million documents into a
single index (4 shards) and have started benchmarking
cross_field/multi_match queries on them. The index has one replica and
takes up a total of 111G. I've run several batches of warming queries, but
queries are not as fast as I had hoped, approximately 400-500ms each.
Given that *top *(on Centos) shows 5-8 GB of free memory
on each server, I would assume that the entire index has been paged into
memory (I had worried about disk performance previously, as we are working
in a virtualized environment).

A stats query on the index in questions shows that the
index is composed of > 7000 segments. This seemed high to me, but maybe
it's appropriate. Regardless, I dispatched an optimize command, but I am
not seeing any progress and the command has not returned. Current merges
remains at zero, and the segment count is not changing. Checking out hot
threads in ElasticHQ, I initially saw an optimize call in the stack that
was blocked on a waitForMerge call. This however has disappeared, and I'm
seeing no evidence that the optimize is occuring.

Does any of this seem out of the norm or unusual? Has
anyone else had similar issues. This is the second time I have tried to
optimize an index since upgrading. I've gotten the same result both time.

Thanks in advance for any help/tips!

  • Elliott

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5391291f-
5c5e-4088-a1f2-93272beef0bb%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a
topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/
unsubscribe.
To unsubscribe from this group and all its topics, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUz
iGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS
QTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gma
il.comhttps://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to a
topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/
unsubscribe.
To unsubscribe from this group and all its topics, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP
jijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8742280e-922
f-4e91-bcb2-6096ca0165e6%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/344b09db-
a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

24

We found the cause for this issue[1]. There was an issue with the way that
merges were picked, resulting in too many segments per shard. The issue is
mostly reproducible with high indexing rates (since it also implies a high
rate of segment creation) and merge throttling on (the default). We are
working on publishing a new bugfix release that includes a fix for this.

[1] Merges might not be picked up when they are ready · Issue #5779 · elastic/elasticsearch · GitHub
https://github.com/elasticsearch/elasticsearch/issues/5779

On Mon, Apr 14, 2014 at 8:56 AM, Robin Wallin walro467@gmail.com wrote:

Yes, that 's the exact same behavior we're seeing. Bulk indexing gradually
slows down to about 1/5th of its usual pace.

I noted something that might be another clue. During an optimize call:
every 30th minute there are some I/O operations, according to Marvel, and
the total segment count, which currently is around 280 000, gets lowered by
about 120-150 segments, then after another 30 minutes it's lowered again
with about the same amount of segments. This has been going on ever since
we first issued the optimize API call after the cluster restart. The
cluster is not serving any search requests or doing any indexing at the
time, so can't be that.

Robin

On Sunday, April 13, 2014 1:06:21 AM UTC+2, Jörg Prante wrote:

I see also exploding numbers in a quick local test on 1.1.0, thousands of
segments are created, but I'm not sure if this behavior is expected or not.
But that does not look like it should. It slows down bulk indexing
significantly.

Issuing an optimize just stalls...

curl '0:9200/test/_optimize?max_num_segments=1'

Does anyone see something similar?

Stack trace excerpt:

"elasticsearch[Doctor Strange][optimize][T#1]" #81 daemon prio=5
os_prio=31 tid=0x00007f85cdd9f800 nid=0xc507 in Object.wait()
[0x000000012c140000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4339)

  • locked <0x00000006cee0fad0> (a org.apache.lucene.index.IndexWriter)
    at org.apache.lucene.index.IndexWriter.waitForMerges(
    IndexWriter.java:2262)
  • locked <0x00000006cee0fad0> (a org.apache.lucene.index.IndexWriter)
    at org.elasticsearch.index.engine.internal.InternalEngine.optimize(
    InternalEngine.java:998)
    at org.elasticsearch.index.shard.service.InternalIndexShard.
    optimize(InternalIndexShard.java:599)
    at org.elasticsearch.action.admin.indices.optimize.
    TransportOptimizeAction.shardOperation(TransportOptimizeAction.java:115)
    at org.elasticsearch.action.admin.indices.optimize.
    TransportOptimizeAction.shardOperation(TransportOptimizeAction.java:49)
    at org.elasticsearch.action.support.broadcast.
    TransportBroadcastOperationAction$AsyncBroadcastAction$2.run(
    TransportBroadcastOperationAction.java:226)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(
    ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(
    ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)

(node name "Doctor Strange" was not my selection btw)

Jörg

On Sat, Apr 12, 2014 at 8:09 PM, Thibaut Britz t.b...@trendiction.comwrote:

Hi,

We also have this issue in 1.0.2. Per index we have over 1000 segments.
Optimize doesn't reduce them.

Would love to see a fix for this.

Thanks,
Thibaut

On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett nik...@gmail.comwrote:

For reference I'm also on 1.1.0 but I'm not seeing more segments then I
expect. I see an average of ~28 per shard on an index I write to
constantly. I don't write all that quickly, < 50 updates a second.

Nik

On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand <adrien...@elasticsearch.
com> wrote:

I managed to reproduce the issue locally, I'm looking into it.

On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin walr...@gmail.comwrote:

Hello,

We are experiencing a related problem with 1.1.0. Segments do not
seem to merge as they should during indexing. The optimize API does
practically nothing in terms of lowering the segments count either. The
problem persists through a cluster restart. The vast amount of segments
seem to be greatly impact the performance of the cluster, in a very
negative way.

We currently have 414 million documents across 3 nodes, each shard
has in average 1200 segments(!).

With 1.0.1 we had even more documents, ~650 million, without any
segment problems. Looking in Marvel we were hovering at around 30-40
segments per shard back then.

Best Regards,
Robin

On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:

Thanks for reporting this, the behavior is definitely unexpected.
I'll test _optimize on very large numbers of shards to see if I can
reproduce the issue.

On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Adrien,

Just an FYI, after resetting the cluster, things seem to have
improved. Optimize calls now lead to CPU/IO activity over their duration.
Max_num_segments=1 does not seem to be working for me on any given call, as
each call would only reduce the segment count by about 600-700. I ran 10
calls in sequence overnight, and actually got down to 4 segments (1/shard)!

I'm glad I got the index optimized, searches are literally 10-20
times faster without 1500/segments per shard to deal with. It's awesome.

That said, any thoughts on why the index wasn't merging on its own,
or why optimize was returning prematurely?

On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw
wrote:

Hi Adrien,

I kept the logs up over the last optimize call, and I did see an
exception. I Ctrl-C'd a curl optimize call before making another one, but
I don't think that that caused this exception. The error is essentially as
follows:

netty - Caught exception while handling client http traffic,
closing connection [id: 0x4d8f1a90, /127.0.0.1:33480 :> /
127.0.0.1:9200]

java.nio.channels.ClosedChannelException at AbstractNioWorker.
cleanUpWriteBuffer(AbstractNioWorker.java:433)
at AbstractNioWorker.writeFromUserCode
at NioServerSocketPipelineSink.handleAcceptedSocket
at NioServerSocketPipelineSink.eventSunk
at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDown
stream
at Channels.write
at OneToOneEncoder.doEncode
at OneToOneEncoder.handleDownstream
at DefaultChannelPipeline.sendDownstream
at DefaultChannelPipeline.sendDownstream
at Channels.write
at AbstractChannel.write
at NettyHttpChannel.sendResponse
at RestOptimizeAction$1.onResponse(95)
at RestOptimizeAction$1.onResponse(85)
at TransportBroadcastOperationAction$AsyncBroadcastAction.finis
hHim
at TransportBroadcastOperationAction$AsyncBroadcastAction.onOpe
ration
at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run

Sorry about the crappy stack trace. Still, looks like this might
point to a problem! The exception fired about an hour after I kicked off
the optimize. Any thoughts?

On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw
wrote:

Hi Adrien,

I did customize my merge policy, although I did so only because I
was so surprised by the number of segments left over after the load. I'm
pretty sure the optimize problem was happening before I made this change,
but either way here are my settings:

"index" : {
"merge" : {
"policy" : {
"max_merged_segment" : "20gb",
"segments_per_tier" : 5,
"floor_segment" : "10mb"
},
"scheduler" : "concurrentmergescheduler"
}
}

Not sure whether this set up could be a contributing factor or
not. Nothing really jumps out at me in the logs. In fact, when i kick off
the optimize, I don't see any logging at all. Should I?

I'm running the following command: curl -XPOST
http://localhost:9200/index/_optimize

Thanks!

On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote:

Hi Elliott,

1500 segments per shard is certainly way too much, and it is not
normal that optimize doesn't manage to reduce the number of segments.

  • Is there anything suspicious in the logs?
  • Have you customized the merge policy or scheduler?[1]
  • Does the issue still reproduce if you restart your cluster?

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
e/current/index-modules-merge.html

On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any other thoughts on this? Would 1500 segments per shard be
significantly impacting performance? Have you guys noticed this behavior
elsewhere?

Thanks.

On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw
wrote:

Adrian,

I ran the following command:

curl -XPUT http://localhost:9200/_settings -d
'{"indices.store.throttle.max_bytes_per_sec" : "10gb"}'

and received a { "acknowledged" : "true" } response. The logs
showed "cluster state updated".

I did have to close my index prior to changing the setting and
reopen afterward.

I've since began another optimize, but again it doesn't look
like much is happening. The optimize isn't returning and the total CPU
usage on every node is holding at about 2% of a single core. I would copy
a hot_threads stack trace, but I'm unfortunately on a closed network and
this isn't possible. I can tell you that refreshes of hot_threads show
vary little happening. The occasional [merge] thread (always in a
LinkedTransferQueue.awaitMatch() state) or [optimize] (doing
nothing on a waitForMerge() call) thread shows up, but it's always
consuming 0-1% CPU. It sure feels like something isn't right. Any
thoughts?

On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand <
adrien...@elasticsearch.com> wrote:

Did you see a message in the logs confirming that the setting
has been updated? It would be interesting to see the output of hot
threads[1] to see what your node is doing.

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/cluster-nodes-hot-threads.html

On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Yes. I have run max_num_segments=1 every time.

On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick <
michae...@serenesoftware.com> wrote:

Have you tried max_num_segments=1 on your optimize?

On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any thoughts on this? I've run optimize several more
times, and the number of segments falls each time, but I'm still over 1000
segments per shard. Has anyone else run into something similar?

On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott
Bradshaw wrote:

OK. Optimize finally returned, so I suppose something
was happening in the background, but I'm still seeing over 6500 segments.
Even after setting max_num_segments=5. Does this seem right? Queries are
a little faster (350-400ms) but still not great. Bigdesk is still showing
a fair amount of file IO.

On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott
Bradshaw wrote:

Hi All,

I've recently upgraded to Elasticsearch 1.1.0. I've got
a 4 node cluster, each with 64G of ram, with 24G allocated to Elasticsearch
on each. I've batch loaded approximately 86 million documents into a
single index (4 shards) and have started benchmarking
cross_field/multi_match queries on them. The index has one replica and
takes up a total of 111G. I've run several batches of warming queries, but
queries are not as fast as I had hoped, approximately 400-500ms each.
Given that *top *(on Centos) shows 5-8 GB of free
memory on each server, I would assume that the entire index has been paged
into memory (I had worried about disk performance previously, as we are
working in a virtualized environment).

A stats query on the index in questions shows that the
index is composed of > 7000 segments. This seemed high to me, but maybe
it's appropriate. Regardless, I dispatched an optimize command, but I am
not seeing any progress and the command has not returned. Current merges
remains at zero, and the segment count is not changing. Checking out hot
threads in ElasticHQ, I initially saw an optimize call in the stack that
was blocked on a waitForMerge call. This however has disappeared, and I'm
seeing no evidence that the optimize is occuring.

Does any of this seem out of the norm or unusual? Has
anyone else had similar issues. This is the second time I have tried to
optimize an index since upgrading. I've gotten the same result both time.

Thanks in advance for any help/tips!

  • Elliott

--
You received this message because you are subscribed to
the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to elasticsearc...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5391291f-
5c5e-4088-a1f2-93272beef0bb%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout
.

--
You received this message because you are subscribed to a
topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/
kqTRRADQBwc/unsubscribe.
To unsubscribe from this group and all its topics, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/
CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyO
DFA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoS
QTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gma
il.comhttps://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to a
topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/
unsubscribe.
To unsubscribe from this group and all its topics, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP
jijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.c
omhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8742280e-922
f-4e91-bcb2-6096ca0165e6%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d
8-4c2d-a917-dbf53eda03ce%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-
csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn
5UO0QNfeHHFGuYENwLD6w%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQs
UptE22kDKbB3NMdcA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5MhJ71sdYz7P4o-N-4E161o2ORMQ-vt%2BdmY%2B54FLOKLw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

25

Great! Is there anything we can do in the meantime in order to decrease the
segment count?

Also, I understand that the issue will solve the amount of segments created
when indexing, but what about optimize? Will it be implicitly fixed as well?

Thanks!

Robin

On Monday, April 14, 2014 10:57:09 AM UTC+2, Adrien Grand wrote:

We found the cause for this issue[1]. There was an issue with the way that
merges were picked, resulting in too many segments per shard. The issue is
mostly reproducible with high indexing rates (since it also implies a high
rate of segment creation) and merge throttling on (the default). We are
working on publishing a new bugfix release that includes a fix for this.

[1] Merges might not be picked up when they are ready · Issue #5779 · elastic/elasticsearch · GitHub
https://github.com/elasticsearch/elasticsearch/issues/5779

On Mon, Apr 14, 2014 at 8:56 AM, Robin Wallin <walr...@gmail.com<javascript:>

wrote:

Yes, that 's the exact same behavior we're seeing. Bulk indexing
gradually slows down to about 1/5th of its usual pace.

I noted something that might be another clue. During an optimize call:
every 30th minute there are some I/O operations, according to Marvel, and
the total segment count, which currently is around 280 000, gets lowered by
about 120-150 segments, then after another 30 minutes it's lowered again
with about the same amount of segments. This has been going on ever since
we first issued the optimize API call after the cluster restart. The
cluster is not serving any search requests or doing any indexing at the
time, so can't be that.

Robin

On Sunday, April 13, 2014 1:06:21 AM UTC+2, Jörg Prante wrote:

I see also exploding numbers in a quick local test on 1.1.0, thousands
of segments are created, but I'm not sure if this behavior is expected or
not. But that does not look like it should. It slows down bulk indexing
significantly.

Issuing an optimize just stalls...

curl '0:9200/test/_optimize?max_num_segments=1'

Does anyone see something similar?

Stack trace excerpt:

"elasticsearch[Doctor Strange][optimize][T#1]" #81 daemon prio=5
os_prio=31 tid=0x00007f85cdd9f800 nid=0xc507 in Object.wait()
[0x000000012c140000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4339)

  • locked <0x00000006cee0fad0> (a org.apache.lucene.index.IndexWriter)
    at org.apache.lucene.index.IndexWriter.waitForMerges(
    IndexWriter.java:2262)
  • locked <0x00000006cee0fad0> (a org.apache.lucene.index.IndexWriter)
    at org.elasticsearch.index.engine.internal.InternalEngine.optimize(
    InternalEngine.java:998)
    at org.elasticsearch.index.shard.service.InternalIndexShard.
    optimize(InternalIndexShard.java:599)
    at org.elasticsearch.action.admin.indices.optimize.
    TransportOptimizeAction.shardOperation(TransportOptimizeAction.java:115)
    at org.elasticsearch.action.admin.indices.optimize.
    TransportOptimizeAction.shardOperation(TransportOptimizeAction.java:49)
    at org.elasticsearch.action.support.broadcast.
    TransportBroadcastOperationAction$AsyncBroadcastAction$2.run(
    TransportBroadcastOperationAction.java:226)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(
    ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(
    ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)

(node name "Doctor Strange" was not my selection btw)

Jörg

On Sat, Apr 12, 2014 at 8:09 PM, Thibaut Britz t.b...@trendiction.comwrote:

Hi,

We also have this issue in 1.0.2. Per index we have over 1000 segments.
Optimize doesn't reduce them.

Would love to see a fix for this.

Thanks,
Thibaut

On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett nik...@gmail.comwrote:

For reference I'm also on 1.1.0 but I'm not seeing more segments then
I expect. I see an average of ~28 per shard on an index I write to
constantly. I don't write all that quickly, < 50 updates a second.

Nik

On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand <
adrien...@elasticsearch.com> wrote:

I managed to reproduce the issue locally, I'm looking into it.

On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin walr...@gmail.comwrote:

Hello,

We are experiencing a related problem with 1.1.0. Segments do not
seem to merge as they should during indexing. The optimize API does
practically nothing in terms of lowering the segments count either. The
problem persists through a cluster restart. The vast amount of segments
seem to be greatly impact the performance of the cluster, in a very
negative way.

We currently have 414 million documents across 3 nodes, each shard
has in average 1200 segments(!).

With 1.0.1 we had even more documents, ~650 million, without any
segment problems. Looking in Marvel we were hovering at around 30-40
segments per shard back then.

Best Regards,
Robin

On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:

Thanks for reporting this, the behavior is definitely unexpected.
I'll test _optimize on very large numbers of shards to see if I can
reproduce the issue.

On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Adrien,

Just an FYI, after resetting the cluster, things seem to have
improved. Optimize calls now lead to CPU/IO activity over their duration.
Max_num_segments=1 does not seem to be working for me on any given call, as
each call would only reduce the segment count by about 600-700. I ran 10
calls in sequence overnight, and actually got down to 4 segments (1/shard)!

I'm glad I got the index optimized, searches are literally 10-20
times faster without 1500/segments per shard to deal with. It's awesome.

That said, any thoughts on why the index wasn't merging on its
own, or why optimize was returning prematurely?

On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw
wrote:

Hi Adrien,

I kept the logs up over the last optimize call, and I did see an
exception. I Ctrl-C'd a curl optimize call before making another one, but
I don't think that that caused this exception. The error is essentially as
follows:

netty - Caught exception while handling client http traffic,
closing connection [id: 0x4d8f1a90, /127.0.0.1:33480 :> /
127.0.0.1:9200]

java.nio.channels.ClosedChannelException at AbstractNioWorker.
cleanUpWriteBuffer(AbstractNioWorker.java:433)
at AbstractNioWorker.writeFromUserCode
at NioServerSocketPipelineSink.handleAcceptedSocket
at NioServerSocketPipelineSink.eventSunk
at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDown
stream
at Channels.write
at OneToOneEncoder.doEncode
at OneToOneEncoder.handleDownstream
at DefaultChannelPipeline.sendDownstream
at DefaultChannelPipeline.sendDownstream
at Channels.write
at AbstractChannel.write
at NettyHttpChannel.sendResponse
at RestOptimizeAction$1.onResponse(95)
at RestOptimizeAction$1.onResponse(85)
at TransportBroadcastOperationAction$AsyncBroadcastAction.finis
hHim
at TransportBroadcastOperationAction$AsyncBroadcastAction.onOpe
ration
at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run

Sorry about the crappy stack trace. Still, looks like this might
point to a problem! The exception fired about an hour after I kicked off
the optimize. Any thoughts?

On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw
wrote:

Hi Adrien,

I did customize my merge policy, although I did so only because
I was so surprised by the number of segments left over after the load. I'm
pretty sure the optimize problem was happening before I made this change,
but either way here are my settings:

"index" : {
"merge" : {
"policy" : {
"max_merged_segment" : "20gb",
"segments_per_tier" : 5,
"floor_segment" : "10mb"
},
"scheduler" : "concurrentmergescheduler"
}
}

Not sure whether this set up could be a contributing factor or
not. Nothing really jumps out at me in the logs. In fact, when i kick off
the optimize, I don't see any logging at all. Should I?

I'm running the following command: curl -XPOST
http://localhost:9200/index/_optimize

Thanks!

On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand wrote:

Hi Elliott,

1500 segments per shard is certainly way too much, and it is
not normal that optimize doesn't manage to reduce the number of segments.

  • Is there anything suspicious in the logs?
  • Have you customized the merge policy or scheduler?[1]
  • Does the issue still reproduce if you restart your cluster?

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/index-modules-merge.html

On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any other thoughts on this? Would 1500 segments per shard be
significantly impacting performance? Have you guys noticed this behavior
elsewhere?

Thanks.

On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw
wrote:

Adrian,

I ran the following command:

curl -XPUT http://localhost:9200/_settings -d
'{"indices.store.throttle.max_bytes_per_sec" : "10gb"}'

and received a { "acknowledged" : "true" } response. The
logs showed "cluster state updated".

I did have to close my index prior to changing the setting
and reopen afterward.

I've since began another optimize, but again it doesn't look
like much is happening. The optimize isn't returning and the total CPU
usage on every node is holding at about 2% of a single core. I would copy
a hot_threads stack trace, but I'm unfortunately on a closed network and
this isn't possible. I can tell you that refreshes of hot_threads show
vary little happening. The occasional [merge] thread (always in a
LinkedTransferQueue.awaitMatch() state) or [optimize] (doing
nothing on a waitForMerge() call) thread shows up, but it's always
consuming 0-1% CPU. It sure feels like something isn't right. Any
thoughts?

On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand <
adrien...@elasticsearch.com> wrote:

Did you see a message in the logs confirming that the
setting has been updated? It would be interesting to see the output of hot
threads[1] to see what your node is doing.

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/cluster-nodes-hot-threads.html

On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Yes. I have run max_num_segments=1 every time.

On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick <
michae...@serenesoftware.com> wrote:

Have you tried max_num_segments=1 on your optimize?

On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any thoughts on this? I've run optimize several more
times, and the number of segments falls each time, but I'm still over 1000
segments per shard. Has anyone else run into something similar?

On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott
Bradshaw wrote:

OK. Optimize finally returned, so I suppose something
was happening in the background, but I'm still seeing over 6500 segments.
Even after setting max_num_segments=5. Does this seem right? Queries are
a little faster (350-400ms) but still not great. Bigdesk is still showing
a fair amount of file IO.

On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott
Bradshaw wrote:

Hi All,

I've recently upgraded to Elasticsearch 1.1.0. I've
got a 4 node cluster, each with 64G of ram, with 24G allocated to
Elasticsearch on each. I've batch loaded approximately 86 million
documents into a single index (4 shards) and have started benchmarking
cross_field/multi_match queries on them. The index has one replica and
takes up a total of 111G. I've run several batches of warming queries, but
queries are not as fast as I had hoped, approximately 400-500ms each.
Given that *top *(on Centos) shows 5-8 GB of free
memory on each server, I would assume that the entire index has been paged
into memory (I had worried about disk performance previously, as we are
working in a virtualized environment).

A stats query on the index in questions shows that the
index is composed of > 7000 segments. This seemed high to me, but maybe
it's appropriate. Regardless, I dispatched an optimize command, but I am
not seeing any progress and the command has not returned. Current merges
remains at zero, and the segment count is not changing. Checking out hot
threads in ElasticHQ, I initially saw an optimize call in the stack that
was blocked on a waitForMerge call. This however has disappeared, and I'm
seeing no evidence that the optimize is occuring.

Does any of this seem out of the norm or unusual? Has
anyone else had similar issues. This is the second time I have tried to
optimize an index since upgrading. I've gotten the same result both time.

Thanks in advance for any help/tips!

  • Elliott

--
You received this message because you are subscribed to
the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to elasticsearc...@googlegroups.c
om.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5391291f-
5c5e-4088-a1f2-93272beef0bb%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/op
tout.

--
You received this message because you are subscribed to a
topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/
kqTRRADQBwc/unsubscribe.
To unsubscribe from this group and all its topics, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/
CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyO
DFA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout
.

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCt%
2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3D
emc0JTouT9%2BBeUk_A%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to a
topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/
unsubscribe.
To unsubscribe from this group and all its topics, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrP
jijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.c
omhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8742280e-922
f-4e91-bcb2-6096ca0165e6%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d
8-4c2d-a917-dbf53eda03ce%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eda52a43-
94ec-4574-b989-32727cf3cfe4%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-
csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn
5UO0QNfeHHFGuYENwLD6w%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQs
UptE22kDKbB3NMdcA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

26

Setting index.merge.scheduler to concurrentmergescheduler[1] (beware
that this is not a live setting so it requires closing the index) should
help mitigate the problem.

Indeed, this fix also implicitely fixes the optimize issue as well: I just
ran again my script that reproduced the optimize issue that Elliott
described and it doesn't reproduce anymore with the fix.

[1]

On Mon, Apr 14, 2014 at 11:20 AM, Robin Wallin walro467@gmail.com wrote:

Great! Is there anything we can do in the meantime in order to decrease
the segment count?

Also, I understand that the issue will solve the amount of segments
created when indexing, but what about optimize? Will it be implicitly fixed
as well?

Thanks!

Robin

On Monday, April 14, 2014 10:57:09 AM UTC+2, Adrien Grand wrote:

We found the cause for this issue[1]. There was an issue with the way
that merges were picked, resulting in too many segments per shard. The
issue is mostly reproducible with high indexing rates (since it also
implies a high rate of segment creation) and merge throttling on (the
default). We are working on publishing a new bugfix release that includes a
fix for this.

[1] Merges might not be picked up when they are ready · Issue #5779 · elastic/elasticsearch · GitHub
https://github.com/elasticsearch/elasticsearch/issues/5779

On Mon, Apr 14, 2014 at 8:56 AM, Robin Wallin walr...@gmail.com wrote:

Yes, that 's the exact same behavior we're seeing. Bulk indexing
gradually slows down to about 1/5th of its usual pace.

I noted something that might be another clue. During an optimize call:
every 30th minute there are some I/O operations, according to Marvel, and
the total segment count, which currently is around 280 000, gets lowered by
about 120-150 segments, then after another 30 minutes it's lowered again
with about the same amount of segments. This has been going on ever since
we first issued the optimize API call after the cluster restart. The
cluster is not serving any search requests or doing any indexing at the
time, so can't be that.

Robin

On Sunday, April 13, 2014 1:06:21 AM UTC+2, Jörg Prante wrote:

I see also exploding numbers in a quick local test on 1.1.0, thousands
of segments are created, but I'm not sure if this behavior is expected or
not. But that does not look like it should. It slows down bulk indexing
significantly.

Issuing an optimize just stalls...

curl '0:9200/test/_optimize?max_num_segments=1'

Does anyone see something similar?

Stack trace excerpt:

"elasticsearch[Doctor Strange][optimize][T#1]" #81 daemon prio=5
os_prio=31 tid=0x00007f85cdd9f800 nid=0xc507 in Object.wait()
[0x000000012c140000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4339)

  • locked <0x00000006cee0fad0> (a org.apache.lucene.index.IndexWriter)
    at org.apache.lucene.index.IndexWriter.waitForMerges(IndexWrite
    r.java:2262)
  • locked <0x00000006cee0fad0> (a org.apache.lucene.index.IndexWriter)
    at org.elasticsearch.index.engine.internal.InternalEngine.optimize(
    InternalEngine.java:998)
    at org.elasticsearch.index.shard.service.InternalIndexShard.opt
    imize(InternalIndexShard.java:599)
    at org.elasticsearch.action.admin.indices.optimize.TransportOpt
    imizeAction.shardOperation(TransportOptimizeAction.java:115)
    at org.elasticsearch.action.admin.indices.optimize.TransportOpt
    imizeAction.shardOperation(TransportOptimizeAction.java:49)
    at org.elasticsearch.action.support.broadcast.TransportBroadcas
    tOperationAction$AsyncBroadcastAction$2.run(Tra
    nsportBroadcastOperationAction.java:226)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
    Executor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
    lExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)

(node name "Doctor Strange" was not my selection btw)

Jörg

On Sat, Apr 12, 2014 at 8:09 PM, Thibaut Britz t.b...@trendiction.comwrote:

Hi,

We also have this issue in 1.0.2. Per index we have over 1000
segments. Optimize doesn't reduce them.

Would love to see a fix for this.

Thanks,
Thibaut

On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett nik...@gmail.comwrote:

For reference I'm also on 1.1.0 but I'm not seeing more segments then
I expect. I see an average of ~28 per shard on an index I write to
constantly. I don't write all that quickly, < 50 updates a second.

Nik

On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand <
adrien...@elasticsearch.com> wrote:

I managed to reproduce the issue locally, I'm looking into it.

On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin walr...@gmail.comwrote:

Hello,

We are experiencing a related problem with 1.1.0. Segments do not
seem to merge as they should during indexing. The optimize API does
practically nothing in terms of lowering the segments count either. The
problem persists through a cluster restart. The vast amount of segments
seem to be greatly impact the performance of the cluster, in a very
negative way.

We currently have 414 million documents across 3 nodes, each shard
has in average 1200 segments(!).

With 1.0.1 we had even more documents, ~650 million, without any
segment problems. Looking in Marvel we were hovering at around 30-40
segments per shard back then.

Best Regards,
Robin

On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:

Thanks for reporting this, the behavior is definitely unexpected.
I'll test _optimize on very large numbers of shards to see if I can
reproduce the issue.

On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Adrien,

Just an FYI, after resetting the cluster, things seem to have
improved. Optimize calls now lead to CPU/IO activity over their duration.
Max_num_segments=1 does not seem to be working for me on any given call, as
each call would only reduce the segment count by about 600-700. I ran 10
calls in sequence overnight, and actually got down to 4 segments (1/shard)!

I'm glad I got the index optimized, searches are literally 10-20
times faster without 1500/segments per shard to deal with. It's awesome.

That said, any thoughts on why the index wasn't merging on its
own, or why optimize was returning prematurely?

On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw
wrote:

Hi Adrien,

I kept the logs up over the last optimize call, and I did see an
exception. I Ctrl-C'd a curl optimize call before making another one, but
I don't think that that caused this exception. The error is essentially as
follows:

netty - Caught exception while handling client http traffic,
closing connection [id: 0x4d8f1a90, /127.0.0.1:33480 :> /
127.0.0.1:9200]

java.nio.channels.ClosedChannelException at AbstractNioWorker.
cleanUpWriteBuffer(AbstractNioWorker.java:433)
at AbstractNioWorker.writeFromUserCode
at NioServerSocketPipelineSink.handleAcceptedSocket
at NioServerSocketPipelineSink.eventSunk
at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDown
stream
at Channels.write
at OneToOneEncoder.doEncode
at OneToOneEncoder.handleDownstream
at DefaultChannelPipeline.sendDownstream
at DefaultChannelPipeline.sendDownstream
at Channels.write
at AbstractChannel.write
at NettyHttpChannel.sendResponse
at RestOptimizeAction$1.onResponse(95)
at RestOptimizeAction$1.onResponse(85)
at TransportBroadcastOperationAction$AsyncBroadcastAction.finis
hHim
at TransportBroadcastOperationAction$AsyncBroadcastAction.onOpe
ration
at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run

Sorry about the crappy stack trace. Still, looks like this
might point to a problem! The exception fired about an hour after I kicked
off the optimize. Any thoughts?

On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw
wrote:

Hi Adrien,

I did customize my merge policy, although I did so only because
I was so surprised by the number of segments left over after the load. I'm
pretty sure the optimize problem was happening before I made this change,
but either way here are my settings:

"index" : {
"merge" : {
"policy" : {
"max_merged_segment" : "20gb",
"segments_per_tier" : 5,
"floor_segment" : "10mb"
},
"scheduler" : "concurrentmergescheduler"
}
}

Not sure whether this set up could be a contributing factor or
not. Nothing really jumps out at me in the logs. In fact, when i kick off
the optimize, I don't see any logging at all. Should I?

I'm running the following command: curl -XPOST
http://localhost:9200/index/_optimize

Thanks!

On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand
wrote:

Hi Elliott,

1500 segments per shard is certainly way too much, and it is
not normal that optimize doesn't manage to reduce the number of segments.

  • Is there anything suspicious in the logs?
  • Have you customized the merge policy or scheduler?[1]
  • Does the issue still reproduce if you restart your cluster?

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/index-modules-merge.html

On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any other thoughts on this? Would 1500 segments per shard be
significantly impacting performance? Have you guys noticed this behavior
elsewhere?

Thanks.

On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw
wrote:

Adrian,

I ran the following command:

curl -XPUT http://localhost:9200/_settings -d
'{"indices.store.throttle.max_bytes_per_sec" : "10gb"}'

and received a { "acknowledged" : "true" } response. The
logs showed "cluster state updated".

I did have to close my index prior to changing the setting
and reopen afterward.

I've since began another optimize, but again it doesn't look
like much is happening. The optimize isn't returning and the total CPU
usage on every node is holding at about 2% of a single core. I would copy
a hot_threads stack trace, but I'm unfortunately on a closed network and
this isn't possible. I can tell you that refreshes of hot_threads show
vary little happening. The occasional [merge] thread (always in a
LinkedTransferQueue.awaitMatch() state) or [optimize]
(doing nothing on a waitForMerge() call) thread shows up, but it's always
consuming 0-1% CPU. It sure feels like something isn't right. Any
thoughts?

On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand <
adrien...@elasticsearch.com> wrote:

Did you see a message in the logs confirming that the
setting has been updated? It would be interesting to see the output of hot
threads[1] to see what your node is doing.

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/cluster-nodes-hot-threads.html

On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Yes. I have run max_num_segments=1 every time.

On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick <
michae...@serenesoftware.com> wrote:

Have you tried max_num_segments=1 on your optimize?

On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any thoughts on this? I've run optimize several more
times, and the number of segments falls each time, but I'm still over 1000
segments per shard. Has anyone else run into something similar?

On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott
Bradshaw wrote:

OK. Optimize finally returned, so I suppose something
was happening in the background, but I'm still seeing over 6500 segments.
Even after setting max_num_segments=5. Does this seem right? Queries are
a little faster (350-400ms) but still not great. Bigdesk is still showing
a fair amount of file IO.

On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott
Bradshaw wrote:

Hi All,

I've recently upgraded to Elasticsearch 1.1.0. I've
got a 4 node cluster, each with 64G of ram, with 24G allocated to
Elasticsearch on each. I've batch loaded approximately 86 million
documents into a single index (4 shards) and have started benchmarking
cross_field/multi_match queries on them. The index has one replica and
takes up a total of 111G. I've run several batches of warming queries, but
queries are not as fast as I had hoped, approximately 400-500ms each.
Given that *top *(on Centos) shows 5-8 GB of free
memory on each server, I would assume that the entire index has been paged
into memory (I had worried about disk performance previously, as we are
working in a virtualized environment).

A stats query on the index in questions shows that the
index is composed of > 7000 segments. This seemed high to me, but maybe
it's appropriate. Regardless, I dispatched an optimize command, but I am
not seeing any progress and the command has not returned. Current merges
remains at zero, and the segment count is not changing. Checking out hot
threads in ElasticHQ, I initially saw an optimize call in the stack that
was blocked on a waitForMerge call. This however has disappeared, and I'm
seeing no evidence that the optimize is occuring.

Does any of this seem out of the norm or unusual? Has
anyone else had similar issues. This is the second time I have tried to
optimize an index since upgrading. I've gotten the same result both time.

Thanks in advance for any help/tips!

  • Elliott

--
You received this message because you are subscribed to
the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to elasticsearc...@googlegroups.c
om.
To view this discussion on the web visit
https://groups.google.com/d/ms
gid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/op
tout.

--
You received this message because you are subscribed to a
topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/
kqTRRADQBwc/unsubscribe.
To unsubscribe from this group and all its topics, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/
CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyO
DFA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/op
tout.

--
You received this message because you are subscribed to
the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to elasticsearc...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCt%
2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3D
emc0JTouT9%2BBeUk_A%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout
.

--
Adrien Grand

--
You received this message because you are subscribed to a
topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/
kqTRRADQBwc/unsubscribe.
To unsubscribe from this group and all its topics, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/
CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%
2BKDsg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8742280e-922
f-4e91-bcb2-6096ca0165e6%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d
8-4c2d-a917-dbf53eda03ce%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eda52a43-94e
c-4574-b989-32727cf3cfe4%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNr
g1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFG
uYENwLD6w%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kD
KbB3NMdcA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

27

Adrian,

Changing to 'concurrentmergescheduler' did improve my situation, but not
completely. Optimize would run steadily after this setting was changed,
but would still time out after approximately an hour. I had to run it
sequentially about a dozen times to get it to fully optimize over night.

Thanks for digging into this for us!

  • Elliott

On Mon, Apr 14, 2014 at 5:43 AM, Adrien Grand <
adrien.grand@elasticsearch.com> wrote:

Setting index.merge.scheduler to concurrentmergescheduler[1] (beware
that this is not a live setting so it requires closing the index) should
help mitigate the problem.

Indeed, this fix also implicitely fixes the optimize issue as well: I just
ran again my script that reproduced the optimize issue that Elliott
described and it doesn't reproduce anymore with the fix.

[1]
Elasticsearch Platform — Find real-time answers at scale | Elastic

On Mon, Apr 14, 2014 at 11:20 AM, Robin Wallin walro467@gmail.com wrote:

Great! Is there anything we can do in the meantime in order to decrease
the segment count?

Also, I understand that the issue will solve the amount of segments
created when indexing, but what about optimize? Will it be implicitly fixed
as well?

Thanks!

Robin

On Monday, April 14, 2014 10:57:09 AM UTC+2, Adrien Grand wrote:

We found the cause for this issue[1]. There was an issue with the way
that merges were picked, resulting in too many segments per shard. The
issue is mostly reproducible with high indexing rates (since it also
implies a high rate of segment creation) and merge throttling on (the
default). We are working on publishing a new bugfix release that includes a
fix for this.

[1] Merges might not be picked up when they are ready · Issue #5779 · elastic/elasticsearch · GitHub
https://github.com/elasticsearch/elasticsearch/issues/5779

On Mon, Apr 14, 2014 at 8:56 AM, Robin Wallin walr...@gmail.com wrote:

Yes, that 's the exact same behavior we're seeing. Bulk indexing
gradually slows down to about 1/5th of its usual pace.

I noted something that might be another clue. During an optimize call:
every 30th minute there are some I/O operations, according to Marvel, and
the total segment count, which currently is around 280 000, gets lowered by
about 120-150 segments, then after another 30 minutes it's lowered again
with about the same amount of segments. This has been going on ever since
we first issued the optimize API call after the cluster restart. The
cluster is not serving any search requests or doing any indexing at the
time, so can't be that.

Robin

On Sunday, April 13, 2014 1:06:21 AM UTC+2, Jörg Prante wrote:

I see also exploding numbers in a quick local test on 1.1.0, thousands
of segments are created, but I'm not sure if this behavior is expected or
not. But that does not look like it should. It slows down bulk indexing
significantly.

Issuing an optimize just stalls...

curl '0:9200/test/_optimize?max_num_segments=1'

Does anyone see something similar?

Stack trace excerpt:

"elasticsearch[Doctor Strange][optimize][T#1]" #81 daemon prio=5
os_prio=31 tid=0x00007f85cdd9f800 nid=0xc507 in Object.wait()
[0x000000012c140000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4339)

  • locked <0x00000006cee0fad0> (a org.apache.lucene.index.IndexWriter)
    at org.apache.lucene.index.IndexWriter.waitForMerges(IndexWrite
    r.java:2262)
  • locked <0x00000006cee0fad0> (a org.apache.lucene.index.IndexWriter)
    at org.elasticsearch.index.engine.internal.InternalEngine.optimize(
    InternalEngine.java:998)
    at org.elasticsearch.index.shard.service.InternalIndexShard.opt
    imize(InternalIndexShard.java:599)
    at org.elasticsearch.action.admin.indices.optimize.TransportOpt
    imizeAction.shardOperation(TransportOptimizeAction.java:115)
    at org.elasticsearch.action.admin.indices.optimize.TransportOpt
    imizeAction.shardOperation(TransportOptimizeAction.java:49)
    at org.elasticsearch.action.support.broadcast.TransportBroadcas
    tOperationAction$AsyncBroadcastAction$2.run(Tra
    nsportBroadcastOperationAction.java:226)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
    Executor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
    lExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)

(node name "Doctor Strange" was not my selection btw)

Jörg

On Sat, Apr 12, 2014 at 8:09 PM, Thibaut Britz <t.b...@trendiction.com

wrote:

Hi,

We also have this issue in 1.0.2. Per index we have over 1000
segments. Optimize doesn't reduce them.

Would love to see a fix for this.

Thanks,
Thibaut

On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett nik...@gmail.comwrote:

For reference I'm also on 1.1.0 but I'm not seeing more segments
then I expect. I see an average of ~28 per shard on an index I write to
constantly. I don't write all that quickly, < 50 updates a second.

Nik

On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand <
adrien...@elasticsearch.com> wrote:

I managed to reproduce the issue locally, I'm looking into it.

On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin walr...@gmail.comwrote:

Hello,

We are experiencing a related problem with 1.1.0. Segments do not
seem to merge as they should during indexing. The optimize API does
practically nothing in terms of lowering the segments count either. The
problem persists through a cluster restart. The vast amount of segments
seem to be greatly impact the performance of the cluster, in a very
negative way.

We currently have 414 million documents across 3 nodes, each shard
has in average 1200 segments(!).

With 1.0.1 we had even more documents, ~650 million, without any
segment problems. Looking in Marvel we were hovering at around 30-40
segments per shard back then.

Best Regards,
Robin

On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:

Thanks for reporting this, the behavior is definitely unexpected.
I'll test _optimize on very large numbers of shards to see if I can
reproduce the issue.

On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Adrien,

Just an FYI, after resetting the cluster, things seem to have
improved. Optimize calls now lead to CPU/IO activity over their duration.
Max_num_segments=1 does not seem to be working for me on any given call, as
each call would only reduce the segment count by about 600-700. I ran 10
calls in sequence overnight, and actually got down to 4 segments (1/shard)!

I'm glad I got the index optimized, searches are literally 10-20
times faster without 1500/segments per shard to deal with. It's awesome.

That said, any thoughts on why the index wasn't merging on its
own, or why optimize was returning prematurely?

On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw
wrote:

Hi Adrien,

I kept the logs up over the last optimize call, and I did see
an exception. I Ctrl-C'd a curl optimize call before making another one,
but I don't think that that caused this exception. The error is
essentially as follows:

netty - Caught exception while handling client http traffic,
closing connection [id: 0x4d8f1a90, /127.0.0.1:33480 :> /
127.0.0.1:9200]

java.nio.channels.ClosedChannelException at AbstractNioWorker.
cleanUpWriteBuffer(AbstractNioWorker.java:433)
at AbstractNioWorker.writeFromUserCode
at NioServerSocketPipelineSink.handleAcceptedSocket
at NioServerSocketPipelineSink.eventSunk
at DefaultChannelPipeline$DefaultChannelhandlerContext.sendDown
stream
at Channels.write
at OneToOneEncoder.doEncode
at OneToOneEncoder.handleDownstream
at DefaultChannelPipeline.sendDownstream
at DefaultChannelPipeline.sendDownstream
at Channels.write
at AbstractChannel.write
at NettyHttpChannel.sendResponse
at RestOptimizeAction$1.onResponse(95)
at RestOptimizeAction$1.onResponse(85)
at TransportBroadcastOperationAction$AsyncBroadcastAction.finis
hHim
at TransportBroadcastOperationAction$AsyncBroadcastAction.onOpe
ration
at TransportBroadcastOperationAction$AsyncBroadcastAction$2.run

Sorry about the crappy stack trace. Still, looks like this
might point to a problem! The exception fired about an hour after I kicked
off the optimize. Any thoughts?

On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott Bradshaw
wrote:

Hi Adrien,

I did customize my merge policy, although I did so only
because I was so surprised by the number of segments left over after the
load. I'm pretty sure the optimize problem was happening before I made
this change, but either way here are my settings:

"index" : {
"merge" : {
"policy" : {
"max_merged_segment" : "20gb",
"segments_per_tier" : 5,
"floor_segment" : "10mb"
},
"scheduler" : "concurrentmergescheduler"
}
}

Not sure whether this set up could be a contributing factor or
not. Nothing really jumps out at me in the logs. In fact, when i kick off
the optimize, I don't see any logging at all. Should I?

I'm running the following command: curl -XPOST
http://localhost:9200/index/_optimize

Thanks!

On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand
wrote:

Hi Elliott,

1500 segments per shard is certainly way too much, and it is
not normal that optimize doesn't manage to reduce the number of segments.

  • Is there anything suspicious in the logs?
  • Have you customized the merge policy or scheduler?[1]
  • Does the issue still reproduce if you restart your cluster?

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/index-modules-merge.html

On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any other thoughts on this? Would 1500 segments per shard
be significantly impacting performance? Have you guys noticed this
behavior elsewhere?

Thanks.

On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw
wrote:

Adrian,

I ran the following command:

curl -XPUT http://localhost:9200/_settings -d
'{"indices.store.throttle.max_bytes_per_sec" : "10gb"}'

and received a { "acknowledged" : "true" } response. The
logs showed "cluster state updated".

I did have to close my index prior to changing the setting
and reopen afterward.

I've since began another optimize, but again it doesn't
look like much is happening. The optimize isn't returning and the total
CPU usage on every node is holding at about 2% of a single core. I would
copy a hot_threads stack trace, but I'm unfortunately on a closed network
and this isn't possible. I can tell you that refreshes of hot_threads show
vary little happening. The occasional [merge] thread (always in a
LinkedTransferQueue.awaitMatch() state) or [optimize]
(doing nothing on a waitForMerge() call) thread shows up, but it's always
consuming 0-1% CPU. It sure feels like something isn't right. Any
thoughts?

On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand <
adrien...@elasticsearch.com> wrote:

Did you see a message in the logs confirming that the
setting has been updated? It would be interesting to see the output of hot
threads[1] to see what your node is doing.

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/cluster-nodes-hot-threads.html

On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Yes. I have run max_num_segments=1 every time.

On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick <
michae...@serenesoftware.com> wrote:

Have you tried max_num_segments=1 on your optimize?

On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any thoughts on this? I've run optimize several more
times, and the number of segments falls each time, but I'm still over 1000
segments per shard. Has anyone else run into something similar?

On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott
Bradshaw wrote:

OK. Optimize finally returned, so I suppose something
was happening in the background, but I'm still seeing over 6500 segments.
Even after setting max_num_segments=5. Does this seem right? Queries are
a little faster (350-400ms) but still not great. Bigdesk is still showing
a fair amount of file IO.

On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott
Bradshaw wrote:

Hi All,

I've recently upgraded to Elasticsearch 1.1.0. I've
got a 4 node cluster, each with 64G of ram, with 24G allocated to
Elasticsearch on each. I've batch loaded approximately 86 million
documents into a single index (4 shards) and have started benchmarking
cross_field/multi_match queries on them. The index has one replica and
takes up a total of 111G. I've run several batches of warming queries, but
queries are not as fast as I had hoped, approximately 400-500ms each.
Given that *top *(on Centos) shows 5-8 GB of free
memory on each server, I would assume that the entire index has been paged
into memory (I had worried about disk performance previously, as we are
working in a virtualized environment).

A stats query on the index in questions shows that
the index is composed of > 7000 segments. This seemed high to me, but
maybe it's appropriate. Regardless, I dispatched an optimize command, but
I am not seeing any progress and the command has not returned. Current
merges remains at zero, and the segment count is not changing. Checking
out hot threads in ElasticHQ, I initially saw an optimize call in the stack
that was blocked on a waitForMerge call. This however has disappeared, and
I'm seeing no evidence that the optimize is occuring.

Does any of this seem out of the norm or unusual?
Has anyone else had similar issues. This is the second time I have tried
to optimize an index since upgrading. I've gotten the same result both
time.

Thanks in advance for any help/tips!

  • Elliott

--
You received this message because you are subscribed to
the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving
emails from it, send an email to elasticsearc...@
googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/ms
gid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/op
tout.

--
You received this message because you are subscribed to
a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/
kqTRRADQBwc/unsubscribe.
To unsubscribe from this group and all its topics, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/
CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyO
DFA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/op
tout.

--
You received this message because you are subscribed to
the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to elasticsearc...@googlegroups.c
om.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCt%
2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3D
emc0JTouT9%2BBeUk_A%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/op
tout.

--
Adrien Grand

--
You received this message because you are subscribed to a
topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/
kqTRRADQBwc/unsubscribe.
To unsubscribe from this group and all its topics, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/
CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%
2BKDsg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout
.

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8742280e-922
f-4e91-bcb2-6096ca0165e6%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d
8-4c2d-a917-dbf53eda03ce%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eda52a43-94e
c-4574-b989-32727cf3cfe4%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNr
g1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93p
FKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kD
KbB3NMdcA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFsEv1%2B5SZOMc9XOVnPnt_FA_wY%3DpGbxdbQJrqLtautfAg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

28

With this configuration

I could tune the segment merging quite well, on 1.1.0.

Not sure what changed in the default merge settings. I suspect the threads
of the segment merging could not keep up the segment creation while bulk
feed under default settings. So I tuned the segments_per_tier to 4, the
merge thread settings, and adjusted some other settings.

After bulk indexing 12,5m docs in 45 minutes, I could run

curl '0:9200/harvard2/_optimize?max_num_segments=1'

as usually - it returned after few minutes (with peaks of 400MB/sec
read/write on SSD) with a single segment of 10g size.

Helpful is also enabling DEBUG level logging with

index.merge.scheduler: DEBUG
index.merge.policy: DEBUG

Jörg

On Mon, Apr 14, 2014 at 1:45 PM, Elliott Bradshaw ebradshaw1@gmail.comwrote:

Adrian,

Changing to 'concurrentmergescheduler' did improve my situation, but not
completely. Optimize would run steadily after this setting was changed,
but would still time out after approximately an hour. I had to run it
sequentially about a dozen times to get it to fully optimize over night.

Thanks for digging into this for us!

  • Elliott

On Mon, Apr 14, 2014 at 5:43 AM, Adrien Grand <
adrien.grand@elasticsearch.com> wrote:

Setting index.merge.scheduler to concurrentmergescheduler[1] (beware
that this is not a live setting so it requires closing the index) should
help mitigate the problem.

Indeed, this fix also implicitely fixes the optimize issue as well: I
just ran again my script that reproduced the optimize issue that Elliott
described and it doesn't reproduce anymore with the fix.

[1]
Elasticsearch Platform — Find real-time answers at scale | Elastic

On Mon, Apr 14, 2014 at 11:20 AM, Robin Wallin walro467@gmail.comwrote:

Great! Is there anything we can do in the meantime in order to decrease
the segment count?

Also, I understand that the issue will solve the amount of segments
created when indexing, but what about optimize? Will it be implicitly fixed
as well?

Thanks!

Robin

On Monday, April 14, 2014 10:57:09 AM UTC+2, Adrien Grand wrote:

We found the cause for this issue[1]. There was an issue with the way
that merges were picked, resulting in too many segments per shard. The
issue is mostly reproducible with high indexing rates (since it also
implies a high rate of segment creation) and merge throttling on (the
default). We are working on publishing a new bugfix release that includes a
fix for this.

[1] Merges might not be picked up when they are ready · Issue #5779 · elastic/elasticsearch · GitHub
https://github.com/elasticsearch/elasticsearch/issues/5779

On Mon, Apr 14, 2014 at 8:56 AM, Robin Wallin walr...@gmail.comwrote:

Yes, that 's the exact same behavior we're seeing. Bulk indexing
gradually slows down to about 1/5th of its usual pace.

I noted something that might be another clue. During an optimize call:
every 30th minute there are some I/O operations, according to Marvel, and
the total segment count, which currently is around 280 000, gets lowered by
about 120-150 segments, then after another 30 minutes it's lowered again
with about the same amount of segments. This has been going on ever since
we first issued the optimize API call after the cluster restart. The
cluster is not serving any search requests or doing any indexing at the
time, so can't be that.

Robin

On Sunday, April 13, 2014 1:06:21 AM UTC+2, Jörg Prante wrote:

I see also exploding numbers in a quick local test on 1.1.0,
thousands of segments are created, but I'm not sure if this behavior is
expected or not. But that does not look like it should. It slows down bulk
indexing significantly.

Issuing an optimize just stalls...

curl '0:9200/test/_optimize?max_num_segments=1'

Does anyone see something similar?

Stack trace excerpt:

"elasticsearch[Doctor Strange][optimize][T#1]" #81 daemon prio=5
os_prio=31 tid=0x00007f85cdd9f800 nid=0xc507 in Object.wait()
[0x000000012c140000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4339)

  • locked <0x00000006cee0fad0> (a org.apache.lucene.index.IndexW
    riter)
    at org.apache.lucene.index.IndexWriter.waitForMerges(IndexWrite
    r.java:2262)
  • locked <0x00000006cee0fad0> (a org.apache.lucene.index.IndexW
    riter)
    at org.elasticsearch.index.engine.internal.InternalEngine.optimize(
    InternalEngine.java:998)
    at org.elasticsearch.index.shard.service.InternalIndexShard.opt
    imize(InternalIndexShard.java:599)
    at org.elasticsearch.action.admin.indices.optimize.TransportOpt
    imizeAction.shardOperation(TransportOptimizeAction.java:115)
    at org.elasticsearch.action.admin.indices.optimize.TransportOpt
    imizeAction.shardOperation(TransportOptimizeAction.java:49)
    at org.elasticsearch.action.support.broadcast.TransportBroadcas
    tOperationAction$AsyncBroadcastAction$2.run(Tra
    nsportBroadcastOperationAction.java:226)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
    Executor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
    lExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)

(node name "Doctor Strange" was not my selection btw)

Jörg

On Sat, Apr 12, 2014 at 8:09 PM, Thibaut Britz <
t.b...@trendiction.com> wrote:

Hi,

We also have this issue in 1.0.2. Per index we have over 1000
segments. Optimize doesn't reduce them.

Would love to see a fix for this.

Thanks,
Thibaut

On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett nik...@gmail.comwrote:

For reference I'm also on 1.1.0 but I'm not seeing more segments
then I expect. I see an average of ~28 per shard on an index I write to
constantly. I don't write all that quickly, < 50 updates a second.

Nik

On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand <
adrien...@elasticsearch.com> wrote:

I managed to reproduce the issue locally, I'm looking into it.

On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin walr...@gmail.comwrote:

Hello,

We are experiencing a related problem with 1.1.0. Segments do not
seem to merge as they should during indexing. The optimize API does
practically nothing in terms of lowering the segments count either. The
problem persists through a cluster restart. The vast amount of segments
seem to be greatly impact the performance of the cluster, in a very
negative way.

We currently have 414 million documents across 3 nodes, each
shard has in average 1200 segments(!).

With 1.0.1 we had even more documents, ~650 million, without any
segment problems. Looking in Marvel we were hovering at around 30-40
segments per shard back then.

Best Regards,
Robin

On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:

Thanks for reporting this, the behavior is definitely
unexpected. I'll test _optimize on very large numbers of shards to see if I
can reproduce the issue.

On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Adrien,

Just an FYI, after resetting the cluster, things seem to have
improved. Optimize calls now lead to CPU/IO activity over their duration.
Max_num_segments=1 does not seem to be working for me on any given call, as
each call would only reduce the segment count by about 600-700. I ran 10
calls in sequence overnight, and actually got down to 4 segments (1/shard)!

I'm glad I got the index optimized, searches are literally
10-20 times faster without 1500/segments per shard to deal with. It's
awesome.

That said, any thoughts on why the index wasn't merging on its
own, or why optimize was returning prematurely?

On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott Bradshaw
wrote:

Hi Adrien,

I kept the logs up over the last optimize call, and I did see
an exception. I Ctrl-C'd a curl optimize call before making another one,
but I don't think that that caused this exception. The error is
essentially as follows:

netty - Caught exception while handling client http traffic,
closing connection [id: 0x4d8f1a90, /127.0.0.1:33480 :> /
127.0.0.1:9200]

java.nio.channels.ClosedChannelException at AbstractNioWorker.
cleanUpWriteBuffer(AbstractNioWorker.java:433)
at AbstractNioWorker.writeFromUserCode
at NioServerSocketPipelineSink.handleAcceptedSocket
at NioServerSocketPipelineSink.eventSunk
at DefaultChannelPipeline$DefaultChannelhandlerContext.
sendDownstream
at Channels.write
at OneToOneEncoder.doEncode
at OneToOneEncoder.handleDownstream
at DefaultChannelPipeline.sendDownstream
at DefaultChannelPipeline.sendDownstream
at Channels.write
at AbstractChannel.write
at NettyHttpChannel.sendResponse
at RestOptimizeAction$1.onResponse(95)
at RestOptimizeAction$1.onResponse(85)
at TransportBroadcastOperationAction$AsyncBroadcastAction.
finishHim
at TransportBroadcastOperationAction$AsyncBroadcastAction.
onOperation
at TransportBroadcastOperationAct
ion$AsyncBroadcastAction$2.run

Sorry about the crappy stack trace. Still, looks like this
might point to a problem! The exception fired about an hour after I kicked
off the optimize. Any thoughts?

On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott
Bradshaw wrote:

Hi Adrien,

I did customize my merge policy, although I did so only
because I was so surprised by the number of segments left over after the
load. I'm pretty sure the optimize problem was happening before I made
this change, but either way here are my settings:

"index" : {
"merge" : {
"policy" : {
"max_merged_segment" : "20gb",
"segments_per_tier" : 5,
"floor_segment" : "10mb"
},
"scheduler" : "concurrentmergescheduler"
}
}

Not sure whether this set up could be a contributing factor
or not. Nothing really jumps out at me in the logs. In fact, when i kick
off the optimize, I don't see any logging at all. Should I?

I'm running the following command: curl -XPOST
http://localhost:9200/index/_optimize

Thanks!

On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand
wrote:

Hi Elliott,

1500 segments per shard is certainly way too much, and it is
not normal that optimize doesn't manage to reduce the number of segments.

  • Is there anything suspicious in the logs?
  • Have you customized the merge policy or scheduler?[1]
  • Does the issue still reproduce if you restart your
    cluster?

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/index-modules-merge.html

On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any other thoughts on this? Would 1500 segments per shard
be significantly impacting performance? Have you guys noticed this
behavior elsewhere?

Thanks.

On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott Bradshaw
wrote:

Adrian,

I ran the following command:

curl -XPUT http://localhost:9200/_settings -d
'{"indices.store.throttle.max_bytes_per_sec" : "10gb"}'

and received a { "acknowledged" : "true" } response. The
logs showed "cluster state updated".

I did have to close my index prior to changing the setting
and reopen afterward.

I've since began another optimize, but again it doesn't
look like much is happening. The optimize isn't returning and the total
CPU usage on every node is holding at about 2% of a single core. I would
copy a hot_threads stack trace, but I'm unfortunately on a closed network
and this isn't possible. I can tell you that refreshes of hot_threads show
vary little happening. The occasional [merge] thread (always in a
LinkedTransferQueue.awaitMatch() state) or [optimize]
(doing nothing on a waitForMerge() call) thread shows up, but it's always
consuming 0-1% CPU. It sure feels like something isn't right. Any
thoughts?

On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand <
adrien...@elasticsearch.com> wrote:

Did you see a message in the logs confirming that the
setting has been updated? It would be interesting to see the output of hot
threads[1] to see what your node is doing.

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/cluster-nodes-hot-threads.html

On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Yes. I have run max_num_segments=1 every time.

On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick <
michae...@serenesoftware.com> wrote:

Have you tried max_num_segments=1 on your optimize?

On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any thoughts on this? I've run optimize several more
times, and the number of segments falls each time, but I'm still over 1000
segments per shard. Has anyone else run into something similar?

On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott
Bradshaw wrote:

OK. Optimize finally returned, so I suppose
something was happening in the background, but I'm still seeing over 6500
segments. Even after setting max_num_segments=5. Does this seem right?
Queries are a little faster (350-400ms) but still not great. Bigdesk is
still showing a fair amount of file IO.

On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott
Bradshaw wrote:

Hi All,

I've recently upgraded to Elasticsearch 1.1.0. I've
got a 4 node cluster, each with 64G of ram, with 24G allocated to
Elasticsearch on each. I've batch loaded approximately 86 million
documents into a single index (4 shards) and have started benchmarking
cross_field/multi_match queries on them. The index has one replica and
takes up a total of 111G. I've run several batches of warming queries, but
queries are not as fast as I had hoped, approximately 400-500ms each.
Given that *top *(on Centos) shows 5-8 GB of free
memory on each server, I would assume that the entire index has been paged
into memory (I had worried about disk performance previously, as we are
working in a virtualized environment).

A stats query on the index in questions shows that
the index is composed of > 7000 segments. This seemed high to me, but
maybe it's appropriate. Regardless, I dispatched an optimize command, but
I am not seeing any progress and the command has not returned. Current
merges remains at zero, and the segment count is not changing. Checking
out hot threads in ElasticHQ, I initially saw an optimize call in the stack
that was blocked on a waitForMerge call. This however has disappeared, and
I'm seeing no evidence that the optimize is occuring.

Does any of this seem out of the norm or unusual?
Has anyone else had similar issues. This is the second time I have tried
to optimize an index since upgrading. I've gotten the same result both
time.

Thanks in advance for any help/tips!

  • Elliott

--
You received this message because you are subscribed
to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving
emails from it, send an email to elasticsearc...@
googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/ms
gid/elasticsearch/5391291f-5c5
e-4088-a1f2-93272beef0bb%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/op
tout.

--
You received this message because you are subscribed to
a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/
kqTRRADQBwc/unsubscribe.
To unsubscribe from this group and all its topics, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/
CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyO
DFA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/op
tout.

--
You received this message because you are subscribed to
the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to elasticsearc...@googlegroups.c
om.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCt%
2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3D
emc0JTouT9%2BBeUk_A%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/op
tout.

--
Adrien Grand

--
You received this message because you are subscribed to a
topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/
kqTRRADQBwc/unsubscribe.
To unsubscribe from this group and all its topics, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/
CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%
2BKDsg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/op
tout.

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8742280e-
922f-4e91-bcb2-6096ca0165e6%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d
8-4c2d-a917-dbf53eda03ce%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eda52a43-94e
c-4574-b989-32727cf3cfe4%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNr
g1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93p
FKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKY
tkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFsEv1%2B5SZOMc9XOVnPnt_FA_wY%3DpGbxdbQJrqLtautfAg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFsEv1%2B5SZOMc9XOVnPnt_FA_wY%3DpGbxdbQJrqLtautfAg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH5ZfojVsoNEdUbKnxhSFsh_teS5hKxbGdHgg4%3D0XT8zQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

29

Jorg,

I've just had a chance to get back to this and your configuration is
working great. Merges are being dispatched as expected, and the total
segment count during loading is staying in the hundreds. Nice stuff!

On Monday, April 14, 2014 1:37:05 PM UTC-4, Jörg Prante wrote:

With this configuration

Elasticsearch configuration for high sustainable bulk feed · GitHub

I could tune the segment merging quite well, on 1.1.0.

Not sure what changed in the default merge settings. I suspect the threads
of the segment merging could not keep up the segment creation while bulk
feed under default settings. So I tuned the segments_per_tier to 4, the
merge thread settings, and adjusted some other settings.

After bulk indexing 12,5m docs in 45 minutes, I could run

curl '0:9200/harvard2/_optimize?max_num_segments=1'

as usually - it returned after few minutes (with peaks of 400MB/sec
read/write on SSD) with a single segment of 10g size.

Helpful is also enabling DEBUG level logging with

index.merge.scheduler: DEBUG
index.merge.policy: DEBUG

Jörg

On Mon, Apr 14, 2014 at 1:45 PM, Elliott Bradshaw <ebrad...@gmail.com<javascript:>

wrote:

Adrian,

Changing to 'concurrentmergescheduler' did improve my situation, but not
completely. Optimize would run steadily after this setting was changed,
but would still time out after approximately an hour. I had to run it
sequentially about a dozen times to get it to fully optimize over night.

Thanks for digging into this for us!

  • Elliott

On Mon, Apr 14, 2014 at 5:43 AM, Adrien Grand <
adrien...@elasticsearch.com <javascript:>> wrote:

Setting index.merge.scheduler to concurrentmergescheduler[1] (beware
that this is not a live setting so it requires closing the index) should
help mitigate the problem.

Indeed, this fix also implicitely fixes the optimize issue as well: I
just ran again my script that reproduced the optimize issue that Elliott
described and it doesn't reproduce anymore with the fix.

[1]
Elasticsearch Platform — Find real-time answers at scale | Elastic

On Mon, Apr 14, 2014 at 11:20 AM, Robin Wallin <walr...@gmail.com<javascript:>

wrote:

Great! Is there anything we can do in the meantime in order to decrease
the segment count?

Also, I understand that the issue will solve the amount of segments
created when indexing, but what about optimize? Will it be implicitly fixed
as well?

Thanks!

Robin

On Monday, April 14, 2014 10:57:09 AM UTC+2, Adrien Grand wrote:

We found the cause for this issue[1]. There was an issue with the way
that merges were picked, resulting in too many segments per shard. The
issue is mostly reproducible with high indexing rates (since it also
implies a high rate of segment creation) and merge throttling on (the
default). We are working on publishing a new bugfix release that includes a
fix for this.

[1] Merges might not be picked up when they are ready · Issue #5779 · elastic/elasticsearch · GitHub
https://github.com/elasticsearch/elasticsearch/issues/5779

On Mon, Apr 14, 2014 at 8:56 AM, Robin Wallin walr...@gmail.comwrote:

Yes, that 's the exact same behavior we're seeing. Bulk indexing
gradually slows down to about 1/5th of its usual pace.

I noted something that might be another clue. During an optimize
call: every 30th minute there are some I/O operations, according to Marvel,
and the total segment count, which currently is around 280 000, gets
lowered by about 120-150 segments, then after another 30 minutes it's
lowered again with about the same amount of segments. This has been going
on ever since we first issued the optimize API call after the cluster
restart. The cluster is not serving any search requests or doing any
indexing at the time, so can't be that.

Robin

On Sunday, April 13, 2014 1:06:21 AM UTC+2, Jörg Prante wrote:

I see also exploding numbers in a quick local test on 1.1.0,
thousands of segments are created, but I'm not sure if this behavior is
expected or not. But that does not look like it should. It slows down bulk
indexing significantly.

Issuing an optimize just stalls...

curl '0:9200/test/_optimize?max_num_segments=1'

Does anyone see something similar?

Stack trace excerpt:

"elasticsearch[Doctor Strange][optimize][T#1]" #81 daemon prio=5
os_prio=31 tid=0x00007f85cdd9f800 nid=0xc507 in Object.wait()
[0x000000012c140000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4339)

  • locked <0x00000006cee0fad0> (a org.apache.lucene.index.IndexW
    riter)
    at org.apache.lucene.index.IndexWriter.waitForMerges(IndexWrite
    r.java:2262)
  • locked <0x00000006cee0fad0> (a org.apache.lucene.index.IndexW
    riter)
    at org.elasticsearch.index.engine.internal.InternalEngine.optimize(
    InternalEngine.java:998)
    at org.elasticsearch.index.shard.service.InternalIndexShard.opt
    imize(InternalIndexShard.java:599)
    at org.elasticsearch.action.admin.indices.optimize.TransportOpt
    imizeAction.shardOperation(TransportOptimizeAction.java:115)
    at org.elasticsearch.action.admin.indices.optimize.TransportOpt
    imizeAction.shardOperation(TransportOptimizeAction.java:49)
    at org.elasticsearch.action.support.broadcast.TransportBroadcas
    tOperationAction$AsyncBroadcastAction$2.run(Tra
    nsportBroadcastOperationAction.java:226)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
    Executor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
    lExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)

(node name "Doctor Strange" was not my selection btw)

Jörg

On Sat, Apr 12, 2014 at 8:09 PM, Thibaut Britz <
t.b...@trendiction.com> wrote:

Hi,

We also have this issue in 1.0.2. Per index we have over 1000
segments. Optimize doesn't reduce them.

Would love to see a fix for this.

Thanks,
Thibaut

On Fri, Apr 11, 2014 at 2:22 PM, Nikolas Everett nik...@gmail.comwrote:

For reference I'm also on 1.1.0 but I'm not seeing more segments
then I expect. I see an average of ~28 per shard on an index I write to
constantly. I don't write all that quickly, < 50 updates a second.

Nik

On Fri, Apr 11, 2014 at 5:46 AM, Adrien Grand <
adrien...@elasticsearch.com> wrote:

I managed to reproduce the issue locally, I'm looking into it.

On Fri, Apr 11, 2014 at 9:45 AM, Robin Wallin walr...@gmail.comwrote:

Hello,

We are experiencing a related problem with 1.1.0. Segments do
not seem to merge as they should during indexing. The optimize API does
practically nothing in terms of lowering the segments count either. The
problem persists through a cluster restart. The vast amount of segments
seem to be greatly impact the performance of the cluster, in a very
negative way.

We currently have 414 million documents across 3 nodes, each
shard has in average 1200 segments(!).

With 1.0.1 we had even more documents, ~650 million, without any
segment problems. Looking in Marvel we were hovering at around 30-40
segments per shard back then.

Best Regards,
Robin

On Friday, April 11, 2014 1:35:42 AM UTC+2, Adrien Grand wrote:

Thanks for reporting this, the behavior is definitely
unexpected. I'll test _optimize on very large numbers of shards to see if I
can reproduce the issue.

On Thu, Apr 10, 2014 at 2:10 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Adrien,

Just an FYI, after resetting the cluster, things seem to have
improved. Optimize calls now lead to CPU/IO activity over their duration.
Max_num_segments=1 does not seem to be working for me on any given call, as
each call would only reduce the segment count by about 600-700. I ran 10
calls in sequence overnight, and actually got down to 4 segments (1/shard)!

I'm glad I got the index optimized, searches are literally
10-20 times faster without 1500/segments per shard to deal with. It's
awesome.

That said, any thoughts on why the index wasn't merging on its
own, or why optimize was returning prematurely?

On Wednesday, April 9, 2014 11:10:56 AM UTC-4, Elliott
Bradshaw wrote:

Hi Adrien,

I kept the logs up over the last optimize call, and I did see
an exception. I Ctrl-C'd a curl optimize call before making another one,
but I don't think that that caused this exception. The error is
essentially as follows:

netty - Caught exception while handling client http traffic,
closing connection [id: 0x4d8f1a90, /127.0.0.1:33480 :> /
127.0.0.1:9200]

java.nio.channels.ClosedChannelException at
AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:
433)
at AbstractNioWorker.writeFromUserCode
at NioServerSocketPipelineSink.handleAcceptedSocket
at NioServerSocketPipelineSink.eventSunk
at DefaultChannelPipeline$DefaultChannelhandlerContext.
sendDownstream
at Channels.write
at OneToOneEncoder.doEncode
at OneToOneEncoder.handleDownstream
at DefaultChannelPipeline.sendDownstream
at DefaultChannelPipeline.sendDownstream
at Channels.write
at AbstractChannel.write
at NettyHttpChannel.sendResponse
at RestOptimizeAction$1.onResponse(95)
at RestOptimizeAction$1.onResponse(85)
at TransportBroadcastOperationAction$AsyncBroadcastAction.
finishHim
at TransportBroadcastOperationAction$AsyncBroadcastAction.
onOperation
at TransportBroadcastOperationAct
ion$AsyncBroadcastAction$2.run

Sorry about the crappy stack trace. Still, looks like this
might point to a problem! The exception fired about an hour after I kicked
off the optimize. Any thoughts?

On Wednesday, April 9, 2014 10:06:57 AM UTC-4, Elliott
Bradshaw wrote:

Hi Adrien,

I did customize my merge policy, although I did so only
because I was so surprised by the number of segments left over after the
load. I'm pretty sure the optimize problem was happening before I made
this change, but either way here are my settings:

"index" : {
"merge" : {
"policy" : {
"max_merged_segment" : "20gb",
"segments_per_tier" : 5,
"floor_segment" : "10mb"
},
"scheduler" : "concurrentmergescheduler"
}
}

Not sure whether this set up could be a contributing factor
or not. Nothing really jumps out at me in the logs. In fact, when i kick
off the optimize, I don't see any logging at all. Should I?

I'm running the following command: curl -XPOST
http://localhost:9200/index/_optimize

Thanks!

On Wednesday, April 9, 2014 8:56:35 AM UTC-4, Adrien Grand
wrote:

Hi Elliott,

1500 segments per shard is certainly way too much, and it
is not normal that optimize doesn't manage to reduce the number of segments.

  • Is there anything suspicious in the logs?
  • Have you customized the merge policy or scheduler?[1]
  • Does the issue still reproduce if you restart your
    cluster?

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/index-modules-merge.html

On Wed, Apr 9, 2014 at 2:38 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any other thoughts on this? Would 1500 segments per shard
be significantly impacting performance? Have you guys noticed this
behavior elsewhere?

Thanks.

On Monday, April 7, 2014 8:56:38 AM UTC-4, Elliott
Bradshaw wrote:

Adrian,

I ran the following command:

curl -XPUT http://localhost:9200/_settings -d
'{"indices.store.throttle.max_bytes_per_sec" : "10gb"}'

and received a { "acknowledged" : "true" } response. The
logs showed "cluster state updated".

I did have to close my index prior to changing the
setting and reopen afterward.

I've since began another optimize, but again it doesn't
look like much is happening. The optimize isn't returning and the total
CPU usage on every node is holding at about 2% of a single core. I would
copy a hot_threads stack trace, but I'm unfortunately on a closed network
and this isn't possible. I can tell you that refreshes of hot_threads show
vary little happening. The occasional [merge] thread (always in a
LinkedTransferQueue.awaitMatch() state) or [optimize]
(doing nothing on a waitForMerge() call) thread shows up, but it's always
consuming 0-1% CPU. It sure feels like something isn't right. Any
thoughts?

On Fri, Apr 4, 2014 at 3:24 PM, Adrien Grand <
adrien...@elasticsearch.com> wrote:

Did you see a message in the logs confirming that the
setting has been updated? It would be interesting to see the output of hot
threads[1] to see what your node is doing.

[1] Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/cluster-nodes-hot-threads.html

On Fri, Apr 4, 2014 at 7:18 PM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Yes. I have run max_num_segments=1 every time.

On Fri, Apr 4, 2014 at 12:26 PM, Michael Sick <
michae...@serenesoftware.com> wrote:

Have you tried max_num_segments=1 on your optimize?

On Fri, Apr 4, 2014 at 11:27 AM, Elliott Bradshaw <
ebrad...@gmail.com> wrote:

Any thoughts on this? I've run optimize several more
times, and the number of segments falls each time, but I'm still over 1000
segments per shard. Has anyone else run into something similar?

On Thursday, April 3, 2014 11:21:29 AM UTC-4, Elliott
Bradshaw wrote:

OK. Optimize finally returned, so I suppose
something was happening in the background, but I'm still seeing over 6500
segments. Even after setting max_num_segments=5. Does this seem right?
Queries are a little faster (350-400ms) but still not great. Bigdesk is
still showing a fair amount of file IO.

On Thursday, April 3, 2014 8:47:32 AM UTC-4, Elliott
Bradshaw wrote:

Hi All,

I've recently upgraded to Elasticsearch 1.1.0.
I've got a 4 node cluster, each with 64G of ram, with 24G allocated to
Elasticsearch on each. I've batch loaded approximately 86 million
documents into a single index (4 shards) and have started benchmarking
cross_field/multi_match queries on them. The index has one replica and
takes up a total of 111G. I've run several batches of warming queries, but
queries are not as fast as I had hoped, approximately 400-500ms each.
Given that *top *(on Centos) shows 5-8 GB of free
memory on each server, I would assume that the entire index has been paged
into memory (I had worried about disk performance previously, as we are
working in a virtualized environment).

A stats query on the index in questions shows that
the index is composed of > 7000 segments. This seemed high to me, but
maybe it's appropriate. Regardless, I dispatched an optimize command, but
I am not seeing any progress and the command has not returned. Current
merges remains at zero, and the segment count is not changing. Checking
out hot threads in ElasticHQ, I initially saw an optimize call in the stack
that was blocked on a waitForMerge call. This however has disappeared, and
I'm seeing no evidence that the optimize is occuring.

Does any of this seem out of the norm or unusual?
Has anyone else had similar issues. This is the second time I have tried
to optimize an index since upgrading. I've gotten the same result both
time.

Thanks in advance for any help/tips!

  • Elliott

--
You received this message because you are subscribed
to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving
emails from it, send an email to elasticsearc...@
googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/ms
gid/elasticsearch/5391291f-5c5
e-4088-a1f2-93272beef0bb%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/5391291f-5c5e-4088-a1f2-93272beef0bb%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/
optout.

--
You received this message because you are subscribed
to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/
kqTRRADQBwc/unsubscribe.
To unsubscribe from this group and all its topics,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/
CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyO
DFA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAP8axnD7BUziGct2%3Db%3DfupaKYFnA5fR2TBsxHoURJumHSyODFA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/op
tout.

--
You received this message because you are subscribed to
the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving
emails from it, send an email to elasticsearc...@
googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCt%
2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3D
emc0JTouT9%2BBeUk_A%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFvoSQTvv%2B6G%3D3GOX27AuYdEwLiW%3Demc0JTouT9%2BBeUk_A%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/op
tout.

--
Adrien Grand

--
You received this message because you are subscribed to
a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/
kqTRRADQBwc/unsubscribe.
To unsubscribe from this group and all its topics, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/
CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%
2BKDsg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6sQrPjijV86nYGoGTAQ%3D3cO_pgyYE6%2B3sGjJPr8%2BKDsg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/op
tout.

--
You received this message because you are subscribed to
the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to elasticsearc...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8742280e-
922f-4e91-bcb2-6096ca0165e6%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8742280e-922f-4e91-bcb2-6096ca0165e6%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout
.

--
Adrien Grand

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/344b09db-a2d
8-4c2d-a917-dbf53eda03ce%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/344b09db-a2d8-4c2d-a917-dbf53eda03ce%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eda52a43-94e
c-4574-b989-32727cf3cfe4%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/eda52a43-94ec-4574-b989-32727cf3cfe4%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNr
g1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7mYNrg1vVauWN8CyD-csXPqtdPad%3DC0QFiTyYOzsU2Bg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93p
FKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAPmjWd1f93pFKC0RqT40uFOg6Zwkn5UO0QNfeHHFGuYENwLD6w%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKY
tkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAE_AicT4JKYtkiRmkpsBSGTMMUjrq7WQsUptE22kDKbB3NMdcA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%
40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/f58c30f8-7123-4974-88a2-d27ef969d57e%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/a3dcbc41-f0be-4f77-9c28-5b073238ef41%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kqTRRADQBwc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5FwLCy4d%2BikrKeGd0yrfvhsCHz-wvM6dpnDZxnN353Pg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFsEv1%2B5SZOMc9XOVnPnt_FA_wY%3DpGbxdbQJrqLtautfAg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAGCt%2BFsEv1%2B5SZOMc9XOVnPnt_FA_wY%3DpGbxdbQJrqLtautfAg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ba4373b4-c895-4339-845d-488907d1c824%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.