i have the index of the documents that is constantly updated. The perfomance of the queries i'm running, are depending from amount segment. I can cron merging. I can set merge policy figures to minimum values like 2, but there are still too many segments and queries are still slow.
How many indices/shards/segments/docs do you have?
1 index 6 shards(and 1 replica) 10M docs
And how many segments do you have per shard?
with lowest params (segmentsPerTier = 2) it is ~ 20 segments per shard after indexing all docs. without this settings it was even more
20 segments per shard should be totally fine... I don't think you can have less without killing your index/update rate. I suspect you are running costly multi-term queries like wildcard, fuzzy, range or prefix queries?
i run, more like this. and with multiple segments it is 3x slower than with one. Should i cron merge with number of segment=1. also update rate is not high it is like 2 docs per sec avg, but actually updates not that frequent. so i can have lower indexing/updating speed.
Not unless your indices are static. Otherwise every optimize call will rewrite the entire index, which would incur a lot of I/O.
Hmm this is the first time I hear about this. Out of curiosity could you try to share the output of the nodes hot threads API while you have some MLT queries running in a loop?
what if i set another value for number of segments of the force merge? will it rewrite entire index of just segments that will be merged? By the way it is some testing results not actual production. Regarding hot threads i need to reindex all docs to have more segments cause now i have only one segment per shard (after force merge).
I/O activity will be a little lighter but this might still aften rewrite significant parts of the index.
and can it cause any problems?
Yes. This I/O that is required for merging will compete with search operations, and your disks will have shorter lives.
but if it will speed up my queries the rest of the time, it works for me. if force_merge will take much less time than time interval i have to run it to keep suitable amount of segments.
hot threads for 1 segment per shard
::: {my-es-1}{PF1j_1AKS-apK6WHe9g7Tw}{10.240.0.15}{my-es-1/10.240.0.15:9300}
Hot threads at 2016-04-11T16:55:09.817Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
14.0% (69.8ms out of 500ms) cpu usage by thread 'elasticsearch[es-1][search][T#25]'
6/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
13.7% (68.7ms out of 500ms) cpu usage by thread 'elasticsearch[es-1][search][T#10]'
2/10 snapshots sharing following 34 elements
sun.nio.ch.NativeThread.current(Native Method)
sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:737)
sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:180)
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
org.apache.lucene.codecs.lucene50.ForUtil.readBlock(ForUtil.java:187)
org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.refillDocs(Lucene50PostingsReader.java:368)
org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockDocsEnum.advance(Lucene50PostingsReader.java:452)
org.apache.lucene.search.TermScorer.advance(TermScorer.java:85)
org.apache.lucene.search.MinShouldMatchSumScorer.advanceTail(MinShouldMatchSumScorer.java:204)
org.apache.lucene.search.MinShouldMatchSumScorer.advanceTail(MinShouldMatchSumScorer.java:214)
org.apache.lucene.search.MinShouldMatchSumScorer.doNext(MinShouldMatchSumScorer.java:239)
org.apache.lucene.search.MinShouldMatchSumScorer.advance(MinShouldMatchSumScorer.java:188)
org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:118)
org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:151)
org.apache.lucene.search.ConjunctionScorer.nextDoc(ConjunctionScorer.java:62)
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:216)
org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:169)
org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:772)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:486)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:324)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)
org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:363)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:375)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
7/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:161)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
10.2% (51.2ms out of 500ms) cpu usage by thread 'elasticsearch[es-1][search][T#21]'
3/10 snapshots sharing following 2 elements
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
what should i see on hot_threads? sometimes there is no thread in response at all.and some times it contains the following http://www.wepaste.com/es-%20hot%20thread/