# Siren-join query cause high io **URL:** \ **Category:** Elasticsearch\ **Created:** [July 26, 2016, 12:48pm UTC](https://discuss.elastic.co/t/siren-join-query-cause-high-io/56396 "2016-07-26T12:48:44Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![win1027](https://avatars.discourse-cdn.com/v4/letter/w/7bcc69/32.png) [@win1027](https://discuss.elastic.co/u/win1027)\ **Post date:** [July 26, 2016, 12:48pm UTC](https://discuss.elastic.co/t/siren-join-query-cause-high-io/56396/1 "2016-07-26T12:48:44Z")
i use siren-join plugin replace es parent-child. but now many siren-join query cause machine high io. ```auto GET test_1/1/_coordinate_search { "from" : 0, "size" : 50, "timeout" : 30000, "query" : { "filterjoin" : { "id" : { "indices" : ["test_2"], "types" : ["0"], "path" : "id", "query" : { "bool" : { "filter" : { "bool" : { "must" : { "wildcard" : { "detail" : "*test detail*" } } } } } }, "maxTermsPerShard" : 10000000 } } }, "sort" : [ { "time" : { "order" : "desc" } } ] } ``` this is my query. test\_1 has 3m docs, test\_2 has 50m docs, and this query "detail" : "_test detail_" hit 1000+ result. this is total cost 100+s, and cause high io and load: ```auto top - 20:42:09 up 36 days, 8:37, 1 user, load average: 28.38, 25.04, 14.90 Tasks: 541 total, 1 running, 540 sleeping, 0 stopped, 0 zombie Cpu0 : 8.0%us, 1.9%sy, 0.0%ni, 1.0%id, 88.4%wa, 0.0%hi, 0.6%si, 0.0%st Cpu1 : 2.3%us, 0.3%sy, 0.0%ni, 21.4%id, 76.1%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 0.7%us, 1.0%sy, 0.0%ni, 54.4%id, 43.9%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 2.6%us, 0.3%sy, 0.0%ni, 52.6%id, 44.4%wa, 0.0%hi, 0.0%si, 0.0%st Cpu4 : 0.3%us, 0.3%sy, 0.0%ni, 79.1%id, 20.2%wa, 0.0%hi, 0.0%si, 0.0%st Cpu5 : 0.3%us, 0.0%sy, 0.0%ni, 67.0%id, 32.7%wa, 0.0%hi, 0.0%si, 0.0%st Cpu6 : 9.6%us, 1.0%sy, 0.0%ni, 15.1%id, 74.0%wa, 0.0%hi, 0.3%si, 0.0%st Cpu7 : 2.3%us, 1.3%sy, 0.0%ni, 50.2%id, 46.2%wa, 0.0%hi, 0.0%si, 0.0%st Cpu8 : 1.0%us, 1.3%sy, 0.0%ni, 60.0%id, 37.4%wa, 0.0%hi, 0.3%si, 0.0%st Cpu9 : 0.3%us, 0.0%sy, 0.0%ni, 72.8%id, 26.8%wa, 0.0%hi, 0.0%si, 0.0%st Cpu10 : 0.7%us, 0.3%sy, 0.0%ni, 79.2%id, 19.9%wa, 0.0%hi, 0.0%si, 0.0%st Cpu11 : 0.6%us, 0.3%sy, 0.0%ni, 79.5%id, 19.5%wa, 0.0%hi, 0.0%si, 0.0%st Cpu12 : 2.6%us, 1.3%sy, 0.0%ni, 20.2%id, 75.9%wa, 0.0%hi, 0.0%si, 0.0%st Cpu13 : 0.3%us, 0.3%sy, 0.0%ni, 81.5%id, 17.8%wa, 0.0%hi, 0.0%si, 0.0%st Cpu14 : 0.3%us, 0.3%sy, 0.0%ni, 94.4%id, 4.9%wa, 0.0%hi, 0.0%si, 0.0%st Cpu15 : 0.3%us, 0.0%sy, 0.0%ni, 92.7%id, 7.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu16 : 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu17 : 0.3%us, 0.0%sy, 0.0%ni, 98.4%id, 1.3%wa, 0.0%hi, 0.0%si, 0.0%st Cpu18 : 2.3%us, 0.3%sy, 0.0%ni, 45.1%id, 52.3%wa, 0.0%hi, 0.0%si, 0.0%st Cpu19 : 0.7%us, 0.7%sy, 0.0%ni, 73.9%id, 24.8%wa, 0.0%hi, 0.0%si, 0.0%st Cpu20 : 0.7%us, 0.0%sy, 0.0%ni, 86.9%id, 12.5%wa, 0.0%hi, 0.0%si, 0.0%st Cpu21 : 0.3%us, 0.0%sy, 0.0%ni, 97.0%id, 2.7%wa, 0.0%hi, 0.0%si, 0.0%st Cpu22 : 1.0%us, 0.3%sy, 0.0%ni, 94.1%id, 4.6%wa, 0.0%hi, 0.0%si, 0.0%st Cpu23 : 0.3%us, 0.3%sy, 0.0%ni, 97.0%id, 2.3%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 131909748k total, 131494052k used, 415696k free, 147640k buffers Swap: 2088952k total, 2088948k used, 4k free, 88329696k cached ``` this machine has 6 hard drives , no raid. How to Optimize? ---
**Author:** ![win1027](https://avatars.discourse-cdn.com/v4/letter/w/7bcc69/32.png) [@win1027](https://discuss.elastic.co/u/win1027)\ **Post date:** [July 26, 2016, 12:49pm UTC](https://discuss.elastic.co/t/siren-join-query-cause-high-io/56396/2 "2016-07-26T12:49:44Z")
2.5% (12.3ms out of 500ms) cpu usage by thread 'elasticsearch[data-10.175.86.220][generic][T#4944]' 5/10 snapshots sharing following 41 elements sun.nio.ch.FileDispatcherImpl.pread0(Native Method) sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52) sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220) sun.nio.ch.IOUtil.read(IOUtil.java:197) sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:708) sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:694) org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179) org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342) org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54) org.apache.lucene.store.DataInput.readVInt(DataInput.java:125) org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:221) org.apache.lucene.codecs.blocktree.IntersectTermsEnumFrame.load(IntersectTermsEnumFrame.java:185) org.apache.lucene.codecs.blocktree.IntersectTermsEnum.pushFrame(IntersectTermsEnum.java:211) org.apache.lucene.codecs.blocktree.IntersectTermsEnum.\_next(IntersectTermsEnum.java:666) org.apache.lucene.codecs.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:501) org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.collectTerms(MultiTermQueryConstantScoreWrapper.java:121) org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.rewrite(MultiTermQueryConstantScoreWrapper.java:152) org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1.bulkScorer(MultiTermQueryConstantScoreWrapper.java:199) org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:604) org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorer(LRUQueryCache.java:625) org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.scorer(IndicesQueryCache.java:263) org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:389) org.apache.lucene.search.Weight.bulkScorer(Weight.java:135) org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:370) org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:604) org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:652) org.elasticsearch.indices.cache.query.IndicesQueryCache$CachingWeightWrapper.bulkScorer(IndicesQueryCache.java:269) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818) org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535) solutions.siren.join.action.terms.collector.HitStream.initialize(HitStream.java:46) solutions.siren.join.action.terms.collector.TermsCollector.collect(TermsCollector.java:68) solutions.siren.join.action.terms.TransportTermsByQueryAction.shardOperation(TransportTermsByQueryAction.java:286) solutions.siren.join.action.terms.TransportTermsByQueryAction.shardOperation(TransportTermsByQueryAction.java:74) org.elasticsearch.action.support.broadcast.TransportBroadcastAction$ShardTransportHandler.messageReceived(TransportBroadcastAction.java:282) org.elasticsearch.action.support.broadcast.TransportBroadcastAction$ShardTransportHandler.messageReceived(TransportBroadcastAction.java:278) org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75) org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:300) org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 26, 2016, 1:28pm UTC](https://discuss.elastic.co/t/siren-join-query-cause-high-io/56396/3 "2016-07-26T13:28:03Z")
I don't know this plugin but for sure you should never write something like `"detail" : "*test detail*"` This is one of the worse query you can run on elasticsearch. Don't do this! ---
**Author:** ![win1027](https://avatars.discourse-cdn.com/v4/letter/w/7bcc69/32.png) [@win1027](https://discuss.elastic.co/u/win1027)\ **Post date:** [July 26, 2016, 1:35pm UTC](https://discuss.elastic.co/t/siren-join-query-cause-high-io/56396/4 "2016-07-26T13:35:55Z")
thanks. then how i use another query to replace "wildcard" : { "detail" : "_test detail_" } ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 26, 2016, 1:49pm UTC](https://discuss.elastic.co/t/siren-join-query-cause-high-io/56396/5 "2016-07-26T13:49:55Z")
You can use a match query instead. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [July 26, 2016, 1:51pm UTC](https://discuss.elastic.co/t/siren-join-query-cause-high-io/56396/6 "2016-07-26T13:51:47Z")
It's potentially a big join. So either: 1. a lot of expensive random disk seeks (searches to find terms from index 1 in index 2) 2. a big scan (serially scan all terms in index 1 to see if they are in the result set from index 2) Either way it will cost. Probably best to take up with the Siren peeps. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:32pm UTC](https://discuss.elastic.co/t/siren-join-query-cause-high-io/56396/7 "2017-07-05T22:32:51Z")