I am using transport client in my JAVA application.
I noticed that indexing operation (indexRequestBuilder.execute().get()) takes up to ~200 ms!!!
What can I do in order to check where is the bottleneck?
Well. That's not may be that bad.
I think you are unhappy with this because if you have 10000 documents to index, it will take you 33 minutes. Is that right?
But what would happen if you use _bulk API instead? Let say you index a bulk of size 10000 docs. How long will it take?
Can you test it?
Also, what your mapping is looking like?
Then, you can run a hot threads API call while indexing. May be it will tell you something...
You are absolutely right, thank you. I am aware of the bulk API. Wanted to maximize the performance of single insert before going that way.
This is going to be the next step if nothing else will work.
Can you explain what is taking "so" long? Http request build? I have 2 replicas, so maybe the replication to the other nodes and waiting for the answer?
My mapping is:
{
"index": {
"mappings": {
"events": {
"properties": {
"affected_entity_id": {
"type": "long"
},
"code": {
"type": "keyword"
},
"data": {
"properties": {
"name": {
"type": "keyword"
},
"type": {
"type": "text",
"index": false
},
"value": {
"type": "text"
}
}
},
"description_template": {
"type": "keyword"
},
"id": {
"type": "long"
},
"index": {
"type": "long"
},
"isOfflineEvent": {
"type": "boolean"
},
"level": {
"type": "keyword"
},
"node_init_timestamp": {
"type": "long"
},
"reporter": {
"type": "keyword"
},
"seq_num": {
"type": "double"
},
"source_node_id": {
"type": "long"
},
"system_version": {
"type": "keyword"
},
"timestamp": {
"type": "long"
},
"username": {
"type": "keyword"
},
"visibility": {
"type": "keyword"
}
}
}
}
Does not look like a "big" mapping. Unless you are sending a huge JSON document.
Definitely.
Elasticsearch has to get the request on the coordinating node, then send the request to the primary shard. There update the transaction log and fsync it and index. Then send the index request to replicas in parallel and do the same thing.
Then send the response back to the coordinating node which has to answer to your client.
Try without replica. You can also disable the fsync on disk although it's absolutely not recommended to do it.
Isn't fsync part of flushing? From what you said, it's sounds that ES goes to the disk for every index
Not exactly for every index operation but for every request.
A request can be a single index operation (what you are doing) or a bulk request (what I encourage you to do).
Ran hot threads API. Found that much CPU is used in order to merge:
85.0% (424.9ms out of 500ms) cpu usage by thread 'elasticsearch[node-2][[events_1498976451215][0]: Lucene Merge Thread #191]'
4/10 snapshots sharing following 11 elements
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:866)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4353)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3928)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
2/10 snapshots sharing following 25 elements
org.apache.lucene.util.packed.PackedInts$MutableImpl.<init>(PackedInts.java:631)
org.apache.lucene.util.packed.Direct8.<init>(Direct8.java:35)
org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:954)
org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:939)
org.apache.lucene.util.packed.GrowableWriter.<init>(GrowableWriter.java:46)
org.apache.lucene.util.packed.PagedGrowableWriter.newMutable(PagedGrowableWriter.java:56)
org.apache.lucene.util.packed.AbstractPagedMutable.fillPages(AbstractPagedMutable.java:57)
org.apache.lucene.util.packed.PagedGrowableWriter.<init>(PagedGrowableWriter.java:50)
org.apache.lucene.util.packed.PagedGrowableWriter.<init>(PagedGrowableWriter.java:43)
org.apache.lucene.util.fst.NodeHash.<init>(NodeHash.java:36)
org.apache.lucene.util.fst.Builder.<init>(Builder.java:175)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$PendingBlock.compileIndex(BlockTreeTermsWriter.java:457)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.writeBlocks(BlockTreeTermsWriter.java:635)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.pushTerm(BlockTreeTermsWriter.java:907)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:871)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4353)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3928)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
2/10 snapshots sharing following 12 elements
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.pushTerm(BlockTreeTermsWriter.java:907)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:871)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344)
org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4353)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3928)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
2/10 snapshots sharing following 11 elements
org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:196)
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:336)
org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4353)
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3928)
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99)
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
Is this expected? anything I can try to tune or change in order to affect this?
I have 1 shard with 2 replicas as stated before...
Thanks
You can disable refresh. It will help to do mass importing.
My case is not a mass import case, and refresh interval must stay 1 second for the client.
Any other configurations and setups that affect this behavior? What causes the frequent merges? refresh operation?
Thanks
Yes. Under heavy load you are producing a new segment every second which means that segments need to be merged at some point.
Merging segments can cause a lot of IO.
I checked this scenario with 5s refresh interval. The performance dropped... How is this making sense?
With bulk? Can you describe exactly what you are doing?
Currently, my scenario is indexing documents 1 by 1, not in bulk. I am trying to figure out what is the main reason for my results (~30 documents/1 second).
I understand from you that this is normal and I should use bulk API, but I just want to know what is the bottleneck.
The thing is that it would be strange trying to solve a problem which does not really exist.
In my tests, using single index requests, I'm injecting 30 docs/s. Using bulk I'm injecting the same dataset at a rate of 15000 docs/s.
So I'd first use bulk then if you see issues, try to fix.
1 Like
The case is getting request from user and returning ack. I must insert the request to Elastic before returning ack. This is why I can't use bulk
Use refresh=wait_for: https://www.elastic.co/guide/en/elasticsearch/reference/5.4/docs-refresh.html#docs-refresh
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.