Hello,
I could see that while rivering the records from MySQL server , the CPU
usage riches almost 100%. We have around 100K record in MySQL Table, river
is created using following code,
curl -XPUT 'localhost:9200/river/myjdbc_river/
meta' -d '{ "type" : "jdbc", "jdbc" : { "driver" :
"com.mysql.jdbc.Driver", "url" : "jdbc:mysql://107.108.209.189:3306/CSPDB",
"user" : "root", "password" : "ssf", "strategy" : "simple", "sql" : "select
name AS username,OSPGuid AS user_ospguid,userId AS user_id,userId AS id,phone
AS userphone,email AS user_email,extProperty AS user_profile,createTime as
user_createtime,modTime as user_modtime from CSPDB.User",
"index" : "user",
"type" : "user_type",
"bulk_size":1000,
"max_bulk_requests":50000,
"poll" : "60s"
}
}'
Hot Thread output
::: [Anup][OSRMJuOoTkaVWy-DgihUwA][inet[/107.108.209.27:9300]]
74.7% (373.4ms out of 500ms) cpu usage by thread 'elasticsearch[Anup][JDBC
river [user_river/simple]][T#1]'
9/10 snapshots sharing following 8 elements
org.xbib.elasticsearch.river.jdbc.support.SimpleValueListener.values(SimpleValueListener.java:22)
org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.processRow(SimpleRiverSource.java:519)
org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.nextRow(SimpleRiverSource.java:499)
org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.merge(SimpleRiverSource.java:312)
org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.fetch(SimpleRiverSource.java:243)
org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow.move(SimpleRiverFlow.java:183)
org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow.run(SimpleRiverFlow.java:120)
java.lang.Thread.run(Thread.java:722)
unique snapshot
sun.nio.cs.UTF_8$Decoder.decodeArrayLoop(UTF_8.java:201)
sun.nio.cs.UTF_8$Decoder.decodeLoop(UTF_8.java:354)
java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:561)
java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:783)
java.nio.charset.Charset.decode(Charset.java:810)
com.mysql.jdbc.StringUtils.toString(StringUtils.java:1871)
com.mysql.jdbc.ResultSetRow.getString(ResultSetRow.java:821)
com.mysql.jdbc.BufferRow.getString(BufferRow.java:542)
com.mysql.jdbc.ResultSetImpl.getStringInternal(ResultSetImpl.java:5816)
com.mysql.jdbc.ResultSetImpl.getString(ResultSetImpl.java:5693)
org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.parseType(SimpleRiverSource.java:750)
org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.processRow(SimpleRiverSource.java:512)
org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.nextRow(SimpleRiverSource.java:499)
org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.merge(SimpleRiverSource.java:312)
org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.fetch(SimpleRiverSource.java:243)
org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow.move(SimpleRiverFlow.java:183)
org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow.run(SimpleRiverFlow.java:120)
java.lang.Thread.run(Thread.java:722)
46.9% (234.5ms out of 500ms) cpu usage by thread
'elasticsearch[Anup][bulk][T#1]'
8/10 snapshots sharing following 13 elements
org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:375)
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:463)
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1551)
org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:590)
org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:495)
org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:375)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:397)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155)
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
2/10 snapshots sharing following 19 elements
org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85)
org.apache.lucene.index.TermsHash.flush(TermsHash.java:116)
org.apache.lucene.index.DocInverter.flush(DocInverter.java:53)
org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:81)
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:466)
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:499)
org.apache.lucene.index.DocumentsWriter.preUpdate(DocumentsWriter.java:349)
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:431)
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1551)
org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:590)
org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:495)
org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:375)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:397)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155)
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
45.6% (228ms out of 500ms) cpu usage by thread
'elasticsearch[Anup][bulk][T#3]'
10/10 snapshots sharing following 11 elements
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1551)
org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:590)
org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:495)
org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:375)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:397)
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155)
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
Can you please tell me what is wrong here. Is some problem with using
[ "bulk_size":1000,
"max_bulk_requests":50000 ]
settings? Any suggestion for configuration to be used while creating river
and its usage when indexing more records( around 1Milion) with shorter poll
time.
I am using all latest version for Elastic Search Server and matching JDBC
river plug-in(). Can any body help in this ?
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fa3213ed-351f-4e02-b6f1-de12be74ec58%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.