"_source" : {"error":"NoClassSettingsException[Failed to load class with value [jdbc]]; nested: ClassNotFoundException[jdbc]; "


(fzhou) #1

Hi,

I have 10 nodes ES setup, node1 and node 10 are master nodes, the rest are
data nodes.
I installed river jdbc on node1, ES master is running on node10 now.

I already load about 3/4 of large table from mysql, and since I
reconfigured cluster use unicast, I want to continue

The river does not work, give me the error like

http://devcassandra1:9200/_river/my_jdbc_river/_status?pretty=true

{
"_index" : "_river",
"_type" : "my_jdbc_river",
"_id" : "_status",
"_version" : 1,
"exists" : true, "_source" : {"error":"NoClassSettingsException[Failed to load class with value [jdbc]]; nested: ClassNotFoundException[jdbc]; ","node":{"id":"ehset9gJQIiQs70_7Sr49Q","name":"devES1-node7","transport_address":"inet[/172.16.12.71:9300]"}}
}

I don't understand why I issue the command on node1, master running on node10, but the status complain node7 has problem, and the problem is ClassNotFoundException(jdbc). When the problem occurred, I can not see mysql login as usually I can see, master log just contains

[2013-12-15 15:24:42,220][INFO ][cluster.metadata ] [devES1-node10] [_river] update_mapping [my_jdbc_river] (dynamic)
[2013-12-15 15:24:42,800][INFO ][cluster.metadata ] [devES1-node10] [_river] update_mapping [my_jdbc_river] (dynamic)

The river command I am using like

curl -XPUT 'localhost:9200/_river/my_jdbc_river/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "simple",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/tt",
"user" : "abc",
"autocommit" : true,
"password" : "abc123",
"sql" : "select ID as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test",
"type" : "tt",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

the tt type in index test already has 17M documents, the source table contains 22M records.

Thanks for help!

Frank

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a2b2c118-84b5-467a-9b49-f9e34e27b378%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(fzhou) #2

On node1

[root@devcassandra1 river-jdbc]# pwd
/usr/share/elasticsearch/plugins/river-jdbc

[root@devcassandra1 river-jdbc]# ls -larth
total 1.3M
-rw-r--r-- 1 root root 308K Dec 15 04:48
elasticsearch-river-jdbc-2.3.1-javadoc.jar
-rw-r--r-- 1 root root 66K Dec 15 04:48 elasticsearch-river-jdbc-2.3.1.jar
drwxr-xr-x 11 root root 4.0K Dec 15 04:48 ..
drwxr-xr-x 2 root root 4.0K Dec 15 04:48 .
-rw-r--r-- 1 root root 852K Dec 15 04:48
mysql-connector-java-5.1.27-bin.jar

[root@devcassandra1 river-jdbc]# lsof|grep river
bash 4751 root cwd DIR 0,81 4096
165437480 /usr/share/elasticsearch/plugins/river-jdbc
java 4944 elasticsearch mem REG 253,2
165423565
/usr/share/elasticsearch/plugins/river-jdbc/elasticsearch-river-jdbc-2.3.1.jar
(path dev=0,81)
java 4944 elasticsearch mem REG 253,2
165423564
/usr/share/elasticsearch/plugins/river-jdbc/elasticsearch-river-jdbc-2.3.1-javadoc.jar
(path dev=0,81)
java 4944 elasticsearch mem REG 253,2
165423566
/usr/share/elasticsearch/plugins/river-jdbc/mysql-connector-java-5.1.27-bin.jar
(path dev=0,81)
java 4944 elasticsearch 27r REG 0,81 872303
165423566
/usr/share/elasticsearch/plugins/river-jdbc/mysql-connector-java-5.1.27-bin.jar
java 4944 elasticsearch 28r REG 0,81 314871
165423564
/usr/share/elasticsearch/plugins/river-jdbc/elasticsearch-river-jdbc-2.3.1-javadoc.jar
java 4944 elasticsearch 29r REG 0,81 66891
165423565
/usr/share/elasticsearch/plugins/river-jdbc/elasticsearch-river-jdbc-2.3.1.jar
lsof 6773 root cwd DIR 0,81 4096
165437480 /usr/share/elasticsearch/plugins/river-jdbc
grep 6774 root cwd DIR 0,81 4096
165437480 /usr/share/elasticsearch/plugins/river-jdbc
lsof 6775 root cwd DIR 0,81 4096
165437480 /usr/share/elasticsearch/plugins/river-jdbc

Thanks
Frank

On Sunday, December 15, 2013 3:46:12 PM UTC-8, Frank Zhou wrote:

Hi,

I have 10 nodes ES setup, node1 and node 10 are master nodes, the rest are
data nodes.
I installed river jdbc on node1, ES master is running on node10 now.

I already load about 3/4 of large table from mysql, and since I
reconfigured cluster use unicast, I want to continue

The river does not work, give me the error like

http://devcassandra1:9200/_river/my_jdbc_river/_status?pretty=true

{
"_index" : "_river",
"_type" : "my_jdbc_river",
"_id" : "_status",
"_version" : 1,
"exists" : true, "_source" : {"error":"NoClassSettingsException[Failed to load class with value [jdbc]]; nested: ClassNotFoundException[jdbc]; ","node":{"id":"ehset9gJQIiQs70_7Sr49Q","name":"devES1-node7","transport_address":"inet[/172.16.12.71:9300]"}}
}

I don't understand why I issue the command on node1, master running on node10, but the status complain node7 has problem, and the problem is ClassNotFoundException(jdbc). When the problem occurred, I can not see mysql login as usually I can see, master log just contains

[2013-12-15 15:24:42,220][INFO ][cluster.metadata ] [devES1-node10] [_river] update_mapping [my_jdbc_river] (dynamic)
[2013-12-15 15:24:42,800][INFO ][cluster.metadata ] [devES1-node10] [_river] update_mapping [my_jdbc_river] (dynamic)

The river command I am using like

curl -XPUT 'localhost:9200/_river/my_jdbc_river/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "simple",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/tt",
"user" : "abc",
"autocommit" : true,
"password" : "abc123",
"sql" : "select ID as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test",
"type" : "tt",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

the tt type in index test already has 17M documents, the source table contains 22M records.

Thanks for help!

Frank

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f65e7e88-7b6c-4666-9ee0-9c6bb55507d7%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(fzhou) #3

I already deleted _river index and recreated several times, now, it only
has 2 shards on node3 and node4

But when I follow the error message check node7, I found

[root@devcassandra7 plugins]# lsof|grep river
java 761 elasticsearch 587r REG 0,165 0
211291214
/var/lib/elasticsearch/devES1/nodes/0/indices/_river/3/index/write.lock
java 761 elasticsearch 1514r REG 0,165 280
211292193
/var/lib/elasticsearch/devES1/nodes/0/indices/_river/3/translog/translog-1387149881929
java 761 elasticsearch 1516r REG 0,165 0
211292939
/var/lib/elasticsearch/devES1/nodes/0/indices/_river/3/index/_0.fdx
java 761 elasticsearch 1517r REG 0,165 0
211292996
/var/lib/elasticsearch/devES1/nodes/0/indices/_river/3/index/_0.fdt

They are new after I create new meta data. If I delete _river again, above
files will disappear.

Does it mean my previous loading left some garbage? Why the new _river
index will always direct the call to node7?

Thanks
Frank

On Sunday, December 15, 2013 3:46:12 PM UTC-8, Frank Zhou wrote:

Hi,

I have 10 nodes ES setup, node1 and node 10 are master nodes, the rest are
data nodes.
I installed river jdbc on node1, ES master is running on node10 now.

I already load about 3/4 of large table from mysql, and since I
reconfigured cluster use unicast, I want to continue

The river does not work, give me the error like

http://devcassandra1:9200/_river/my_jdbc_river/_status?pretty=true

{
"_index" : "_river",
"_type" : "my_jdbc_river",
"_id" : "_status",
"_version" : 1,
"exists" : true, "_source" : {"error":"NoClassSettingsException[Failed to load class with value [jdbc]]; nested: ClassNotFoundException[jdbc]; ","node":{"id":"ehset9gJQIiQs70_7Sr49Q","name":"devES1-node7","transport_address":"inet[/172.16.12.71:9300]"}}
}

I don't understand why I issue the command on node1, master running on node10, but the status complain node7 has problem, and the problem is ClassNotFoundException(jdbc). When the problem occurred, I can not see mysql login as usually I can see, master log just contains

[2013-12-15 15:24:42,220][INFO ][cluster.metadata ] [devES1-node10] [_river] update_mapping [my_jdbc_river] (dynamic)
[2013-12-15 15:24:42,800][INFO ][cluster.metadata ] [devES1-node10] [_river] update_mapping [my_jdbc_river] (dynamic)

The river command I am using like

curl -XPUT 'localhost:9200/_river/my_jdbc_river/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "simple",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/tt",
"user" : "abc",
"autocommit" : true,
"password" : "abc123",
"sql" : "select ID as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test",
"type" : "tt",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

the tt type in index test already has 17M documents, the source table contains 22M records.

Thanks for help!

Frank

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5a068c0f-8f14-42e0-a314-03763dda7638%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(fzhou) #4

I closed target index tt, and completely restart whole cluster, then, test
again with new type name my_SMSMT_river

This time, the exception changed to node6

http://devcassandra1:9200/_river/my_SMSMT_river/_status?pretty=true

{
"_index" : "_river",
"_type" : "my_SMSMT_river",
"_id" : "_status",
"_version" : 1,
"exists" : true, "_source" : {"error":"NoClassSettingsException[Failed to load class with value [jdbc]]; nested: ClassNotFoundException[jdbc]; ","node":{"id":"MMDEBAm5RNG9ddidUlhM8g","name":"devES1-node6","transport_address":"inet[/172.16.12.70:9300]"}}
}

Thanks

Frank

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1cb414a7-6738-4534-85ca-8a32083f6761%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #5

Did you install jdbc river on all nodes?
How do you create the river?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 02:05, Frank Zhou zhouxuesong@gmail.com a écrit :

I closed target index tt, and completely restart whole cluster, then, test again with new type name my_SMSMT_river

This time, the exception changed to node6

http://devcassandra1:9200/_river/my_SMSMT_river/_status?pretty=true
{
"_index" : "_river",
"_type" : "my_SMSMT_river",
"_id" : "_status",
"_version" : 1,
"exists" : true, "_source" : {"error":"NoClassSettingsException[Failed to load class with value [jdbc]]; nested: ClassNotFoundException[jdbc]; ","node":{"id":"MMDEBAm5RNG9ddidUlhM8g","name":"devES1-node6","transport_address":"inet[/172.16.12.70:9300]"}}
}

Thanks
Frank

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1cb414a7-6738-4534-85ca-8a32083f6761%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/76D3069D-1459-4D3B-8A1D-302BEC22669B%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.


(fzhou) #6

Hi, David

I had installed river jdbc on node1 and node4, then, realized maybe
multiple install cause problem, I removed node4 installation by removed
plugin/river-jdbc directory, and restart node4.

I did try to use plugin head to pre-create river with 1 shard 1 replica,
same error
Then, use the cluster default 5 shards 1 replica, same error,

This command I used to create type under _river index

curl -XPUT 'localhost:9200/_river/my_jdbc_river2/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "simple",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/tt",
"user" : "abc",
"autocommit" : true,
"password" : "abc123",
"sql" : "select ID as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test",
"type" : "tt",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

I did try my_jdbc_river and my_jdbc_river2 two types, both of them get the
same error.

Thanks
Frank

On Sun, Dec 15, 2013 at 8:58 PM, David Pilato david@pilato.fr wrote:

Did you install jdbc river on all nodes?
How do you create the river?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 02:05, Frank Zhou zhouxuesong@gmail.com a écrit :

I closed target index tt, and completely restart whole cluster, then, test
again with new type name my_SMSMT_river

This time, the exception changed to node6

http://devcassandra1:9200/_river/my_SMSMT_river/_status?pretty=true

{
"_index" : "_river",
"_type" : "my_SMSMT_river",
"_id" : "_status",
"_version" : 1,
"exists" : true, "_source" : {"error":"NoClassSettingsException[Failed to load class with value [jdbc]]; nested: ClassNotFoundException[jdbc]; ","node":{"id":"MMDEBAm5RNG9ddidUlhM8g","name":"devES1-node6","transport_address":"inet[/172.16.12.70:9300]"}}
}

Thanks

Frank

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1cb414a7-6738-4534-85ca-8a32083f6761%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/76D3069D-1459-4D3B-8A1D-302BEC22669B%40pilato.fr
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABeXrnc5-ww4Zb1VBpU7Bt_BA%3Do__8rDREM6CjxoYT3WxBZeTw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(fzhou) #7

Hi, David

It seems really matter with the existing type

I create new river type with same sql different source mysql db, and
different index type, I can see river login from mysql DB.

curl -XPUT 'localhost:9200/_river/my_SMSMT_river_19/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "simple",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/tt19",
"user" : "abc",
"autocommit" : true,
"password" : "abc123",
"sql" : "select ID as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test",
"type" : "tt_19",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

Thanks
Frank

On Sun, Dec 15, 2013 at 8:58 PM, David Pilato david@pilato.fr wrote:

Did you install jdbc river on all nodes?
How do you create the river?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 02:05, Frank Zhou zhouxuesong@gmail.com a écrit :

I closed target index tt, and completely restart whole cluster, then, test
again with new type name my_SMSMT_river

This time, the exception changed to node6

http://devcassandra1:9200/_river/my_SMSMT_river/_status?pretty=true

{
"_index" : "_river",
"_type" : "my_SMSMT_river",
"_id" : "_status",
"_version" : 1,
"exists" : true, "_source" : {"error":"NoClassSettingsException[Failed to load class with value [jdbc]]; nested: ClassNotFoundException[jdbc]; ","node":{"id":"MMDEBAm5RNG9ddidUlhM8g","name":"devES1-node6","transport_address":"inet[/172.16.12.70:9300]"}}
}

Thanks

Frank

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1cb414a7-6738-4534-85ca-8a32083f6761%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/76D3069D-1459-4D3B-8A1D-302BEC22669B%40pilato.fr
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABeXrndFViOZRJF_%2Bv7o1xrigmcW-RNdAD6xM2v5tYk%3DQOiFng%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #8

My question was: "Did you install jdbc river on ALL nodes?"
I wrote you have 10 nodes, right?

First install the river on all of them.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 06:04, Frank Zhou zhouxuesong@gmail.com a écrit :

Hi, David

I had installed river jdbc on node1 and node4, then, realized maybe multiple install cause problem, I removed node4 installation by removed plugin/river-jdbc directory, and restart node4.

I did try to use plugin head to pre-create river with 1 shard 1 replica, same error
Then, use the cluster default 5 shards 1 replica, same error,

This command I used to create type under _river index

curl -XPUT 'localhost:9200/_river/my_jdbc_river2/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "simple",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/tt",
"user" : "abc",
"autocommit" : true,
"password" : "abc123",
"sql" : "select ID as _id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID from SMSMT"
},
"index" : {
"index" : "test",
"type" : "tt",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

I did try my_jdbc_river and my_jdbc_river2 two types, both of them get the same error.

Thanks
Frank

On Sun, Dec 15, 2013 at 8:58 PM, David Pilato david@pilato.fr wrote:
Did you install jdbc river on all nodes?
How do you create the river?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 02:05, Frank Zhou zhouxuesong@gmail.com a écrit :

I closed target index tt, and completely restart whole cluster, then, test again with new type name my_SMSMT_river

This time, the exception changed to node6

http://devcassandra1:9200/_river/my_SMSMT_river/_status?pretty=true
{
"_index" : "_river",
"_type" : "my_SMSMT_river",
"_id" : "_status",
"_version" : 1,
"exists" : true, "_source" : {"error":"NoClassSettingsException[Failed to load class with value [jdbc]]; nested: ClassNotFoundException[jdbc]; ","node":{"id":"MMDEBAm5RNG9ddidUlhM8g","name":"devES1-node6","transport_address":"inet[/172.16.12.70:9300]"}}
}

Thanks
Frank

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1cb414a7-6738-4534-85ca-8a32083f6761%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/76D3069D-1459-4D3B-8A1D-302BEC22669B%40pilato.fr.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABeXrnc5-ww4Zb1VBpU7Bt_BA%3Do__8rDREM6CjxoYT3WxBZeTw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5CD9C017-B939-4A18-9F4C-8749C4E71350%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.


(fzhou) #9

Oh! First time know I should install driver on all of the nodes. I will do
it and want to know why

  1. Does it mean all of the data/non-data nodes have ability to directly
    connect to source mysql database and fetch data?
  2. Why one node driver installation can support the new index type creation
    without problem?

Thanks

On Sun, Dec 15, 2013 at 10:01 PM, David Pilato david@pilato.fr wrote:

My question was: "Did you install jdbc river on ALL nodes?"
I wrote you have 10 nodes, right?

First install the river on all of them.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 06:04, Frank Zhou zhouxuesong@gmail.com a écrit :

Hi, David

I had installed river jdbc on node1 and node4, then, realized maybe
multiple install cause problem, I removed node4 installation by removed
plugin/river-jdbc directory, and restart node4.

I did try to use plugin head to pre-create river with 1 shard 1 replica,
same error
Then, use the cluster default 5 shards 1 replica, same error,

This command I used to create type under _river index

curl -XPUT 'localhost:9200/_river/my_jdbc_river2/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "simple",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/tt",
"user" : "abc",
"autocommit" : true,
"password" : "abc123",
"sql" : "select ID as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test",
"type" : "tt",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

I did try my_jdbc_river and my_jdbc_river2 two types, both of them get the
same error.

Thanks
Frank

On Sun, Dec 15, 2013 at 8:58 PM, David Pilato david@pilato.fr wrote:

Did you install jdbc river on all nodes?
How do you create the river?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 02:05, Frank Zhou zhouxuesong@gmail.com a écrit :

I closed target index tt, and completely restart whole cluster, then,
test again with new type name my_SMSMT_river

This time, the exception changed to node6

http://devcassandra1:9200/_river/my_SMSMT_river/_status?pretty=true

{
"_index" : "_river",
"_type" : "my_SMSMT_river",
"_id" : "_status",
"_version" : 1,
"exists" : true, "_source" : {"error":"NoClassSettingsException[Failed to load class with value [jdbc]]; nested: ClassNotFoundException[jdbc]; ","node":{"id":"MMDEBAm5RNG9ddidUlhM8g","name":"devES1-node6","transport_address":"inet[/172.16.12.70:9300]"}}
}

Thanks

Frank

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1cb414a7-6738-4534-85ca-8a32083f6761%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/76D3069D-1459-4D3B-8A1D-302BEC22669B%40pilato.fr
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CABeXrnc5-ww4Zb1VBpU7Bt_BA%3Do__8rDREM6CjxoYT3WxBZeTw%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5CD9C017-B939-4A18-9F4C-8749C4E71350%40pilato.fr
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABeXrndeNCiTUA3iP1UG95VNhtMAtwJNopVEaOjc%3DjNs5%3DQ8Uw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #10

A river can run on any data node (not sure about non data nodes though).
Because when a node stops, river needs to be started on another one. It means that rivers are replicated.

That's the reason you need to have all plugins installed on each node.

Makes sense?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 19:01, Frank Zhou zhouxuesong@gmail.com a écrit :

Oh! First time know I should install driver on all of the nodes. I will do it and want to know why

  1. Does it mean all of the data/non-data nodes have ability to directly connect to source mysql database and fetch data?
  2. Why one node driver installation can support the new index type creation without problem?

Thanks

On Sun, Dec 15, 2013 at 10:01 PM, David Pilato david@pilato.fr wrote:
My question was: "Did you install jdbc river on ALL nodes?"
I wrote you have 10 nodes, right?

First install the river on all of them.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 06:04, Frank Zhou zhouxuesong@gmail.com a écrit :

Hi, David

I had installed river jdbc on node1 and node4, then, realized maybe multiple install cause problem, I removed node4 installation by removed plugin/river-jdbc directory, and restart node4.

I did try to use plugin head to pre-create river with 1 shard 1 replica, same error
Then, use the cluster default 5 shards 1 replica, same error,

This command I used to create type under _river index

curl -XPUT 'localhost:9200/_river/my_jdbc_river2/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "simple",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/tt",
"user" : "abc",
"autocommit" : true,
"password" : "abc123",
"sql" : "select ID as _id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID from SMSMT"
},
"index" : {
"index" : "test",
"type" : "tt",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

I did try my_jdbc_river and my_jdbc_river2 two types, both of them get the same error.

Thanks
Frank

On Sun, Dec 15, 2013 at 8:58 PM, David Pilato david@pilato.fr wrote:
Did you install jdbc river on all nodes?
How do you create the river?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 02:05, Frank Zhou zhouxuesong@gmail.com a écrit :

I closed target index tt, and completely restart whole cluster, then, test again with new type name my_SMSMT_river

This time, the exception changed to node6

http://devcassandra1:9200/_river/my_SMSMT_river/_status?pretty=true
{
"_index" : "_river",
"_type" : "my_SMSMT_river",
"_id" : "_status",
"_version" : 1,
"exists" : true, "_source" : {"error":"NoClassSettingsException[Failed to load class with value [jdbc]]; nested: ClassNotFoundException[jdbc]; ","node":{"id":"MMDEBAm5RNG9ddidUlhM8g","name":"devES1-node6","transport_address":"inet[/172.16.12.70:9300]"}}
}

Thanks
Frank

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1cb414a7-6738-4534-85ca-8a32083f6761%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/76D3069D-1459-4D3B-8A1D-302BEC22669B%40pilato.fr.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABeXrnc5-ww4Zb1VBpU7Bt_BA%3Do__8rDREM6CjxoYT3WxBZeTw%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5CD9C017-B939-4A18-9F4C-8749C4E71350%40pilato.fr.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABeXrndeNCiTUA3iP1UG95VNhtMAtwJNopVEaOjc%3DjNs5%3DQ8Uw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/A7E14344-78A1-4E93-8499-90B397CEB728%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.


(fzhou) #11

Hi, David

I installed driver on all of the data/non-data nodes, (because previously,
I just installed driver on node1 which is not data node, and it works,
indeed loaded 17million documents), then, restarted the cluster, make sure
all of the nodes loaded river driver

Then, I issue the command

curl -XPUT 'localhost:9200/_river/my_ng30-SMSMT_river_19/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "simple",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/ng19",
"user" : "abc",
"autocommit" : true,
"password" : "abc123",
"sql" : "select ID+541008901 as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test",
"type" : "ng30-SMSMT",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

Which try to continue the failed in middle work on /test/ng30-SMSMT type.

The error I immediately got is

[2013-12-16 15:16:21,477][DEBUG][action.admin.cluster.node.stats]
[devES1-node1] failed to execute on node [S4g3RKOwT8K5EmNN6XtYzg]
org.elasticsearch.transport.RemoteTransportException:
[devES1-node6][inet[/172.16.12.70:9300]][cluster/nodes/stats/n]
Caused by: org.elasticsearch.index.engine.EngineClosedException: [jdbc][1]
CurrentState[CLOSED]
at
org.elasticsearch.index.engine.robin.RobinEngine.ensureOpen(RobinEngine.java:969)
at
org.elasticsearch.index.engine.robin.RobinEngine.segmentsStats(RobinEngine.java:1181)
at
org.elasticsearch.index.shard.service.InternalIndexShard.segmentStats(InternalIndexShard.java:509)
at
org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:154)
at
org.elasticsearch.indices.InternalIndicesService.stats(InternalIndicesService.java:212)
at org.elasticsearch.node.service.NodeService.stats(NodeService.java:165)
at
org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:100)
at
org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:43)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:273)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:264)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2013-12-16 15:16:21,498][DEBUG][action.admin.cluster.node.stats]
[devES1-node1] failed to execute on node [M9cRX1vIRRKKi_GGAGyd3w]
org.elasticsearch.transport.RemoteTransportException:
[devES1-node8][inet[/172.16.12.72:9300]][cluster/nodes/stats/n]
Caused by: org.elasticsearch.index.engine.EngineClosedException: [jdbc][3]
CurrentState[CLOSED]
at
org.elasticsearch.index.engine.robin.RobinEngine.ensureOpen(RobinEngine.java:969)
at
org.elasticsearch.index.engine.robin.RobinEngine.segmentsStats(RobinEngine.java:1181)
at
org.elasticsearch.index.shard.service.InternalIndexShard.segmentStats(InternalIndexShard.java:509)
at
org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:154)
at
org.elasticsearch.indices.InternalIndicesService.stats(InternalIndicesService.java:212)
at org.elasticsearch.node.service.NodeService.stats(NodeService.java:165)
at
org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:100)
at
org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:43)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:273)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:264)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

It is different than original ClassNotFoundException, it is
EnginelosedException.

I checked the index itself, including all the the shards for test, they are
OK, nothing wrong.

Thanks
Frank

On Mon, Dec 16, 2013 at 10:50 AM, David Pilato david@pilato.fr wrote:

A river can run on any data node (not sure about non data nodes though).
Because when a node stops, river needs to be started on another one. It
means that rivers are replicated.

That's the reason you need to have all plugins installed on each node.

Makes sense?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 19:01, Frank Zhou zhouxuesong@gmail.com a écrit :

Oh! First time know I should install driver on all of the nodes. I will do
it and want to know why

  1. Does it mean all of the data/non-data nodes have ability to directly
    connect to source mysql database and fetch data?
  2. Why one node driver installation can support the new index type
    creation without problem?

Thanks

On Sun, Dec 15, 2013 at 10:01 PM, David Pilato david@pilato.fr wrote:

My question was: "Did you install jdbc river on ALL nodes?"
I wrote you have 10 nodes, right?

First install the river on all of them.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 06:04, Frank Zhou zhouxuesong@gmail.com a écrit :

Hi, David

I had installed river jdbc on node1 and node4, then, realized maybe
multiple install cause problem, I removed node4 installation by removed
plugin/river-jdbc directory, and restart node4.

I did try to use plugin head to pre-create river with 1 shard 1 replica,
same error
Then, use the cluster default 5 shards 1 replica, same error,

This command I used to create type under _river index

curl -XPUT 'localhost:9200/_river/my_jdbc_river2/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "simple",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/tt",
"user" : "abc",
"autocommit" : true,
"password" : "abc123",
"sql" : "select ID as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test",
"type" : "tt",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

I did try my_jdbc_river and my_jdbc_river2 two types, both of them get
the same error.

Thanks
Frank

On Sun, Dec 15, 2013 at 8:58 PM, David Pilato david@pilato.fr wrote:

Did you install jdbc river on all nodes?
How do you create the river?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 02:05, Frank Zhou zhouxuesong@gmail.com a écrit :

I closed target index tt, and completely restart whole cluster, then,
test again with new type name my_SMSMT_river

This time, the exception changed to node6

http://devcassandra1:9200/_river/my_SMSMT_river/_status?pretty=true

{
"_index" : "_river",
"_type" : "my_SMSMT_river",
"_id" : "_status",
"_version" : 1,
"exists" : true, "_source" : {"error":"NoClassSettingsException[Failed to load class with value [jdbc]]; nested: ClassNotFoundException[jdbc]; ","node":{"id":"MMDEBAm5RNG9ddidUlhM8g","name":"devES1-node6","transport_address":"inet[/172.16.12.70:9300]"}}
}

Thanks

Frank

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1cb414a7-6738-4534-85ca-8a32083f6761%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/76D3069D-1459-4D3B-8A1D-302BEC22669B%40pilato.fr
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CABeXrnc5-ww4Zb1VBpU7Bt_BA%3Do__8rDREM6CjxoYT3WxBZeTw%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5CD9C017-B939-4A18-9F4C-8749C4E71350%40pilato.fr
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CABeXrndeNCiTUA3iP1UG95VNhtMAtwJNopVEaOjc%3DjNs5%3DQ8Uw%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/A7E14344-78A1-4E93-8499-90B397CEB728%40pilato.fr
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABeXrnc2da_6p75caJpg6jvoNXy-WVfVniMNpfeVJ4Hw8OboFQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #12

What is your elasticsearch version?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 17 déc. 2013 à 00:38, Frank Zhou zhouxuesong@gmail.com a écrit :

Hi, David

I installed driver on all of the data/non-data nodes, (because previously, I just installed driver on node1 which is not data node, and it works, indeed loaded 17million documents), then, restarted the cluster, make sure all of the nodes loaded river driver

Then, I issue the command

curl -XPUT 'localhost:9200/_river/my_ng30-SMSMT_river_19/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "simple",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/ng19",
"user" : "abc",
"autocommit" : true,
"password" : "abc123",
"sql" : "select ID+541008901 as _id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID from SMSMT"
},
"index" : {
"index" : "test",
"type" : "ng30-SMSMT",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

Which try to continue the failed in middle work on /test/ng30-SMSMT type.

The error I immediately got is

[2013-12-16 15:16:21,477][DEBUG][action.admin.cluster.node.stats] [devES1-node1] failed to execute on node [S4g3RKOwT8K5EmNN6XtYzg]
org.elasticsearch.transport.RemoteTransportException: [devES1-node6][inet[/172.16.12.70:9300]][cluster/nodes/stats/n]
Caused by: org.elasticsearch.index.engine.EngineClosedException: [jdbc][1] CurrentState[CLOSED]
at org.elasticsearch.index.engine.robin.RobinEngine.ensureOpen(RobinEngine.java:969)
at org.elasticsearch.index.engine.robin.RobinEngine.segmentsStats(RobinEngine.java:1181)
at org.elasticsearch.index.shard.service.InternalIndexShard.segmentStats(InternalIndexShard.java:509)
at org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:154)
at org.elasticsearch.indices.InternalIndicesService.stats(InternalIndicesService.java:212)
at org.elasticsearch.node.service.NodeService.stats(NodeService.java:165)
at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:100)
at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:43)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:273)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:264)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2013-12-16 15:16:21,498][DEBUG][action.admin.cluster.node.stats] [devES1-node1] failed to execute on node [M9cRX1vIRRKKi_GGAGyd3w]
org.elasticsearch.transport.RemoteTransportException: [devES1-node8][inet[/172.16.12.72:9300]][cluster/nodes/stats/n]
Caused by: org.elasticsearch.index.engine.EngineClosedException: [jdbc][3] CurrentState[CLOSED]
at org.elasticsearch.index.engine.robin.RobinEngine.ensureOpen(RobinEngine.java:969)
at org.elasticsearch.index.engine.robin.RobinEngine.segmentsStats(RobinEngine.java:1181)
at org.elasticsearch.index.shard.service.InternalIndexShard.segmentStats(InternalIndexShard.java:509)
at org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:154)
at org.elasticsearch.indices.InternalIndicesService.stats(InternalIndicesService.java:212)
at org.elasticsearch.node.service.NodeService.stats(NodeService.java:165)
at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:100)
at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:43)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:273)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:264)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

It is different than original ClassNotFoundException, it is EnginelosedException.

I checked the index itself, including all the the shards for test, they are OK, nothing wrong.

Thanks
Frank

On Mon, Dec 16, 2013 at 10:50 AM, David Pilato david@pilato.fr wrote:
A river can run on any data node (not sure about non data nodes though).
Because when a node stops, river needs to be started on another one. It means that rivers are replicated.

That's the reason you need to have all plugins installed on each node.

Makes sense?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 19:01, Frank Zhou zhouxuesong@gmail.com a écrit :

Oh! First time know I should install driver on all of the nodes. I will do it and want to know why

  1. Does it mean all of the data/non-data nodes have ability to directly connect to source mysql database and fetch data?
  2. Why one node driver installation can support the new index type creation without problem?

Thanks

On Sun, Dec 15, 2013 at 10:01 PM, David Pilato david@pilato.fr wrote:
My question was: "Did you install jdbc river on ALL nodes?"
I wrote you have 10 nodes, right?

First install the river on all of them.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 06:04, Frank Zhou zhouxuesong@gmail.com a écrit :

Hi, David

I had installed river jdbc on node1 and node4, then, realized maybe multiple install cause problem, I removed node4 installation by removed plugin/river-jdbc directory, and restart node4.

I did try to use plugin head to pre-create river with 1 shard 1 replica, same error
Then, use the cluster default 5 shards 1 replica, same error,

This command I used to create type under _river index

curl -XPUT 'localhost:9200/_river/my_jdbc_river2/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "simple",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/tt",
"user" : "abc",
"autocommit" : true,
"password" : "abc123",
"sql" : "select ID as _id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID from SMSMT"
},
"index" : {
"index" : "test",
"type" : "tt",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

I did try my_jdbc_river and my_jdbc_river2 two types, both of them get the same error.

Thanks
Frank

On Sun, Dec 15, 2013 at 8:58 PM, David Pilato david@pilato.fr wrote:
Did you install jdbc river on all nodes?
How do you create the river?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 02:05, Frank Zhou zhouxuesong@gmail.com a écrit :

I closed target index tt, and completely restart whole cluster, then, test again with new type name my_SMSMT_river

This time, the exception changed to node6

http://devcassandra1:9200/_river/my_SMSMT_river/_status?pretty=true
{
"_index" : "_river",
"_type" : "my_SMSMT_river",
"_id" : "_status",
"_version" : 1,
"exists" : true, "_source" : {"error":"NoClassSettingsException[Failed to load class with value [jdbc]]; nested: ClassNotFoundException[jdbc]; ","node":{"id":"MMDEBAm5RNG9ddidUlhM8g","name":"devES1-node6","transport_address":"inet[/172.16.12.70:9300]"}}
}

Thanks
Frank

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1cb414a7-6738-4534-85ca-8a32083f6761%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/76D3069D-1459-4D3B-8A1D-302BEC22669B%40pilato.fr.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABeXrnc5-ww4Zb1VBpU7Bt_BA%3Do__8rDREM6CjxoYT3WxBZeTw%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5CD9C017-B939-4A18-9F4C-8749C4E71350%40pilato.fr.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABeXrndeNCiTUA3iP1UG95VNhtMAtwJNopVEaOjc%3DjNs5%3DQ8Uw%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/A7E14344-78A1-4E93-8499-90B397CEB728%40pilato.fr.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABeXrnc2da_6p75caJpg6jvoNXy-WVfVniMNpfeVJ4Hw8OboFQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/793C89E5-F715-40B1-BD80-E4B79D6CB7EC%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.


(fzhou) #13

Hi, David

{

  • ok: true
  • status: 200
  • name: devES1-node1
  • version: {
    • number: 0.90.7
    • build_hash: 36897d07dadcb70886db7f149e645ed3d44eb5f2
    • build_timestamp: 2013-11-13T12:06:54Z
    • build_snapshot: false
    • lucene_version: 4.5.1
      }
  • tagline: You Know, for Search

}

Thanks
Frank

On Mon, Dec 16, 2013 at 9:51 PM, David Pilato david@pilato.fr wrote:

What is your elasticsearch version?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 17 déc. 2013 à 00:38, Frank Zhou zhouxuesong@gmail.com a écrit :

Hi, David

I installed driver on all of the data/non-data nodes, (because previously,
I just installed driver on node1 which is not data node, and it works,
indeed loaded 17million documents), then, restarted the cluster, make sure
all of the nodes loaded river driver

Then, I issue the command

curl -XPUT 'localhost:9200/_river/my_ng30-SMSMT_river_19/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "simple",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/ng19",
"user" : "abc",
"autocommit" : true,
"password" : "abc123",
"sql" : "select ID+541008901 as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test",
"type" : "ng30-SMSMT",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

Which try to continue the failed in middle work on /test/ng30-SMSMT type.

The error I immediately got is

[2013-12-16 15:16:21,477][DEBUG][action.admin.cluster.node.stats]
[devES1-node1] failed to execute on node [S4g3RKOwT8K5EmNN6XtYzg]
org.elasticsearch.transport.RemoteTransportException:
[devES1-node6][inet[/172.16.12.70:9300]][cluster/nodes/stats/n]
Caused by: org.elasticsearch.index.engine.EngineClosedException: [jdbc][1]
CurrentState[CLOSED]
at
org.elasticsearch.index.engine.robin.RobinEngine.ensureOpen(RobinEngine.java:969)
at
org.elasticsearch.index.engine.robin.RobinEngine.segmentsStats(RobinEngine.java:1181)
at
org.elasticsearch.index.shard.service.InternalIndexShard.segmentStats(InternalIndexShard.java:509)
at
org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:154)
at
org.elasticsearch.indices.InternalIndicesService.stats(InternalIndicesService.java:212)
at org.elasticsearch.node.service.NodeService.stats(NodeService.java:165)
at
org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:100)
at
org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:43)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:273)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:264)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2013-12-16 15:16:21,498][DEBUG][action.admin.cluster.node.stats]
[devES1-node1] failed to execute on node [M9cRX1vIRRKKi_GGAGyd3w]
org.elasticsearch.transport.RemoteTransportException:
[devES1-node8][inet[/172.16.12.72:9300]][cluster/nodes/stats/n]
Caused by: org.elasticsearch.index.engine.EngineClosedException: [jdbc][3]
CurrentState[CLOSED]
at
org.elasticsearch.index.engine.robin.RobinEngine.ensureOpen(RobinEngine.java:969)
at
org.elasticsearch.index.engine.robin.RobinEngine.segmentsStats(RobinEngine.java:1181)
at
org.elasticsearch.index.shard.service.InternalIndexShard.segmentStats(InternalIndexShard.java:509)
at
org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:154)
at
org.elasticsearch.indices.InternalIndicesService.stats(InternalIndicesService.java:212)
at org.elasticsearch.node.service.NodeService.stats(NodeService.java:165)
at
org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:100)
at
org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:43)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:273)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:264)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

It is different than original ClassNotFoundException, it is
EnginelosedException.

I checked the index itself, including all the the shards for test, they
are OK, nothing wrong.

Thanks
Frank

On Mon, Dec 16, 2013 at 10:50 AM, David Pilato david@pilato.fr wrote:

A river can run on any data node (not sure about non data nodes though).
Because when a node stops, river needs to be started on another one. It
means that rivers are replicated.

That's the reason you need to have all plugins installed on each node.

Makes sense?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 19:01, Frank Zhou zhouxuesong@gmail.com a écrit :

Oh! First time know I should install driver on all of the nodes. I will
do it and want to know why

  1. Does it mean all of the data/non-data nodes have ability to directly
    connect to source mysql database and fetch data?
  2. Why one node driver installation can support the new index type
    creation without problem?

Thanks

On Sun, Dec 15, 2013 at 10:01 PM, David Pilato david@pilato.fr wrote:

My question was: "Did you install jdbc river on ALL nodes?"
I wrote you have 10 nodes, right?

First install the river on all of them.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 06:04, Frank Zhou zhouxuesong@gmail.com a écrit :

Hi, David

I had installed river jdbc on node1 and node4, then, realized maybe
multiple install cause problem, I removed node4 installation by removed
plugin/river-jdbc directory, and restart node4.

I did try to use plugin head to pre-create river with 1 shard 1 replica,
same error
Then, use the cluster default 5 shards 1 replica, same error,

This command I used to create type under _river index

curl -XPUT 'localhost:9200/_river/my_jdbc_river2/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "simple",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/tt",
"user" : "abc",
"autocommit" : true,
"password" : "abc123",
"sql" : "select ID as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test",
"type" : "tt",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

I did try my_jdbc_river and my_jdbc_river2 two types, both of them get
the same error.

Thanks
Frank

On Sun, Dec 15, 2013 at 8:58 PM, David Pilato david@pilato.fr wrote:

Did you install jdbc river on all nodes?
How do you create the river?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 02:05, Frank Zhou zhouxuesong@gmail.com a écrit :

I closed target index tt, and completely restart whole cluster, then,
test again with new type name my_SMSMT_river

This time, the exception changed to node6

http://devcassandra1:9200/_river/my_SMSMT_river/_status?pretty=true

{
"_index" : "_river",
"_type" : "my_SMSMT_river",
"_id" : "_status",
"_version" : 1,
"exists" : true, "_source" : {"error":"NoClassSettingsException[Failed to load class with value [jdbc]]; nested: ClassNotFoundException[jdbc]; ","node":{"id":"MMDEBAm5RNG9ddidUlhM8g","name":"devES1-node6","transport_address":"inet[/172.16.12.70:9300]"}}
}

Thanks

Frank

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1cb414a7-6738-4534-85ca-8a32083f6761%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/76D3069D-1459-4D3B-8A1D-302BEC22669B%40pilato.fr
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CABeXrnc5-ww4Zb1VBpU7Bt_BA%3Do__8rDREM6CjxoYT3WxBZeTw%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5CD9C017-B939-4A18-9F4C-8749C4E71350%40pilato.fr
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CABeXrndeNCiTUA3iP1UG95VNhtMAtwJNopVEaOjc%3DjNs5%3DQ8Uw%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/A7E14344-78A1-4E93-8499-90B397CEB728%40pilato.fr
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CABeXrnc2da_6p75caJpg6jvoNXy-WVfVniMNpfeVJ4Hw8OboFQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/793C89E5-F715-40B1-BD80-E4B79D6CB7EC%40pilato.fr
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABeXrncQne1Hgck%2Bs%3DnyR6Tg4MavAUzAVxQj1xgrqMOG6Ys2TQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #14

Have a look at this thread: https://groups.google.com/d/msg/elasticsearch/ImLCIV7Grys/F_k7IlEgmhEJ

It's a DEBUG trace which should not appear. But I guess everything is working now, right?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 17 décembre 2013 at 07:22:40, Frank Zhou (zhouxuesong@gmail.com) a écrit:

Hi, David

{
ok: true
status: 200
name: devES1-node1
version: {
number: 0.90.7
build_hash: 36897d07dadcb70886db7f149e645ed3d44eb5f2
build_timestamp: 2013-11-13T12:06:54Z
build_snapshot: false
lucene_version: 4.5.1
}
tagline: You Know, for Search
}

Thanks
Frank

On Mon, Dec 16, 2013 at 9:51 PM, David Pilato david@pilato.fr wrote:
What is your elasticsearch version?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 17 déc. 2013 à 00:38, Frank Zhou zhouxuesong@gmail.com a écrit :

Hi, David

I installed driver on all of the data/non-data nodes, (because previously, I just installed driver on node1 which is not data node, and it works, indeed loaded 17million documents), then, restarted the cluster, make sure all of the nodes loaded river driver

Then, I issue the command

curl -XPUT 'localhost:9200/_river/my_ng30-SMSMT_river_19/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "simple",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/ng19",
"user" : "abc",
"autocommit" : true,
"password" : "abc123",
"sql" : "select ID+541008901 as _id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID from SMSMT"
},
"index" : {
"index" : "test",
"type" : "ng30-SMSMT",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

Which try to continue the failed in middle work on /test/ng30-SMSMT type.

The error I immediately got is

[2013-12-16 15:16:21,477][DEBUG][action.admin.cluster.node.stats] [devES1-node1] failed to execute on node [S4g3RKOwT8K5EmNN6XtYzg]
org.elasticsearch.transport.RemoteTransportException: [devES1-node6][inet[/172.16.12.70:9300]][cluster/nodes/stats/n]
Caused by: org.elasticsearch.index.engine.EngineClosedException: [jdbc][1] CurrentState[CLOSED]
at org.elasticsearch.index.engine.robin.RobinEngine.ensureOpen(RobinEngine.java:969)
at org.elasticsearch.index.engine.robin.RobinEngine.segmentsStats(RobinEngine.java:1181)
at org.elasticsearch.index.shard.service.InternalIndexShard.segmentStats(InternalIndexShard.java:509)
at org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:154)
at org.elasticsearch.indices.InternalIndicesService.stats(InternalIndicesService.java:212)
at org.elasticsearch.node.service.NodeService.stats(NodeService.java:165)
at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:100)
at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:43)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:273)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:264)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2013-12-16 15:16:21,498][DEBUG][action.admin.cluster.node.stats] [devES1-node1] failed to execute on node [M9cRX1vIRRKKi_GGAGyd3w]
org.elasticsearch.transport.RemoteTransportException: [devES1-node8][inet[/172.16.12.72:9300]][cluster/nodes/stats/n]
Caused by: org.elasticsearch.index.engine.EngineClosedException: [jdbc][3] CurrentState[CLOSED]
at org.elasticsearch.index.engine.robin.RobinEngine.ensureOpen(RobinEngine.java:969)
at org.elasticsearch.index.engine.robin.RobinEngine.segmentsStats(RobinEngine.java:1181)
at org.elasticsearch.index.shard.service.InternalIndexShard.segmentStats(InternalIndexShard.java:509)
at org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:154)
at org.elasticsearch.indices.InternalIndicesService.stats(InternalIndicesService.java:212)
at org.elasticsearch.node.service.NodeService.stats(NodeService.java:165)
at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:100)
at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:43)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:273)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:264)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

It is different than original ClassNotFoundException, it is EnginelosedException.

I checked the index itself, including all the the shards for test, they are OK, nothing wrong.

Thanks
Frank

On Mon, Dec 16, 2013 at 10:50 AM, David Pilato david@pilato.fr wrote:
A river can run on any data node (not sure about non data nodes though).
Because when a node stops, river needs to be started on another one. It means that rivers are replicated.

That's the reason you need to have all plugins installed on each node.

Makes sense?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 19:01, Frank Zhou zhouxuesong@gmail.com a écrit :

Oh! First time know I should install driver on all of the nodes. I will do it and want to know why

  1. Does it mean all of the data/non-data nodes have ability to directly connect to source mysql database and fetch data?
  2. Why one node driver installation can support the new index type creation without problem?

Thanks

On Sun, Dec 15, 2013 at 10:01 PM, David Pilato david@pilato.fr wrote:
My question was: "Did you install jdbc river on ALL nodes?"
I wrote you have 10 nodes, right?

First install the river on all of them.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 06:04, Frank Zhou zhouxuesong@gmail.com a écrit :

Hi, David

I had installed river jdbc on node1 and node4, then, realized maybe multiple install cause problem, I removed node4 installation by removed plugin/river-jdbc directory, and restart node4.

I did try to use plugin head to pre-create river with 1 shard 1 replica, same error
Then, use the cluster default 5 shards 1 replica, same error,

This command I used to create type under _river index

curl -XPUT 'localhost:9200/_river/my_jdbc_river2/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "simple",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/tt",
"user" : "abc",
"autocommit" : true,
"password" : "abc123",
"sql" : "select ID as _id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID from SMSMT"
},
"index" : {
"index" : "test",
"type" : "tt",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

I did try my_jdbc_river and my_jdbc_river2 two types, both of them get the same error.

Thanks
Frank

On Sun, Dec 15, 2013 at 8:58 PM, David Pilato david@pilato.fr wrote:
Did you install jdbc river on all nodes?
How do you create the river?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 déc. 2013 à 02:05, Frank Zhou zhouxuesong@gmail.com a écrit :

I closed target index tt, and completely restart whole cluster, then, test again with new type name my_SMSMT_river

This time, the exception changed to node6

http://devcassandra1:9200/_river/my_SMSMT_river/_status?pretty=true
{
"_index" : "_river",
"_type" : "my_SMSMT_river",
"_id" : "_status",
"_version" : 1,
"exists" : true, "_source" : {"error":"NoClassSettingsException[Failed to load class with value [jdbc]]; nested: ClassNotFoundException[jdbc]; ","node":{"id":"MMDEBAm5RNG9ddidUlhM8g","name":"devES1-node6","transport_address":"inet[/172.16.12.70:9300]"}}
}

Thanks
Frank

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1cb414a7-6738-4534-85ca-8a32083f6761%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/76D3069D-1459-4D3B-8A1D-302BEC22669B%40pilato.fr.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABeXrnc5-ww4Zb1VBpU7Bt_BA%3Do__8rDREM6CjxoYT3WxBZeTw%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5CD9C017-B939-4A18-9F4C-8749C4E71350%40pilato.fr.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABeXrndeNCiTUA3iP1UG95VNhtMAtwJNopVEaOjc%3DjNs5%3DQ8Uw%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/A7E14344-78A1-4E93-8499-90B397CEB728%40pilato.fr.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABeXrnc2da_6p75caJpg6jvoNXy-WVfVniMNpfeVJ4Hw8OboFQ%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/793C89E5-F715-40B1-BD80-E4B79D6CB7EC%40pilato.fr.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABeXrncQne1Hgck%2Bs%3DnyR6Tg4MavAUzAVxQj1xgrqMOG6Ys2TQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52b00075.440badfc.6956%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/groups/opt_out.


(fzhou) #15

Hi, David

I am not sure, seems like now there are 4 sessions in mysql , run my 4
queries,

| 14989 | abc | devcassandra1.dfengg.com:34544 | ng30 | Query |
2818 | Writing to net | select ID as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,Cust
|
| 16726 | abc | 172.16.12.73:46379 | ng19 | Query |
88 | Writing to net | select ID+ 541009000 as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,Camp
|
| 16757 | abc | devcassandra2.dfengg.com:50452 | ng6 | Query |
63 | Writing to net | select ID+94502600 as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,Campai
|
| 16768 | abc | devcassandra3.dfengg.com:50952 | ng35 | Query |
49 | Writing to net | select ID+1597134900 as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,Camp
|

I am using "oneshot" now, and hope those 4 queries inject to /test2/SMSMT
type. But until now, I just saw a /jdbc index got created there, and size
continue increase, nothing added into /test2/SMSMT type. It is different
than previous test which I see data in /test/ng30-SMSMT immediately after
issue simple strategy query.

Could you help me review the jdbc river commands I am using this time?

Thanks
Frank

The commands I am using this time

  1. From ng30 load into SMSMT

curl -XPUT 'localhost:9200/_river/SMSMT30/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "oneshot",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/ng30",
"user" : "d3",
"autocommit" : true,
"password" : "ondemand",
"sql" : "select ID as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test2",
"type" : "SMSMT",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

  1. From ng19 load into SMSMT

mysql> select max(id),count(1) from ng30.SMSMT;
+-----------+----------+
| max(id) | count(1) |
+-----------+----------+
| 541008901 | 22317315 |
+-----------+----------+
1 row in set (1 min 9.71 sec)

curl -XPUT 'localhost:9200/_river/SMSMT19/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "oneshot",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/ng19",
"user" : "d3",
"autocommit" : true,
"password" : "ondemand",
"sql" : "select ID+ 541009000 as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test2",
"type" : "SMSMT",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

  1. From ng6 load into SMSMT

mysql> select max(id)+541009000,count(1) from ng19.SMSMT;
+-------------------+----------+
| max(id)+541009000 | count(1) |
+-------------------+----------+
| 945025141 | 7771470 |
+-------------------+----------+

curl -XPUT 'localhost:9200/_river/SMSMT6/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "oneshot",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/ng6",
"user" : "d3",
"autocommit" : true,
"password" : "ondemand",
"sql" : "select ID+94502600 as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test2",
"type" : "SMSMT",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

  1. From ng35 load into SMSMT

mysql> select max(id)+94502600,count(1) from ng6.SMSMT;
+------------------+----------+
| max(id)+94502600 | count(1) |
+------------------+----------+
| 1597134881 | 10162439 |
+------------------+----------+
1 row in set (1 min 37.13 sec)

curl -XPUT 'localhost:9200/_river/SMSMT35/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "oneshot",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/ng35",
"user" : "d3",
"autocommit" : true,
"password" : "ondemand",
"sql" : "select ID+1597134900 as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test2",
"type" : "SMSMT",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2a3dc22b-b7d5-4ee4-98a4-f2075ff41c54%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #16

Sorry. I can't. I've never played with jdbc river.
Probably Jörg could help you here.

Good luck.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 17 décembre 2013 at 08:56:31, Frank Zhou (zhouxuesong@gmail.com) a écrit:

Hi, David

I am not sure, seems like now there are 4 sessions in mysql , run my 4 queries,

| 14989 | abc | devcassandra1.dfengg.com:34544 | ng30 | Query | 2818 | Writing to net | select ID as _id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,Cust |
| 16726 | abc | 172.16.12.73:46379 | ng19 | Query | 88 | Writing to net | select ID+ 541009000 as _id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,Camp |
| 16757 | abc | devcassandra2.dfengg.com:50452 | ng6 | Query | 63 | Writing to net | select ID+94502600 as _id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,Campai |
| 16768 | abc | devcassandra3.dfengg.com:50952 | ng35 | Query | 49 | Writing to net | select ID+1597134900 as _id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,Camp |

I am using "oneshot" now, and hope those 4 queries inject to /test2/SMSMT type. But until now, I just saw a /jdbc index got created there, and size continue increase, nothing added into /test2/SMSMT type. It is different than previous test which I see data in /test/ng30-SMSMT immediately after issue simple strategy query.

Could you help me review the jdbc river commands I am using this time?

Thanks
Frank

The commands I am using this time

  1. From ng30 load into SMSMT

curl -XPUT 'localhost:9200/_river/SMSMT30/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "oneshot",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/ng30",
"user" : "d3",
"autocommit" : true,
"password" : "ondemand",
"sql" : "select ID as _id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID from SMSMT"
},
"index" : {
"index" : "test2",
"type" : "SMSMT",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

  1. From ng19 load into SMSMT

mysql> select max(id),count(1) from ng30.SMSMT;
+-----------+----------+
| max(id) | count(1) |
+-----------+----------+
| 541008901 | 22317315 |
+-----------+----------+
1 row in set (1 min 9.71 sec)

curl -XPUT 'localhost:9200/_river/SMSMT19/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "oneshot",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/ng19",
"user" : "d3",
"autocommit" : true,
"password" : "ondemand",
"sql" : "select ID+ 541009000 as _id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID from SMSMT"
},
"index" : {
"index" : "test2",
"type" : "SMSMT",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

  1. From ng6 load into SMSMT

mysql> select max(id)+541009000,count(1) from ng19.SMSMT;
+-------------------+----------+
| max(id)+541009000 | count(1) |
+-------------------+----------+
| 945025141 | 7771470 |
+-------------------+----------+

curl -XPUT 'localhost:9200/_river/SMSMT6/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "oneshot",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/ng6",
"user" : "d3",
"autocommit" : true,
"password" : "ondemand",
"sql" : "select ID+94502600 as _id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID from SMSMT"
},
"index" : {
"index" : "test2",
"type" : "SMSMT",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

  1. From ng35 load into SMSMT

mysql> select max(id)+94502600,count(1) from ng6.SMSMT;
+------------------+----------+
| max(id)+94502600 | count(1) |
+------------------+----------+
| 1597134881 | 10162439 |
+------------------+----------+
1 row in set (1 min 37.13 sec)

curl -XPUT 'localhost:9200/_river/SMSMT35/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "oneshot",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/ng35",
"user" : "d3",
"autocommit" : true,
"password" : "ondemand",
"sql" : "select ID+1597134900 as _id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID from SMSMT"
},
"index" : {
"index" : "test2",
"type" : "SMSMT",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2a3dc22b-b7d5-4ee4-98a4-f2075ff41c54%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.52b009ec.7724c67e.6956%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/groups/opt_out.


(fzhou) #17

Thank you David

I will ask for jorg about this question, actually, after river run, got
picture of shards like

[root@devcassandra1 ~]# es shards
_river 0 p STARTED 0 99b 99 172.16.12.73 devES1-node9
_river 0 r STARTED 0 99b 99 172.16.12.71 devES1-node7
_river 1 p STARTED 6 39.9kb 40863 172.16.12.63 devES1-node2
_river 1 r STARTED 6 39.9kb 40863 172.16.12.72 devES1-node8
_river 2 p STARTED 0 99b 99 172.16.12.65 devES1-node3
_river 2 r STARTED 0 99b 99 172.16.12.70 devES1-node6
_river 3 p STARTED 11 21.9kb 22459 172.16.12.66 devES1-node4
_river 3 r STARTED 10 19.1kb 19636 172.16.12.71 devES1-node7
_river 4 p STARTED 0 99b 99 172.16.12.67 devES1-node5
_river 4 r STARTED 0 99b 99 172.16.12.72 devES1-node8
jdbc 0 r STARTED 851090 10.2gb 10957897275 172.16.12.67 devES1-node5
jdbc 0 p STARTED 851187 10gb 10807739448 172.16.12.70 devES1-node6
jdbc 1 p STARTED 852096 9gb 9700589839 172.16.12.73 devES1-node9
jdbc 1 r STARTED 851260 8.9gb 9649676808 172.16.12.71 devES1-node7
jdbc 2 p STARTED 832711 8.7gb 9426796249 172.16.12.63 devES1-node2
jdbc 2 r STARTED 833427 8.6gb 9258395002 172.16.12.72 devES1-node8
jdbc 3 p STARTED 832805 10.8gb 11699736047 172.16.12.65 devES1-node3
jdbc 3 r STARTED 826966 10gb 10790370783 172.16.12.67 devES1-node5
jdbc 4 p STARTED 846341 10.9gb 11777169838 172.16.12.66 devES1-node4
jdbc 4 r STARTED 843730 8.8gb 9503302370 172.16.12.70 devES1-node6
test2 0 p STARTED 0 99b 99 172.16.12.73 devES1-node9
test2 0 r STARTED 0 99b 99 172.16.12.63 devES1-node2
test2 1 r STARTED 0 99b 99 172.16.12.65 devES1-node3
test2 1 p STARTED 0 99b 99 172.16.12.63 devES1-node2
test2 2 p STARTED 0 99b 99 172.16.12.65 devES1-node3
test2 2 r STARTED 0 99b 99 172.16.12.66 devES1-node4
test2 3 r STARTED 0 99b 99 172.16.12.67 devES1-node5
test2 3 p STARTED 0 99b 99 172.16.12.66 devES1-node4
test2 4 p STARTED 0 99b 99 172.16.12.67 devES1-node5
test2 4 r STARTED 0 99b 99 172.16.12.71 devES1-node7
test2 5 p STARTED 0 99b 99 172.16.12.71 devES1-node7
test2 5 r STARTED 0 99b 99 172.16.12.72 devES1-node8
test2 6 p STARTED 0 99b 99 172.16.12.72 devES1-node8
test2 6 r STARTED 0 99b 99 172.16.12.70 devES1-node6
test2 7 r STARTED 0 99b 99 172.16.12.73 devES1-node9
test2 7 p STARTED 0 99b 99 172.16.12.70 devES1-node6
test3 0 p STARTED 0 99b 99 172.16.12.73 devES1-node9
test3 0 r STARTED 0 79b 79 172.16.12.63 devES1-node2

It always go to jdbc index instead of test2, hmmm

Anyway, thank you very much!
Frank

On Tue, Dec 17, 2013 at 12:23 AM, David Pilato david@pilato.fr wrote:

Sorry. I can't. I've never played with jdbc river.
Probably Jörg could help you here.

Good luck.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 17 décembre 2013 at 08:56:31, Frank Zhou (zhouxuesong@gmail.com//zhouxuesong@gmail.com)
a écrit:

Hi, David

I am not sure, seems like now there are 4 sessions in mysql , run my 4
queries,

| 14989 | abc | devcassandra1.dfengg.com:34544 | ng30 | Query
| 2818 | Writing to net | select ID as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,Cust
|
| 16726 | abc | 172.16.12.73:46379 | ng19 | Query
| 88 | Writing to net | select ID+ 541009000 as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,Camp
|
| 16757 | abc | devcassandra2.dfengg.com:50452 | ng6 | Query
| 63 | Writing to net | select ID+94502600 as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,Campai
|
| 16768 | abc | devcassandra3.dfengg.com:50952 | ng35 | Query
| 49 | Writing to net | select ID+1597134900 as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,Camp
|

I am using "oneshot" now, and hope those 4 queries inject to /test2/SMSMT
type. But until now, I just saw a /jdbc index got created there, and size
continue increase, nothing added into /test2/SMSMT type. It is different
than previous test which I see data in /test/ng30-SMSMT immediately after
issue simple strategy query.

Could you help me review the jdbc river commands I am using this time?

Thanks
Frank

The commands I am using this time

  1. From ng30 load into SMSMT

curl -XPUT 'localhost:9200/_river/SMSMT30/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "oneshot",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/ng30",
"user" : "d3",
"autocommit" : true,
"password" : "ondemand",
"sql" : "select ID as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test2",
"type" : "SMSMT",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

  1. From ng19 load into SMSMT

mysql> select max(id),count(1) from ng30.SMSMT;
+-----------+----------+
| max(id) | count(1) |
+-----------+----------+
| 541008901 | 22317315 |
+-----------+----------+
1 row in set (1 min 9.71 sec)

curl -XPUT 'localhost:9200/_river/SMSMT19/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "oneshot",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/ng19",
"user" : "d3",
"autocommit" : true,
"password" : "ondemand",
"sql" : "select ID+ 541009000 as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test2",
"type" : "SMSMT",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

  1. From ng6 load into SMSMT

mysql> select max(id)+541009000,count(1) from ng19.SMSMT;
+-------------------+----------+
| max(id)+541009000 | count(1) |
+-------------------+----------+
| 945025141 | 7771470 |
+-------------------+----------+

curl -XPUT 'localhost:9200/_river/SMSMT6/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "oneshot",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/ng6",
"user" : "d3",
"autocommit" : true,
"password" : "ondemand",
"sql" : "select ID+94502600 as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test2",
"type" : "SMSMT",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

  1. From ng35 load into SMSMT

mysql> select max(id)+94502600,count(1) from ng6.SMSMT;
+------------------+----------+
| max(id)+94502600 | count(1) |
+------------------+----------+
| 1597134881 | 10162439 |
+------------------+----------+
1 row in set (1 min 37.13 sec)

curl -XPUT 'localhost:9200/_river/SMSMT35/_meta' -d '
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "oneshot",
"poll" : "1m",
"max_retries" : 5,
"fetchsize" : 100,
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://172.16.2.95:3306/ng35",
"user" : "d3",
"autocommit" : true,
"password" : "ondemand",
"sql" : "select ID+1597134900 as
_id,BusinessId,MessageSent,MessageTo,Type,MessageFrom,CallBack,Protocol,CampaignId,CustomerId,Industry,Subject,MessageText,FormattedMessageText,AppointmentId,ScheduledDeliveryTime,ActualDeliveryTime,ReadTime,ConfirmedTime,LatestDeliveryTime,MessageId,MessageStatusDescription,MessageStatus,Status,CreatedDate,CreatedUserID,LastModifiedDate,LastModifiedUserID,ScheduledTime,VisitId,BouncedTime,AssetId,DeliveryTime,ScompTime,ScompText,BouncedText,StructuredMessageText,ReferralID,AssetScheduleId,LastVisitTime,AppointmentHotlistId,EntityID
from SMSMT"
},
"index" : {
"index" : "test2",
"type" : "SMSMT",
"fetchsize" : 30000,
"bulk_size": 30000,
"max_bulk_requests": 40
}
}'

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2a3dc22b-b7d5-4ee4-98a4-f2075ff41c54%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/tVi-JPlVBHs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.52b009ec.7724c67e.6956%40MacBook-Air-de-David.local
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABeXrne4O0k4BYfUe73aK34FP9MDViuJ2cAia7p%3DM%3Dz0tpyqJg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #18