# Help on ---- remote server bulk load **URL:** \ **Category:** Elasticsearch\ **Created:** [April 4, 2013, 6:04am UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445 "2013-04-04T06:04:27Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![mohammad\_rafi\_g](https://avatars.discourse-cdn.com/v4/letter/m/58956e/32.png) [@mohammad\_rafi\_g](https://discuss.elastic.co/u/mohammad_rafi_g)\ **Post date:** [April 4, 2013, 6:04am UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/1 "2013-04-04T06:04:27Z")
Hi to all. Subject: how to load bulk documents on our server machine[my company server ip add is like xxx.xxx.x.251 ] from my local system[my ip add add is xxx.xxx.x.182]. actually we need to push some data in our server[say some thing like., ip address is xxx.xxx.x.251] from my lap[my ip address is like xxx.xxx.x.182] can any one tell me how to load data in my server machine using java api. If is there any technique to do it? please tell. regards and thanks to all. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mohammad\_rafi\_g](https://avatars.discourse-cdn.com/v4/letter/m/58956e/32.png) [@mohammad\_rafi\_g](https://discuss.elastic.co/u/mohammad_rafi_g)\ **Post date:** [April 4, 2013, 7:00am UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/2 "2013-04-04T07:00:56Z")
Hi to all, ``` in my java application I have been used the following code for ``` bulk load. ......... ......... Client client = new TransportClient().addTransportAddress(new InetSocketTransportAddress("xxx.xxx.x.251", 9300)); ....... BulkRequestBuilder bulkRequest = client.prepareBulk(); bulkRequest.add(client.prepareIndex("jan2103", "cdr3",Long.toString(Id)) .setSource( jsonBuilder().startObject() .field("id", nextLine[0]) .field("Call\_source", nextLine[1]) .field("call\_destination", nextLine[2]) .field("billing\_number", nextLine[3]) .field("plan", nextLine[4]) .field("call\_type", nextLine[5]) .field("call\_start\_time", nextLine[6]) .field("call\_connected", nextLine[7]) .field("call\_ended", nextLine[8]) .field("duration", nextLine[9]) .field("amount", nextLine[10]) .endObject())); BulkResponse bulkResponse = bulkRequest.execute().actionGet(); ............ ............ When i run the above code from my system[ip address is: (like) xxx.xxx.x.182] I got the following error like: .......................................... INFO: [Blink] failed to get node info for [#transport#-1][inet[/192.168.2.251:9300]], disconnecting... org.elasticsearch.transport.NodeDisconnectedException: [][inet[/192.168.2.251:9300]][cluster/nodes/info] disconnected java.util.concurrent.ExecutionException: org.elasticsearch.client.transport.NoNodeAvailableException: No node available Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: No node available .............................................. Please Can any one tell me, what i have to do for load my data into server machine's elasticsearch? Tell me also, if, i have done wrong with my code. or is there any other way for load data into remote systems. regards and thanks to all. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 4, 2013, 7:10am UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/3 "2013-04-04T07:10:59Z")
Check your cluster name. If you changed it, follow instructions here: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. If it does not work, check your node firewall settings… HTH -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 4 avr. 2013 à 09:00, rafi [me.mrafi.en@gmail.com](mailto:me.mrafi.en@gmail.com) a écrit : > Hi to all, > > ``` > in my java application I have been used the following code for bulk load. > ......... > ......... > Client client = new TransportClient().addTransportAddress(new InetSocketTransportAddress("xxx.xxx.x.251", 9300)); > ....... > BulkRequestBuilder bulkRequest = client.prepareBulk(); > bulkRequest.add(client.prepareIndex("jan2103", "cdr3",Long.toString(Id)) > .setSource( > jsonBuilder().startObject() > .field("id", nextLine[0]) > .field("Call_source", nextLine[1]) > .field("call_destination", nextLine[2]) > .field("billing_number", nextLine[3]) > .field("plan", nextLine[4]) > .field("call_type", nextLine[5]) > .field("call_start_time", nextLine[6]) > .field("call_connected", nextLine[7]) > .field("call_ended", nextLine[8]) > .field("duration", nextLine[9]) > .field("amount", nextLine[10]) > .endObject())); > BulkResponse bulkResponse = bulkRequest.execute().actionGet(); > ............ > ............ > > ``` > > When i run the above code from my system[ip address is: (like) xxx.xxx.x.182] I got the following error like: > .......................................... > INFO: [Blink] failed to get node info for [#transport#-1][inet[/192.168.2.251:9300]], disconnecting... > org.elasticsearch.transport.NodeDisconnectedException: [inet[/192.168.2.251:9300]][cluster/nodes/info] disconnected > java.util.concurrent.ExecutionException: org.elasticsearch.client.transport.NoNodeAvailableException: No node available > Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: No node available > .............................................. > > Please Can any one tell me, what i have to do for load my data into server machine's elasticsearch? > Tell me also, if, i have done wrong with my code. or is there any other way for load data into remote systems. > > regards and thanks to all. > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 4, 2013, 7:15am UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/4 "2013-04-04T07:15:35Z")
You must tell the transport client the cluster name to connect to. Example Settings settings = ImmutableSettings.settingsBuilder() .put("cluster.name", "test") .build(); client = new TransportClient(settings); Jörg Am 04.04.13 09:00, schrieb rafi: > Client client = new TransportClient().addTransportAddress(new > InetSocketTransportAddress("xxx.xxx.x.251", 9300)); -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mohammad\_rafi\_g](https://avatars.discourse-cdn.com/v4/letter/m/58956e/32.png) [@mohammad\_rafi\_g](https://discuss.elastic.co/u/mohammad_rafi_g)\ **Post date:** [April 4, 2013, 9:41am UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/5 "2013-04-04T09:41:10Z")
hi., David and Prante Thanks for reply. As per your suggestion, > > > David > > > Check your cluster name. > > > If you changed it, follow instructions here: > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/java-api/client/#transportclient) I have checked my cluster name and check the above link. But it is not working and throwing same exception. > > > > If it does not work, check your node firewall settings… I don't know, how to check my node fire wall settings. Can you Please tell me how to check it and How to solve my problem > > > Prante > > > Settings settings = ImmutableSettings.settingsBuilder() > > > .put("cluster.name", "test").build(); > > > client = new TransportClient(settings); I also checked this code but not working. ...............Now i am sharing my source code[which deals with Elasticsearch] with full exception. public static void main(String arguments) { Long Id = new Long(1); String nextLine; Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name", "elasticsearch257").build(); Client client = new TransportClient(settings).addTransportAddress(new InetSocketTransportAddress("xxx.xxx.x.251", 9300)); BulkRequestBuilder bulkRequest = client.prepareBulk(); for (; (nextLine = reader.readNext()) != null; Id++) { bulkRequest .add(client.prepareIndex("jan2013", "cdr3",Long.toString(Id)) .setSource( jsonBuilder().startObject() .field("id", nextLine[0]) .field("Call\_source", nextLine[1]) .field("call\_destination", nextLine[2]) .field("billing\_number", nextLine[3]).endObject())); } BulkResponse bulkResponse = bulkRequest.execute().actionGet(); client.close(); ............................................................................................................................. and Exception in log is: Apr 4, 2013 2:45:58 PM org.elasticsearch.plugins INFO: [Black Panther] loaded , sites Apr 4, 2013 2:46:07 PM org.elasticsearch.client.transport INFO: [Black Panther] failed to get node info for\*[#transport#-1][inet[/xxx.xxx.x.251:9300]], - disconnecting... org.elasticsearch.transport.NodeDisconnectedException: [ine_t[/xxx.xxx.x.251:9300]][cluster/nodes/info] disconnected Apr 4, 2013 2:46:07 PM org.elasticsearch.client.transport INFO: [Black Panther] failed to get node info for [#transport#-1][inet[/xxx.xxx.x.251:9300]], disconnecting... org.elasticsearch.transport.NodeDisconnectedException: [][ine t[/xxx.xxx.x.251:9300]][cluster/nodes/info]_ disconnected java.util.concurrent.ExecutionException: org.elasticsearch.client.transport.NoNodeAvailableException: No node available at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at com.enhancesys.elasticsearch.general.dev.IndexCSVData.main(IndexCSVData.java:90) Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: No node available at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:205) at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:97) at org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:141) at org.elasticsearch.client.transport.TransportClient.bulk(TransportClient.java:328) at org.elasticsearch.action.bulk.BulkRequestBuilder.doExecute(BulkRequestBuilder.java:128) at org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:53) at org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:47) at com.enhancesys.elasticsearch.general.dev.IndexCSVDataDependent.call(IndexCSVDataDependent.java:16) at com.enhancesys.elasticsearch.general.dev.IndexCSVDataDependent.call(IndexCSVDataDependent.java:1) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ...................................................................................................................................................................................... \*This is my problem. From my system[with ip address say, xxx.xxx.x.182], I want to load data into another system[ip add say, xxx.xxx.x.251] elaestichsearch. \*I am totally confused, how to do this. So please anyone can tell me how to do load some documents from my system into another system's elasticsearch. thanks and regards to you all. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 4, 2013, 9:58am UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/6 "2013-04-04T09:58:17Z")
So your cluster name running at xxx.xxx.xxx.251 is `elasticsearch257`? To check firewall, I often simply use telnet: telnet xxx.xxx.xxx.251 9300 should give you something like: Trying ::1... Connected to localhost. Escape character is '^]'. If it's not working, then: Your node on xxx.xxx.xxx.251 is not running Your node is blocked by a firewall or any network box. -\> ask your network or system admin HTH -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 4 avr. 2013 à 11:41, rafi [me.mrafi.en@gmail.com](mailto:me.mrafi.en@gmail.com) a écrit : > hi., David and Prante > Thanks for reply. > > As per your suggestion, > > > > > David > > > > Check your cluster name. > > > > If you changed it, follow instructions here: > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/java-api/client/#transportclient) > > I have checked my cluster name and check the above link. > But it is not working and throwing same exception. > > > > > > If it does not work, check your node firewall settings… > > I don't know, how to check my node fire wall settings. > Can you Please tell me how to check it and How to solve my problem > > > > > Prante > > > > Settings settings = ImmutableSettings.settingsBuilder() > > > > .put("cluster.name", "test").build(); > > > > client = new TransportClient(settings); > > I also checked this code but not working. > > ...............Now i am sharing my source code[which deals with Elasticsearch] with full exception. > public static void main(String arguments) { > Long Id = new Long(1); > String nextLine; > Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name", "elasticsearch257").build(); > Client client = new TransportClient(settings).addTransportAddress(new InetSocketTransportAddress("xxx.xxx.x.251", 9300)); > BulkRequestBuilder bulkRequest = client.prepareBulk(); > for (; (nextLine = reader.readNext()) != null; Id++) { > bulkRequest > .add(client.prepareIndex("jan2013", "cdr3",Long.toString(Id)) > .setSource( > jsonBuilder().startObject() > .field("id", nextLine[0]) > .field("Call\_source", nextLine[1]) > .field("call\_destination", nextLine[2]) > .field("billing\_number", nextLine[3]).endObject())); } > BulkResponse bulkResponse = bulkRequest.execute().actionGet(); > client.close(); > ............................................................................................................................. > and Exception in log is: > > Apr 4, 2013 2:45:58 PM org.elasticsearch.plugins > INFO: [Black Panther] loaded , sites > Apr 4, 2013 2:46:07 PM org.elasticsearch.client.transport > INFO: [Black Panther] failed to get node info for [#transport#-1][inet[/xxx.xxx.x.251:9300]], disconnecting... > org.elasticsearch.transport.NodeDisconnectedException: [inet[/xxx.xxx.x.251:9300]][cluster/nodes/info] disconnected > Apr 4, 2013 2:46:07 PM org.elasticsearch.client.transport > INFO: [Black Panther] failed to get node info for [#transport#-1][inet[/xxx.xxx.x.251:9300]], disconnecting... > org.elasticsearch.transport.NodeDisconnectedException: [inet[/xxx.xxx.x.251:9300]][cluster/nodes/info] disconnected > java.util.concurrent.ExecutionException: org.elasticsearch.client.transport.NoNodeAvailableException: No node available > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > at com.enhancesys.elasticsearch.general.dev.IndexCSVData.main(IndexCSVData.java:90) > Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: No node available > at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:205) > at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:97) > at org.elasticsearch.client.support.AbstractClient.bulk(AbstractClient.java:141) > at org.elasticsearch.client.transport.TransportClient.bulk(TransportClient.java:328) > at org.elasticsearch.action.bulk.BulkRequestBuilder.doExecute(BulkRequestBuilder.java:128) > at org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:53) > at org.elasticsearch.action.support.BaseRequestBuilder.execute(BaseRequestBuilder.java:47) > at com.enhancesys.elasticsearch.general.dev.IndexCSVDataDependent.call(IndexCSVDataDependent.java:16) > at com.enhancesys.elasticsearch.general.dev.IndexCSVDataDependent.call(IndexCSVDataDependent.java:1) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ...................................................................................................................................................................................... > > This is my problem. > From my system[with ip address say, xxx.xxx.x.182], I want to load data into another system[ip add say, xxx.xxx.x.251] elaestichsearch. > I am totally confused, how to do this. > So please anyone can tell me how to do load some documents from my system into another system's elasticsearch. > > thanks and regards to you all. > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mohammad\_rafi\_g](https://avatars.discourse-cdn.com/v4/letter/m/58956e/32.png) [@mohammad\_rafi\_g](https://discuss.elastic.co/u/mohammad_rafi_g)\ **Post date:** [April 5, 2013, 7:18am UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/7 "2013-04-05T07:18:10Z")
hi David., Thanks for reply. I will try as per you said. If any thing new i got i will let you know. Once again thank you very much regards to all -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mohammad\_rafi\_g](https://avatars.discourse-cdn.com/v4/letter/m/58956e/32.png) [@mohammad\_rafi\_g](https://discuss.elastic.co/u/mohammad_rafi_g)\ **Post date:** [April 15, 2013, 11:37am UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/8 "2013-04-15T11:37:44Z")
Hi, David., As per you Said., > ``` > To check firewall, I often simply use telnet: > > ``` > > telnet xxx.xxx.xxx.251 9300 > should give you something like: > Trying ::1... > Connected to localhost. > Escape character is '^]'. I don't Know where I have to Check the above command. Is there any separate tool/software to test this command. Please tell me which tool is needed to test. In my office we are using PUTTY for communicating with server. But i don't know how to test my server's elasticsearch availability by using PUTTY. Once again Thanks You Very Much... Regards Rafi -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 15, 2013, 11:44am UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/9 "2013-04-15T11:44:10Z")
Please ask your system/network administrators. They should be able to help you to diagnose things. I'm afraid I can't help more here. -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 15 avr. 2013 à 13:37, rafi [me.mrafi.en@gmail.com](mailto:me.mrafi.en@gmail.com) a écrit : > Hi, David., > > As per you Said., > > > ``` > > To check firewall, I often simply use telnet: > > > > ``` > > > > telnet xxx.xxx.xxx.251 9300 > > should give you something like: > > Trying ::1... > > Connected to localhost. > > Escape character is '^]'. > > I don't Know where I have to Check the above command. > Is there any separate tool/software to test this command. > Please tell me which tool is needed to test. > > In my office we are using PUTTY for communicating with server. > But i don't know how to test my server's elasticsearch availability by using PUTTY. > > Once again Thanks You Very Much... > > Regards > Rafi > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mohammad\_rafi\_g](https://avatars.discourse-cdn.com/v4/letter/m/58956e/32.png) [@mohammad\_rafi\_g](https://discuss.elastic.co/u/mohammad_rafi_g)\ **Post date:** [April 15, 2013, 12:04pm UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/10 "2013-04-15T12:04:21Z")
Hi David. Thank You Very Much for Your Support. ok I will Check it my Own. Once again Thanks. Please Excuse me if i make you Fear. Regards Rafi -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mohammad\_rafi\_g](https://avatars.discourse-cdn.com/v4/letter/m/58956e/32.png) [@mohammad\_rafi\_g](https://discuss.elastic.co/u/mohammad_rafi_g)\ **Post date:** [May 14, 2013, 6:37am UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/11 "2013-05-14T06:37:46Z")
Hi., to all After All., Now I have able to load Bulk data from my local system to our server system. Now we are using 0.20.5V of elasticserach. and java 1.6.0\_32 java version. Actually we have implemented some application and created a Runnable Jar and then we have run that jar in server machine. now its working fine. our Application taking Average of 130sec for 1Miilion data [which has 8fields]. If Any One Knows,Plz tell me., is this the good Performance for Indexing or not? Please tell me for Indexing 1Milloin with 8 fields of data, how much time it will take? Thanks to All.. Best Regards Mohammad Rafi. On Thursday, April 4, 2013 11:34:27 AM UTC+5:30, rafi wrote: > Hi to all. > > Subject: how to load bulk documents on our server machine[my company > server ip add is like xxx.xxx.x.251 ] from my local system[my ip add add > is xxx.xxx.x.182]. > actually we need to push some data in our > server[say some thing like., ip address is xxx.xxx.x.251] from my lap[my ip > address is like xxx.xxx.x.182] > can any one tell me how to load data in my server machine using java api. > If is there any technique to do it? > please tell. > > regards and thanks to all. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 14, 2013, 6:55am UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/12 "2013-05-14T06:55:01Z")
On a single node? With standard hard disks? More than 7.5k per sec is not bad at all. You can probably increase it using more boxes (one shard per box), disabling replicas, disable refresh, use SSD drives… HTH De : [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] De la part de rafi Envoyé : mardi 14 mai 2013 08:38 À : [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Objet : Re: help on ---- remote server bulk load Hi., to all After All., Now I have able to load Bulk data from my local system to our server system. Now we are using 0.20.5V of elasticserach. and java 1.6.0\_32 java version. Actually we have implemented some application and created a Runnable Jar and then we have run that jar in server machine. now its working fine. our Application taking Average of 130sec for 1Miilion data [which has 8fields]. If Any One Knows,Plz tell me., is this the good Performance for Indexing or not? Please tell me for Indexing 1Milloin with 8 fields of data, how much time it will take? Thanks to All.. Best Regards Mohammad Rafi. On Thursday, April 4, 2013 11:34:27 AM UTC+5:30, rafi wrote: Hi to all. Subject: how to load bulk documents on our server machine[my company server ip add is like xxx.xxx.x.251 ] from my local system[my ip add add is xxx.xxx.x.182]. actually we need to push some data in our server[say some thing like., ip address is xxx.xxx.x.251] from my lap[my ip address is like xxx.xxx.x.182] can any one tell me how to load data in my server machine using java api. If is there any technique to do it? please tell. regards and thanks to all. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mohammad\_rafi\_g](https://avatars.discourse-cdn.com/v4/letter/m/58956e/32.png) [@mohammad\_rafi\_g](https://discuss.elastic.co/u/mohammad_rafi_g)\ **Post date:** [May 14, 2013, 8:18am UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/13 "2013-05-14T08:18:36Z")
Hi, David Pilato., > > On a single node? With standard hard disks? > > More than 7.5k per sec is not bad at all. Actually We Have two Nodes[both are master nodes only] Running in our Server and our hard disks is SSD Only. we have 8 shards and 0 replicas. and with bulk size is: 8k [we have tried for 1k, 4k ,10k, 12k also but at 8k we we have good speed with minimum resource\] For the above Configuration also is our Indexing speed good? Now i am sharing a peace of code of our Application ................................................................................................................................. ``` XContentBuilder builder = XContentFactory.jsonBuilder(); builder.startObject(); int position = 0; for (Object fieldName : csvFields) { builder.field((String) fieldName, ``` nextLine[position++]); } builder.endObject(); bulkRequest.add(Requests.indexRequest("cdrdata").type("cdr") .id(nextLine[0]).create(true).source(builder)); //here we have one method like " .setRefresh(true) " but we are not using refresh here ``` //here we have taken sleep method for pause the index operation ``` for few seconds like., ``` try { Thread.sleep(1000); // ``` because if we not pause the indexing operation the ES node Heap size is Incrementing up to its max size[5gb] and } catch (Exception e) { } // causing NoNodeException or Some thing Like., TimeOut Exception ``` bulkRequest.execute(new ActionListener() { @Override public void onResponse(BulkResponse bulkResponse) { ``` //some print stmts } @Override public void onFailure(Throwable e) {// error print stmts } }); ..................................................................................................................... here., Is there any thing to change for boost the speed of Indexing Operation.. > > You can probably increase it using more boxes (one shard per box), > > disabling replicas, disable refresh, use SSD drives… have we 8 shards and 0 replicas. and we have not use any refresh while doing bulk Indexing.. But I don't know is the refreshing operation enable default or not? if the refresh operation is default then how to disable it? > > You can probably increase it using more boxes (one shard per box) And You have mentioned here use One Shard per Box ., i clearly not able to understand what is box here, is it server or node? Also give me any suggestions to speed up my indexing operation. Once again thanks for replying., David Pilato Best Regards Mohammad Rafi. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 14, 2013, 8:35am UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/14 "2013-05-14T08:35:28Z")
IMHO, the best you can give to ES is to have one shard per node and one node per server. So, with 8 shards, you can have up to 8 nodes so up to 8 servers. That said, you can try to tune it and start with only two shards as you have two nodes. Then increase the number of shards, reindex and see where it goes. Don't run more than one node on a single server (or only if you have more than 64Gb of RAM). Refresh is enabled by default. Every second. To disable it: See the middle of this page (Bulk Indexing Usage): [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings/) Does it help? -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 14 mai 2013 à 10:18, rafi [me.mrafi.en@gmail.com](mailto:me.mrafi.en@gmail.com) a écrit : > Hi, David Pilato., > > > > On a single node? With standard hard disks? > > > More than 7.5k per sec is not bad at all. > > Actually We Have two Nodes[both are master nodes only] Running in our Server and our hard disks is SSD Only. > > we have 8 shards and 0 replicas. and with bulk size is: 8k > > [we have tried for 1k, 4k ,10k, 12k also but at 8k we we have good speed with minimum resource\] > > For the above Configuration also is our Indexing speed good? > > Now i am sharing a peace of code of our Application > > ................................................................................................................................. > > ``` > XContentBuilder builder = XContentFactory.jsonBuilder(); > builder.startObject(); > int position = 0; > for (Object fieldName : csvFields) { > builder.field((String) fieldName, nextLine[position++]); } > builder.endObject(); > bulkRequest.add(Requests.indexRequest("cdrdata").type("cdr") > .id(nextLine[0]).create(true).source(builder)); //here we have one method like " .setRefresh(true) " but we are not using refresh here > > //here we have taken sleep method for pause the index operation for few seconds like., > > try { Thread.sleep(1000); // because if we not pause the indexing operation the ES node Heap size is Incrementing up to its max size[5gb] and > } catch (Exception e) { } // causing NoNodeException or Some thing Like., TimeOut Exception > > bulkRequest.execute(new ActionListener() { > @Override > public void onResponse(BulkResponse bulkResponse) { //some print stmts } > @Override > public void onFailure(Throwable e) {// error print stmts } > }); > > ``` > > ..................................................................................................................... > > here., Is there any thing to change for boost the speed of Indexing Operation.. > > > > You can probably increase it using more boxes (one shard per box), disabling replicas, disable refresh, use SSD drives… > > have we 8 shards and 0 replicas. and we have not use any refresh while doing bulk Indexing.. > > But I don't know is the refreshing operation enable default or not? if the refresh operation is default then how to disable it? > > > > You can probably increase it using more boxes (one shard per box) > > And You have mentioned here use One Shard per Box ., i clearly not able to understand what is box here, is it server or node? > > Also give me any suggestions to speed up my indexing operation. > > Once again thanks for replying., David Pilato > > Best Regards > > Mohammad Rafi. > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mohammad\_rafi\_g](https://avatars.discourse-cdn.com/v4/letter/m/58956e/32.png) [@mohammad\_rafi\_g](https://discuss.elastic.co/u/mohammad_rafi_g)\ **Post date:** [May 14, 2013, 9:40am UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/15 "2013-05-14T09:40:54Z")
Hi, David Pilato Thanks a Lot for Replying. and your posts are very useful to me. > > IMHO, the best you can give to ES is to have one shard per node and one > > node per server. > > So, with 8 shards, you can have up to 8 nodes so up to 8 servers. > > Don't run more than one node on a single server (or only if you have more > > than 64Gb of RAM). ``` Ok David., Now I got some clear idea on it. ``` Actually., The Server Machine which we are using has 64Gb of Ram but as of Our Works we have Splited our Server into approximately 10 Virtual Machines And now, our server has 10Gb which we are using for ES Activities > > That said, you can try to tune it and start with only two shards as you > > have two nodes. Then increase the number of shards, reindex and see where > > it goes. ``` So now we will use Only One Node[that means we will run only one ``` ES Instance] and we will prepare some statistics for "having 1shard, 2 shards, .. for testing the speed of index operation". And I will Let U Know what happen? Thanks again for Replying. Best Regards Mohammad Rafi. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 14, 2013, 9:46am UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/16 "2013-05-14T09:46:46Z")
Yes please. Will be happy to hear news from your tests. About Virtual machines, I definitely prefer having one single machine per hardware box. You can start a JVM with 30Gb RAM instead of 10 VM with smaller RAM. Also, don't forget that in case of a crash (hard disk failure for example), you will probably lost all your replicas as well (as they are on the same physical box - even if it's another _virtual node_)… -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 14 mai 2013 à 11:40, rafi [me.mrafi.en@gmail.com](mailto:me.mrafi.en@gmail.com) a écrit : > Hi, David Pilato > Thanks a Lot for Replying. and your posts are very useful to me. > > > > IMHO, the best you can give to ES is to have one shard per node and one node per server. > > > So, with 8 shards, you can have up to 8 nodes so up to 8 servers. > > > Don't run more than one node on a single server (or only if you have more than 64Gb of RAM). > > ``` > Ok David., Now I got some clear idea on it. > > ``` > > Actually., The Server Machine which we are using has 64Gb of Ram but as of Our Works we have Splited our Server into approximately 10 Virtual Machines > And now, our server has 10Gb which we are using for ES Activities > > > > That said, you can try to tune it and start with only two shards as you have two nodes. Then increase the number of shards, reindex and see where it goes. > > ``` > So now we will use Only One Node[that means we will run only one ES Instance] and we will prepare some statistics for "having 1shard, 2 shards, .. for testing the speed of index operation". > And I will Let U Know what happen? > > ``` > > Thanks again for Replying. > > Best Regards > Mohammad Rafi. > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mohammad\_rafi\_g](https://avatars.discourse-cdn.com/v4/letter/m/58956e/32.png) [@mohammad\_rafi\_g](https://discuss.elastic.co/u/mohammad_rafi_g)\ **Post date:** [May 27, 2013, 1:57pm UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/17 "2013-05-27T13:57:46Z")
Hi., David Pilato > > Yes please. Will be happy to hear news from your tests. Actually We Start our Elasticserach server before 30-40 days. now, if we load 10million of data it is taking average of 130seconds for index 1Million data. ``` After all have restart the elasticserach server [what we do is, ``` we just shutdown the ES server and again started the ES server.] I have tested with differnt options and got different timings for index 10 million of data [with 8 fields]. After REstating server., now, if we load 10million of data it is taking average of 60seconds for index 1Million data.[with 2 nodes and index has 8shards and 0 replica with refresh ] if we take like., 8 shards 0 replica with refresh with 2 nodes -----taking avg of 60 sec for indexing 1 million of data [which have 8 fields] 8 shards 0 replica with out refresh with 2 nodes-----taking much time for indexing 1 million of data [which have 8 fields]\ 8 shards 0 replica with refresh with 1 nodes -----taking much time for indexing 1 million of data\ 1 shard 0 replica with refresh with 2 nodes -----taking much time for indexing 1 million of data [which have 8 fields]\ 2 shard 0 replica with refresh with 2 nodes -----taking much time for indexing 1 million of data [which have 8 fields]\ 3 shard 0 replica with refresh with 2 nodes-----taking much time for indexing 1 million of data [which have 8 fields]\ 6 shard 0 replica with refresh with 2 nodes------taking avg of 60 sec for indexing 1 million of data [which have 8 fields] So thus We have found that with our server we have the best performance on 6-8shards with 2 nodes. Now i have one question in my mind. Why our ES Server was indexed an avg speed of 130sec for 1Millon before Restarting ES Server. And why it has taken only 60sec after restarting ES Server? Please Tell me if You know, Is there any Reason behind this? and Is there any Option to fix this with out restarting ES server? Thanks and Regards, Mohammad Rafi -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:34am UTC](https://discuss.elastic.co/t/help-on-remote-server-bulk-load/11445/18 "2017-07-06T02:34:34Z")