# Monitoring, Backup & Restore Tools for Elastic search **URL:** \ **Category:** Elasticsearch\ **Created:** [July 31, 2013, 2:50pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015 "2013-07-31T14:50:56Z")\ **Posts on this page:** 15\ **Page:** 2
**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\ **Post date:** [September 6, 2013, 11:24pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/21 "2013-09-06T23:24:55Z")
Craig, By stating the phrase "100m files", do you mean "100 million individual files" or do you mean "100-megabyte-sized files"? I suspect the latter (e.g. a single file of 100 megabytes in size) and not the former (100 million directory entries of type 'file'). Brian On Friday, September 6, 2013 9:54:37 AM UTC-4, Craig Lebowitz wrote: > Jorg one question for you: if I extract the knapsack archive, will the > file system have a problem with 100m files? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Graham\_Lenton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/graham_lenton/32/860_2.png) [@Graham\_Lenton](https://discuss.elastic.co/u/Graham_Lenton)\ **Post date:** [September 7, 2013, 9:57am UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/22 "2013-09-07T09:57:54Z")
Re monitoring tools. The ones you listed are all free and do a good job. Elasticsearch-head is especially good for seeing the cluster health and shard allocation. It's browse mode is good for inspecting a particular documents \_source. Bigdesk is great for real time performance data - which is collected for each node, but only viewable one at a time (in a single window). You could run multiple instances, at the cost of increasing the monitoring load upon your cluster. The history is stored only in your browser, so it's impossible to see what happened before. Paramedic and ElasticHQ are both somewhat similar in their functionality to Bigdesk, I haven't used them too much although both have nice GUI. For serious production use I would highly recommend Sematext SPM. It is a commercial product, where you run JMX agents and a 'sender' service on your nodes, which uploads the metrics into the sematext cloud. This gives you the advantage of having a proper history available. Their pricing is pretty reasonable for the peace of mind you have. Check out [Elasticsearch Monitoring](http://sematext.com/spm/elasticsearch-performance-monitoring/) On Wednesday, 31 July 2013 15:50:56 UTC+1, pranav amin wrote: > Hi There, > > I'm a new user for Elastic search and plan to use it in Production > environment. > > Can you please help me with some information - > > Monitoring tools - > I looked at the following. > a) Elasticsearch head > b) Bigdesk > c) Paramedic > d) ElasticHQ > > Not sure which of the above are free and recommend by others? > > Backup & Restore - > a) Can i do backup and restore of a Node in a Cluster? Are there any > free/cost based tools (UI or Script based) that can do this and are > frequently recommended by others? > b) Are there any free/cost based tools (UI or Script based) that can do > backup & restore of ACTIVE AND NON-ACTIVE cluster? > > Thanks for the help and advice. > Much appreciated. > > Thanks > Pranav. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Adrian\_2](https://avatars.discourse-cdn.com/v4/letter/a/c57346/32.png) [@Adrian\_2](https://discuss.elastic.co/u/Adrian_2)\ **Post date:** [September 9, 2013, 7:22pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/23 "2013-09-09T19:22:23Z")
On Fri, Sep 06, 2013 at 10:59:27PM +0200, [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: Jörg, > Good question, I think Linux ext4 can handle 100m dir entries quite well if > it was configured to do so - but, many file based tools will not work Ext4 has no limit on directory entries. The limit for directory entries on ext3 is 32k. > properly with 100m dir entries, for example ls or rm -rf. Either your main > memory will blow up, or wait times will become excessively long. This would be basically because the shell will expand the command if wildcards are used. You can curcumvent this using other tools. Just have a look at find and xargs if the shell is unable to handle the argument list. Best regards, Adrian -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 10, 2013, 1:02am UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/24 "2013-09-10T01:02:52Z")
You might get surprised that even xargs has arg list limits. Check this with getconf ARG\_MAX Most Linux systems will return 2621440 which is far from being able to handle 100m dir entries via arg lists. Just my 2c. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pranav\_amin](https://avatars.discourse-cdn.com/v4/letter/p/a183cd/32.png) [@pranav\_amin](https://discuss.elastic.co/u/pranav_amin)\ **Post date:** [September 13, 2013, 9:09pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/25 "2013-09-13T21:09:12Z")
thanks for recommending. when i try to use this it complains - local@slsdev-int-09u:/mnt/sl/elastic/elasticsearch-0.90.1$ curl -XPOST 5.36.242.139:9200/test/test/\_export [2013-09-13 21:08:13,612][INFO][cluster.service] [Pranav] detected\_master [Paul][PoBefbfWQ6K9XRJraJF0hA][inet[/5.36.242.138:9300]]{master=true}, added {[Tom][3SrFwYh8RmqyVkbhHfqyzQ][inet[/5.36.242.140:9300 ]]{master=true},[Paul][PoBefbfWQ6K9XRJraJF0hA][inet[/5.36.242.138:9300]]{master=true},}, reason: zen-disco-receive(from master [[Paul][PoBefbfWQ6K9XRJraJF0hA][inet[/5.36.242.138:9300]]{master=true}]) {"error":"RemoteTransportException[[Paul][inet[/5.36.242.138:9300]][index]]; nested: MapperParsingException[failed to parse, document is empty]; ","status":400}local@slsdev-int-09u:/mnt/sl/elastic/elasticsearch-0.90.1$ Any idea what the above error means? Thanks in advance. On Wed, Sep 4, 2013 at 11:42 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > You can try my knapsack plugin which has a "size" URL parameter to control > the bulk export. > > [GitHub - jprante/elasticsearch-knapsack: Knapsack plugin is an import/export tool for Elasticsearch](https://github.com/jprante/elasticsearch-knapsack) > > Jörg > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pranav\_amin](https://avatars.discourse-cdn.com/v4/letter/p/a183cd/32.png) [@pranav\_amin](https://discuss.elastic.co/u/pranav_amin)\ **Post date:** [September 13, 2013, 9:14pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/26 "2013-09-13T21:14:19Z")
fyi - i have data in the test indice so not sure why the error says document is empty. There are two records i added. local@slsdev-int-09u:/mnt/sl/elastic/elasticsearch-0.90.1$ curl -XGET 5.36.242.140:9200/test/test/\_search {"took":26,"timed\_out":false,"\_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":2,"max\_score":1.0,"hits":[{"\_index":"test","\_type":"test","\_id":"1","\_score":1.0, "\_source" : {"key":"value 1"}},{"\_index":"test","\_type":"test","\_id":"2","\_score":1.0, "\_source" : {"key":"value 2"}}]}}local@slsdev-int-09 On Fri, Sep 13, 2013 at 5:09 PM, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) wrote: > thanks for recommending. > > when i try to use this it complains - > > local@slsdev-int-09u:/mnt/sl/elastic/elasticsearch-0.90.1$ curl -XPOST > 5.36.242.139:9200/test/test/\_export > [2013-09-13 21:08:13,612][INFO][cluster.service] [Pranav] > detected\_master [Paul][PoBefbfWQ6K9XRJraJF0hA][inet[/5.36.242.138:9300]]{master=true}, > added {[Tom][3SrFwYh8RmqyVkbhHfqyzQ][inet[/5.36.242.140:9300 > ]]{master=true},[Paul][PoBefbfWQ6K9XRJraJF0hA][inet[/5.36.242.138:9300]]{master=true},}, > reason: zen-disco-receive(from master > [[Paul][PoBefbfWQ6K9XRJraJF0hA][inet[/5.36.242.138:9300]]{master=true}]) > {"error":"RemoteTransportException[[Paul][inet[/5.36.242.138:9300]][index]]; > nested: MapperParsingException[failed to parse, document is empty]; > ","status":400}local@slsdev-int-09u:/mnt/sl/elastic/elasticsearch-0.90.1$ > > Any idea what the above error means? > > Thanks in advance. > > On Wed, Sep 4, 2013 at 11:42 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > You can try my knapsack plugin which has a "size" URL parameter to > > control the bulk export. > > > > [GitHub - jprante/elasticsearch-knapsack: Knapsack plugin is an import/export tool for Elasticsearch](https://github.com/jprante/elasticsearch-knapsack) > > > > Jörg > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 14, 2013, 6:11am UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/27 "2013-09-14T06:11:42Z")
I think the plugin is not loaded. Did you restart es? Can you see in logs at the begining that the plugin is loaded? -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 13 sept. 2013 à 23:09, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) a écrit : thanks for recommending. when i try to use this it complains - local@slsdev-int-09u:/mnt/sl/elastic/elasticsearch-0.90.1$ curl -XPOST 5.36.242.139:9200/test/test/\_export [2013-09-13 21:08:13,612][INFO][cluster.service] [Pranav] detected\_master [Paul][PoBefbfWQ6K9XRJraJF0hA][inet[/5.36.242.138:9300]]{master=true}, added {[Tom][3SrFwYh8RmqyVkbhHfqyzQ][inet[/5.36.242.140:9300]]{master=true},[Paul][PoBefbfWQ6K9XRJraJF0hA][inet[/5.36.242.138:9300]]{master=true},}, reason: zen-disco-receive(from master [[Paul][PoBefbfWQ6K9XRJraJF0hA][inet[/5.36.242.138:9300]]{master=true}]) {"error":"RemoteTransportException[[Paul][inet[/5.36.242.138:9300]][index]]; nested: MapperParsingException[failed to parse, document is empty]; ","status":400}local@slsdev-int-09u:/mnt/sl/elastic/elasticsearch-0.90.1$ Any idea what the above error means? Thanks in advance. On Wed, Sep 4, 2013 at 11:42 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > You can try my knapsack plugin which has a "size" URL parameter to control the bulk export. > > [GitHub - jprante/elasticsearch-knapsack: Knapsack plugin is an import/export tool for Elasticsearch](https://github.com/jprante/elasticsearch-knapsack) > > Jörg > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 14, 2013, 12:19pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/28 "2013-09-14T12:19:26Z")
It is possible there are less docs in the index than shards are in the index. Please open an issue on the knapsack github issues. I will have a look. > **[jprante/elasticsearch-knapsack](https://github.com/jprante/elasticsearch-knapsack/issues)** > > elasticsearch-knapsack - Knapsack plugin is an import/export tool for Elasticsearch Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pranav\_amin](https://avatars.discourse-cdn.com/v4/letter/p/a183cd/32.png) [@pranav\_amin](https://discuss.elastic.co/u/pranav_amin)\ **Post date:** [September 16, 2013, 3:16pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/29 "2013-09-16T15:16:08Z")
I did restart the ES. May be i'm missing something for \_source field? How do i check i have this field in the Index since knapsack works on the document if it has \_source field. On Sat, Sep 14, 2013 at 2:11 AM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > I think the plugin is not loaded. > Did you restart es? > Can you see in logs at the begining that the plugin is loaded? > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 13 sept. 2013 à 23:09, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) a écrit : > > thanks for recommending. > > when i try to use this it complains - > > local@slsdev-int-09u:/mnt/sl/elastic/elasticsearch-0.90.1$ curl -XPOST > 5.36.242.139:9200/test/test/\_export > [2013-09-13 21:08:13,612][INFO][cluster.service] [Pranav] > detected\_master [Paul][PoBefbfWQ6K9XRJraJF0hA][inet[/5.36.242.138:9300]]{master=true}, > added {[Tom][3SrFwYh8RmqyVkbhHfqyzQ][inet[/5.36.242.140:9300 > ]]{master=true},[Paul][PoBefbfWQ6K9XRJraJF0hA][inet[/5.36.242.138:9300]]{master=true},}, > reason: zen-disco-receive(from master > [[Paul][PoBefbfWQ6K9XRJraJF0hA][inet[/5.36.242.138:9300]]{master=true}]) > {"error":"RemoteTransportException[[Paul][inet[/5.36.242.138:9300]][index]]; > nested: MapperParsingException[failed to parse, document is empty]; > ","status":400}local@slsdev-int-09u:/mnt/sl/elastic/elasticsearch-0.90.1$ > > Any idea what the above error means? > > Thanks in advance. > > On Wed, Sep 4, 2013 at 11:42 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > You can try my knapsack plugin which has a "size" URL parameter to > > control the bulk export. > > > > [GitHub - jprante/elasticsearch-knapsack: Knapsack plugin is an import/export tool for Elasticsearch](https://github.com/jprante/elasticsearch-knapsack) > > > > Jörg > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pranav\_amin](https://avatars.discourse-cdn.com/v4/letter/p/a183cd/32.png) [@pranav\_amin](https://discuss.elastic.co/u/pranav_amin)\ **Post date:** [September 16, 2013, 3:19pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/30 "2013-09-16T15:19:22Z")
have i got the \_source field populated for this indiced based on this data? it looks to me like that, but wanted to confirm. local@slsdev-int-09u:/mnt/sl/elastic/elasticpm/clintongormley-Elasticsearch.pm-d68cd02$ curl -XGET 5.36.242.139:9200/test/test/1 {"\_index":"test","\_type":"test","\_id":"1","\_version":1,"exists":true, "\_source" : {"key":"value 1"}}local@slsdev-int-09u :/mnt/sl/elastic/elasticpm/clintongor i will log a ticket if i have got everything right. On Sat, Sep 14, 2013 at 8:19 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > It is possible there are less docs in the index than shards are in the > index. Please open an issue on the knapsack github issues. I will have a > look. > > [Issues · jprante/elasticsearch-knapsack · GitHub](https://github.com/jprante/elasticsearch-knapsack/issues) > > Jörg > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pranav\_amin](https://avatars.discourse-cdn.com/v4/letter/p/a183cd/32.png) [@pranav\_amin](https://discuss.elastic.co/u/pranav_amin)\ **Post date:** [September 16, 2013, 5:24pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/31 "2013-09-16T17:24:37Z")
now i see a different error for knapsack - local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3$ curl -XPUT localhost:9200/test/test/1 -d '{"\_source" : {"enabled" : true}, "key":"value 1"}' {"ok":true,"\_index":"test","\_type":"test","\_id":"1","\_version":1}local@pldsvv-07 :/var/elasticsearch/elasticsearch-0.90.3$ local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3$ curl -XGET localhost:9200/test/test/1 {"\_index":"test","\_type":"test","\_id":"1","\_version":1,"exists":true, "\_source" : {"\_source" : {"enabled" : true}, "key":"value 1"}}local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3$ local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3$ local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3$ curl -XPOST localhost:9200/test/test/\_export {"exports":,"totalExported":0,"\_shards":{"total":5,"successful":0,"failed":5,"failures":[{"index":"test","shard":0,"reason":"BroadcastShardOperationFailedException[[test][0] ]; nested: SearchParseException[[test][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [_na_]]]; nested: SearchParseException[[test][0]: from[-1],size[-1]: Parse Failure [No export fields defined]]; "},{"index":"test","shard":1,"reason":"BroadcastShardOperationFailedException[[test][1] ]; nested: SearchParseException[[test][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [_na_]]]; nested: SearchParseException[[test][1]: from[-1],size[-1]: Parse Failure [No export fields defined]]; "},{"index":"test","shard":4,"reason":"BroadcastShardOperationFailedException[[test][4] ]; nested: SearchParseException[[test][4]: from[-1],size[-1]: Parse Failure [Failed to parse source [_na_]]]; nested: SearchParseException[[test][4]: from[-1],size[-1]: Parse Failure [No export fields defined]]; "},{"index":"test","shard":3,"reason":"BroadcastShardOperationFailedException[[test][3] ]; nested: SearchParseException[[test][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [_na_]]]; nested: SearchParseException[[test][3]: from[-1],size[-1]: Parse Failure [No export fields defined]]; "},{"index":"test","shard":2,"reason":"BroadcastShardOperationFailedException[[test][2] ]; nested: SearchParseException[[test][2]: from[-1],size[-1]: Parse Failure [Failed to parse source [_na_]]]; nested: SearchParseException[[test][2]: from[-1],size[-1]: Parse Failure [No export fields defined]]; "}]}}local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3$ any idea what i'm doing wrong here? thanks On Mon, Sep 16, 2013 at 11:19 AM, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com)wrote: > have i got the \_source field populated for this indiced based on this > data? it looks to me like that, but wanted to confirm. > > local@slsdev-int-09u:/mnt/sl/elastic/elasticpm/clintongormley-Elasticsearch.pm-d68cd02$ > curl -XGET 5.36.242.139:9200/test/test/1 > {"\_index":"test","\_type":"test","\_id":"1","\_version":1,"exists":true, > "\_source" : {"key":"value 1"}}local@slsdev-int-09u > :/mnt/sl/elastic/elasticpm/clintongor > > i will log a ticket if i have got everything right. > > On Sat, Sep 14, 2013 at 8:19 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > It is possible there are less docs in the index than shards are in the > > index. Please open an issue on the knapsack github issues. I will have a > > look. > > > > [Issues · jprante/elasticsearch-knapsack · GitHub](https://github.com/jprante/elasticsearch-knapsack/issues) > > > > Jörg > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dobe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dobe/32/2152_2.png) [@dobe](https://discuss.elastic.co/u/dobe)\ **Post date:** [September 17, 2013, 4:05pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/32 "2013-09-17T16:05:28Z")
Hi Pranav Can you try 0.90.4 version of ES. It looks like its a 0.90.3 issue. see [Compatibility with 0.90.3 · Issue #35 · crate/elasticsearch-inout-plugin · GitHub](https://github.com/crate/elasticsearch-inout-plugin/issues/35#issuecomment-24589390) and [BytesStreamOutput default size should be 2k instead of 32k · Issue #3638 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/3638) cheers, Bernd On Friday, August 30, 2013 8:30:11 PM UTC+2, pranav amin wrote: > Hi, > > I tried to use the elasticsearch-inout plug in for export. It worked fine. > Now when i try to import it - it fails with > > lo...@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$ \curl -X POST ' > [http://localhost:9200/abc\_newindex/abc/\_import](http://localhost:9200/abc_newindex/abc/_import)' -d '{ > > ``` > "directory": "/tmp/es-data", > > "compression": "gzip" > > ``` > > } > > ' > > [2013-08-30 18:02:08,818][INFO][cluster.metadata] [Eric] > [abc\_newindex] creating index, cause [auto(bulk api)], shards [5]/[1], > mappings > > [2013-08-30 18:02:20,943][WARN][index.engine.robin] [Eric] > [abc\_newindex][0] failed engine > > [2013-08-30 18:02:21,227][DEBUG][action.bulk] [Eric] > [abc\_newindex][0], node[wHAaefkjSre\_x6Lj8XS\_6A], [P], s[STARTED]: Failed to > execute [org.elasticsearch.action.bulk.BulkShardRequest@591bc6d9] > > java.lang.OutOfMemoryError: Java heap space > > {"imports":,"failures":[{"node\_id":"Iow0kmoGRwOLSqZ5jEivAQ","reason":"org.elasticsearch.action.FailedNodeException: > Failed node [Iow0kmoGRwOLSqZ5jEivAQ]; > org.elasticsearch.transport.RemoteTransportException: > [Druig][inet[/10.3.176.23:9300]][el-crate-import/n]; > org.elasticsearch.transport.ActionNotFoundTransportException: No handler > for action > [el-crate-import/n]"},{"node\_id":"wHAaefkjSre\_x6Lj8XS\_6A","reason":"org.elasticsearch.action.FailedNodeException: > Failed node [wHAaefkjSre\_x6Lj8XS\_6A]; java.lang.OutOfMemoryError: Java heap > space"}]}lo...@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$\Exception in thread "elasticsearch[Eric][bulk][T#2]" > java.lang.OutOfMemoryError: Java heap space > > [2013-08-30 18:02:24,845][WARN][index.engine.robin] [Eric] > [abc\_newindex][0] failed engine > > Any idea what is wrong? I can increase the heap size but not sure of when > i will see the error again since i have limited RAM on the box. > > On Fri, Aug 30, 2013 at 12:13 PM, parul patel \<[parulp...@gmail.com](mailto:parulp...@gmail.com)\ > > > wrote: > > > hey guys, > > > > sorry false alarm. i needed to restart Elasticsearch after the plug in > > was installed. > > now it is working. > > > > thanks a lot for the help. and sorry about my mistake. > > > > On Fri, Aug 30, 2013 at 12:06 PM, parul patel \<[parulp...@gmail.com](mailto:parulp...@gmail.com)\ > > > > > wrote: > > > > > Thanks. > > > I installed it like this, it looked successful. but i don't know how to > > > make it detect by ES. > > > > > > oc...@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$\sudo ../plugin -i elasticsearch-inout-plugin -url > > > file:///$PWD/target/elasticsearch-inout-plugin-0.5.0.jar > > > > > > -\> Installing elasticsearch-inout-plugin... > > > > > > Trying > > > file:////var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master/target/elasticsearch-inout-plugin-0.5.0.jar > > > ... > > > > > > Downloading ..DONE > > > > > > Installed elasticsearch-inout-plugin into > > > /var/elasticsearch/elasticsearch-0.90.3/plugins/elasticsearch-inout-plugin > > > > > > lo...@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$\ > > > > > > On Fri, Aug 30, 2013 at 11:58 AM, David Pilato \<[da...@pilato.fr](mailto:da...@pilato.fr)\ > > > > > > > wrote: > > > > > > > Your plugin is not installed nor detected by elasticsearch. > > > > > > > > HTH > > > > > > > > -- > > > > David 😉 > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > Le 30 août 2013 à 17:51, parul patel \<[parulp...@gmail.com](mailto:parulp...@gmail.com) \\> > > > > a écrit : > > > > > > > > Hi bernd, > > > > > > > > when i try to use the plug in for export it gives me the following > > > > issue, any idea what am i missing? > > > > > > > > local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ > > > > curl -X POST '[http://localhost:9200/\_reindex](http://localhost:9200/_reindex)' > > > > {"error":"InvalidIndexNameException[[\_reindex] Invalid index name > > > > [_reindex], must not start with '_']","status":400} > > > > > > > > local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ > > > > curl -X POST '[http://localhost:9200/abc/\_export](http://localhost:9200/abc/_export)' -d '{ > > > > "fields": ["\_id", "\_source", "\_version", "\_index", "\_type"], > > > > "output\_file": "/tmp/es-data/dump-abc-${shard}.gz", > > > > "compression": "gzip" > > > > } > > > > ' > > > > {"error":"InvalidTypeNameException[mapping type name [_export] can't > > > > start with '_']","status":400} > > > > > > > > advance thanks for the help > > > > > > > > On Thu, Aug 1, 2013 at 10:10 AM, pranav amin \<[parulp...@gmail.com](mailto:parulp...@gmail.com)\ > > > > > > > > > wrote: > > > > > > > > > Thank br Bernd. > > > > > Much appreciated. I will play around with this tool and let you know > > > > > if i come across any questions. > > > > > Thanks again. > > > > > > > > > > If you have time and have used any Monitoring tools for ES which you > > > > > would recommend, please let me know. I had played around with some and > > > > > mentioned below. > > > > > > > > > > Thanks again > > > > > Pranav. > > > > > > > > > > On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote: > > > > > > > > > > > Hi Pranav > > > > > > > > > > > > You might want to take a look at [Crate.io · GitHub](https://github.com/crate/)\*\* > > > > > > elasticsearch-inout-plugin[https://github.com/crate/elasticsearch-inout-plugin](https://github.com/crate/elasticsearch-inout-plugin)we use it for backup restore. The plugin allows you to export/import json > > > > > > data distributed - without having to download the data to the client. > > > > > > However there is no UI for this plugin. > > > > > > > > > > > > br Bernd > > > > > > > > > > > > On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote: > > > > > > > > > > > > > Hi There, > > > > > > > > > > > > > > I'm a new user for Elastic search and plan to use it in Production > > > > > > > environment. > > > > > > > > > > > > > > Can you please help me with some information - > > > > > > > > > > > > > > Monitoring tools - > > > > > > > I looked at the following. > > > > > > > a) Elasticsearch head > > > > > > > b) Bigdesk > > > > > > > c) Paramedic > > > > > > > d) ElasticHQ > > > > > > > > > > > > > > Not sure which of the above are free and recommend by others? > > > > > > > > > > > > > > Backup & Restore - > > > > > > > a) Can i do backup and restore of a Node in a Cluster? Are there any > > > > > > > free/cost based tools (UI or Script based) that can do this and are > > > > > > > frequently recommended by others? > > > > > > > b) Are there any free/cost based tools (UI or Script based) that can > > > > > > > do backup & restore of ACTIVE AND NON-ACTIVE cluster? > > > > > > > > > > > > > > Thanks for the help and advice. > > > > > > > Much appreciated. > > > > > > > > > > > > > > Thanks > > > > > > > Pranav. > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to a topic in the > > > > > > > Google Groups "elasticsearch" group. > > > > > > > To unsubscribe from this topic, visit > > > > > > > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe) > > > > > > > . > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Vik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vik/32/2052_2.png) [@Vik](https://discuss.elastic.co/u/Vik)\ **Post date:** [October 14, 2013, 2:52pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/33 "2013-10-14T14:52:21Z")
Hi, I am having some problems with the knapsack plugin. I have managed to export the data for an index into the tar.gz file using the \_export feature. However when trying to import the data into another index, i can only see the index being created. No data appears to have been imported. I can see the following message in the elasticsearch logs when i run the export command. starting import of test2 import of test2 completed Any ideas on what might be happeing? On Wednesday, September 4, 2013 4:42:38 PM UTC+1, Jörg Prante wrote: > You can try my knapsack plugin which has a "size" URL parameter to control > the bulk export. > > [GitHub - jprante/elasticsearch-knapsack: Knapsack plugin is an import/export tool for Elasticsearch](https://github.com/jprante/elasticsearch-knapsack) > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 14, 2013, 3:30pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/34 "2013-10-14T15:30:17Z")
Please can you open an issue on knapsack github? And please, do not capture old threads. Thanks Jörg Am 14.10.2013 16:52 schrieb "V" [vmnadig@gmail.com](mailto:vmnadig@gmail.com): > Hi, > I am having some problems with the knapsack plugin. I have managed to > export the data for an index into the tar.gz file using the \_export > feature. However when trying to import the data into another index, i can > only see the index being created. No data appears to have been imported. > > I can see the following message in the elasticsearch logs when i run the > export command. > starting import of test2 > import of test2 completed > > Any ideas on what might be happeing? > > On Wednesday, September 4, 2013 4:42:38 PM UTC+1, Jörg Prante wrote: > > > You can try my knapsack plugin which has a "size" URL parameter to > > control the bulk export. > > > > [https://github.com/jprante/\*\*elasticsearch-knapsack](https://github.com/jprante/**elasticsearch-knapsack)[https://github.com/jprante/elasticsearch-knapsack](https://github.com/jprante/elasticsearch-knapsack) > > > > Jörg > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:12am UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/35 "2017-07-06T02:12:33Z")
[Previous page](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015.md?page=1)