# Monitoring, Backup & Restore Tools for Elastic search **URL:** \ **Category:** Elasticsearch\ **Created:** [July 31, 2013, 2:50pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015 "2013-07-31T14:50:56Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![pranav\_amin](https://avatars.discourse-cdn.com/v4/letter/p/a183cd/32.png) [@pranav\_amin](https://discuss.elastic.co/u/pranav_amin)\ **Post date:** [July 31, 2013, 2:50pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/1 "2013-07-31T14:50:56Z")
Hi There, I'm a new user for Elastic search and plan to use it in Production environment. Can you please help me with some information - Monitoring tools - I looked at the following. a) Elasticsearch head b) Bigdesk c) Paramedic d) ElasticHQ Not sure which of the above are free and recommend by others? Backup & Restore - a) Can i do backup and restore of a Node in a Cluster? Are there any free/cost based tools (UI or Script based) that can do this and are frequently recommended by others? b) Are there any free/cost based tools (UI or Script based) that can do backup & restore of ACTIVE AND NON-ACTIVE cluster? Thanks for the help and advice. Much appreciated. Thanks Pranav. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dobe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dobe/32/2152_2.png) [@dobe](https://discuss.elastic.co/u/dobe)\ **Post date:** [August 1, 2013, 4:57am UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/2 "2013-08-01T04:57:23Z")
Hi Pranav You might want to take a look at [GitHub - crate/elasticsearch-inout-plugin: An Elasticsearch plugin which provides the ability to export data by query on server side.](https://github.com/crate/elasticsearch-inout-plugin) we use it for backup restore. The plugin allows you to export/import json data distributed - without having to download the data to the client. However there is no UI for this plugin. br Bernd On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote: > Hi There, > > I'm a new user for Elastic search and plan to use it in Production > environment. > > Can you please help me with some information - > > Monitoring tools - > I looked at the following. > a) Elasticsearch head > b) Bigdesk > c) Paramedic > d) ElasticHQ > > Not sure which of the above are free and recommend by others? > > Backup & Restore - > a) Can i do backup and restore of a Node in a Cluster? Are there any > free/cost based tools (UI or Script based) that can do this and are > frequently recommended by others? > b) Are there any free/cost based tools (UI or Script based) that can do > backup & restore of ACTIVE AND NON-ACTIVE cluster? > > Thanks for the help and advice. > Much appreciated. > > Thanks > Pranav. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pranav\_amin](https://avatars.discourse-cdn.com/v4/letter/p/a183cd/32.png) [@pranav\_amin](https://discuss.elastic.co/u/pranav_amin)\ **Post date:** [August 1, 2013, 2:10pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/3 "2013-08-01T14:10:21Z")
Thank br Bernd. Much appreciated. I will play around with this tool and let you know if i come across any questions. Thanks again. If you have time and have used any Monitoring tools for ES which you would recommend, please let me know. I had played around with some and mentioned below. Thanks again Pranav. On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote: > Hi Pranav > > You might want to take a look at > [GitHub - crate/elasticsearch-inout-plugin: An Elasticsearch plugin which provides the ability to export data by query on server side.](https://github.com/crate/elasticsearch-inout-plugin) we use it for backup > restore. The plugin allows you to export/import json data distributed - > without having to download the data to the client. However there is no UI > for this plugin. > > br Bernd > > On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote: > > > Hi There, > > > > I'm a new user for Elastic search and plan to use it in Production > > environment. > > > > Can you please help me with some information - > > > > Monitoring tools - > > I looked at the following. > > a) Elasticsearch head > > b) Bigdesk > > c) Paramedic > > d) ElasticHQ > > > > Not sure which of the above are free and recommend by others? > > > > Backup & Restore - > > a) Can i do backup and restore of a Node in a Cluster? Are there any > > free/cost based tools (UI or Script based) that can do this and are > > frequently recommended by others? > > b) Are there any free/cost based tools (UI or Script based) that can do > > backup & restore of ACTIVE AND NON-ACTIVE cluster? > > > > Thanks for the help and advice. > > Much appreciated. > > > > Thanks > > Pranav. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pranav\_amin](https://avatars.discourse-cdn.com/v4/letter/p/a183cd/32.png) [@pranav\_amin](https://discuss.elastic.co/u/pranav_amin)\ **Post date:** [August 30, 2013, 3:51pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/4 "2013-08-30T15:51:56Z")
Hi bernd, when i try to use the plug in for export it gives me the following issue, any idea what am i missing? local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ curl -X POST '[http://localhost:9200/\_reindex](http://localhost:9200/_reindex)' {"error":"InvalidIndexNameException[[\_reindex] Invalid index name [_reindex], must not start with '_']","status":400} local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ curl -X POST '[http://localhost:9200/abc/\_export](http://localhost:9200/abc/_export)' -d '{ "fields": ["\_id", "\_source", "\_version", "\_index", "\_type"], "output\_file": "/tmp/es-data/dump-abc-${shard}.gz", "compression": "gzip" } ' {"error":"InvalidTypeNameException[mapping type name [_export] can't start with '_']","status":400} advance thanks for the help On Thu, Aug 1, 2013 at 10:10 AM, pranav amin [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) wrote: > Thank br Bernd. > Much appreciated. I will play around with this tool and let you know if i > come across any questions. > Thanks again. > > If you have time and have used any Monitoring tools for ES which you would > recommend, please let me know. I had played around with some and mentioned > below. > > Thanks again > Pranav. > > On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote: > > > Hi Pranav > > > > You might want to take a look at [Crate.io 路 GitHub](https://github.com/crate/)\*\* > > elasticsearch-inout-plugin[https://github.com/crate/elasticsearch-inout-plugin](https://github.com/crate/elasticsearch-inout-plugin)we use it for backup restore. The plugin allows you to export/import json > > data distributed - without having to download the data to the client. > > However there is no UI for this plugin. > > > > br Bernd > > > > On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote: > > > > > Hi There, > > > > > > I'm a new user for Elastic search and plan to use it in Production > > > environment. > > > > > > Can you please help me with some information - > > > > > > Monitoring tools - > > > I looked at the following. > > > a) Elasticsearch head > > > b) Bigdesk > > > c) Paramedic > > > d) ElasticHQ > > > > > > Not sure which of the above are free and recommend by others? > > > > > > Backup & Restore - > > > a) Can i do backup and restore of a Node in a Cluster? Are there any > > > free/cost based tools (UI or Script based) that can do this and are > > > frequently recommended by others? > > > b) Are there any free/cost based tools (UI or Script based) that can do > > > backup & restore of ACTIVE AND NON-ACTIVE cluster? > > > > > > Thanks for the help and advice. > > > Much appreciated. > > > > > > Thanks > > > Pranav. > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 30, 2013, 3:58pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/5 "2013-08-30T15:58:19Z")
Your plugin is not installed nor detected by elasticsearch. HTH -- David 馃槈 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 30 ao没t 2013 脿 17:51, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) a 茅crit : > Hi bernd, > > when i try to use the plug in for export it gives me the following issue, any idea what am i missing? > > local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ > curl -X POST '[http://localhost:9200/\_reindex](http://localhost:9200/_reindex)' > {"error":"InvalidIndexNameException[[\_reindex] Invalid index name [_reindex], must not start with '_']","status":400} > > local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ > curl -X POST '[http://localhost:9200/abc/\_export](http://localhost:9200/abc/_export)' -d '{ > "fields": ["\_id", "\_source", "\_version", "\_index", "\_type"], > "output\_file": "/tmp/es-data/dump-abc-${shard}.gz", > "compression": "gzip" > } > ' > {"error":"InvalidTypeNameException[mapping type name [_export] can't start with '_']","status":400} > > advance thanks for the help > > On Thu, Aug 1, 2013 at 10:10 AM, pranav amin [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) wrote: > > > Thank br Bernd. > > Much appreciated. I will play around with this tool and let you know if i come across any questions. > > Thanks again. > > > > If you have time and have used any Monitoring tools for ES which you would recommend, please let me know. I had played around with some and mentioned below. > > > > Thanks again > > Pranav. > > > > On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote: > > > > > Hi Pranav > > > > > > You might want to take a look at [GitHub - crate/elasticsearch-inout-plugin: An Elasticsearch plugin which provides the ability to export data by query on server side.](https://github.com/crate/elasticsearch-inout-plugin) we use it for backup restore. The plugin allows you to export/import json data distributed - without having to download the data to the client. However there is no UI for this plugin. > > > > > > br Bernd > > > > > > On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote: > > > > > > > Hi There, > > > > > > > > I'm a new user for Elastic search and plan to use it in Production environment. > > > > > > > > Can you please help me with some information - > > > > > > > > Monitoring tools - > > > > I looked at the following. > > > > a) Elasticsearch head > > > > b) Bigdesk > > > > c) Paramedic > > > > d) ElasticHQ > > > > > > > > Not sure which of the above are free and recommend by others? > > > > > > > > Backup & Restore - > > > > a) Can i do backup and restore of a Node in a Cluster? Are there any free/cost based tools (UI or Script based) that can do this and are frequently recommended by others? > > > > b) Are there any free/cost based tools (UI or Script based) that can do backup & restore of ACTIVE AND NON-ACTIVE cluster? > > > > > > > > Thanks for the help and advice. > > > > Much appreciated. > > > > > > > > Thanks > > > > Pranav. > > > > -- > > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pranav\_amin](https://avatars.discourse-cdn.com/v4/letter/p/a183cd/32.png) [@pranav\_amin](https://discuss.elastic.co/u/pranav_amin)\ **Post date:** [August 30, 2013, 4:06pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/6 "2013-08-30T16:06:58Z")
Thanks. I installed it like this, it looked successful. but i don't know how to make it detect by ES. ocal@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$sudo ../plugin -i elasticsearch-inout-plugin -url file:///$PWD/target/elasticsearch-inout-plugin-0.5.0.jar\ -\> Installing elasticsearch-inout-plugin... Trying file:////var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master/target/elasticsearch-inout-plugin-0.5.0.jar\ ... Downloading ..DONE Installed elasticsearch-inout-plugin into /var/elasticsearch/elasticsearch-0.90.3/plugins/elasticsearch-inout-plugin local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$ On Fri, Aug 30, 2013 at 11:58 AM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > Your plugin is not installed nor detected by elasticsearch. > > HTH > > -- > David 馃槈 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 30 ao没t 2013 脿 17:51, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) a 茅crit : > > Hi bernd, > > when i try to use the plug in for export it gives me the following issue, > any idea what am i missing? > > local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ > curl -X POST '[http://localhost:9200/\_reindex](http://localhost:9200/_reindex)' > {"error":"InvalidIndexNameException[[\_reindex] Invalid index name > [_reindex], must not start with '_']","status":400} > > local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ > curl -X POST '[http://localhost:9200/abc/\_export](http://localhost:9200/abc/_export)' -d '{ > "fields": ["\_id", "\_source", "\_version", "\_index", "\_type"], > "output\_file": "/tmp/es-data/dump-abc-${shard}.gz", > "compression": "gzip" > } > ' > {"error":"InvalidTypeNameException[mapping type name [_export] can't start > with '_']","status":400} > > advance thanks for the help > > On Thu, Aug 1, 2013 at 10:10 AM, pranav amin [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com)wrote: > > > Thank br Bernd. > > Much appreciated. I will play around with this tool and let you know if i > > come across any questions. > > Thanks again. > > > > If you have time and have used any Monitoring tools for ES which you > > would recommend, please let me know. I had played around with some and > > mentioned below. > > > > Thanks again > > Pranav. > > > > On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote: > > > > > Hi Pranav > > > > > > You might want to take a look at [Crate.io 路 GitHub](https://github.com/crate/)\*\* > > > elasticsearch-inout-plugin[https://github.com/crate/elasticsearch-inout-plugin](https://github.com/crate/elasticsearch-inout-plugin)we use it for backup restore. The plugin allows you to export/import json > > > data distributed - without having to download the data to the client. > > > However there is no UI for this plugin. > > > > > > br Bernd > > > > > > On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote: > > > > > > > Hi There, > > > > > > > > I'm a new user for Elastic search and plan to use it in Production > > > > environment. > > > > > > > > Can you please help me with some information - > > > > > > > > Monitoring tools - > > > > I looked at the following. > > > > a) Elasticsearch head > > > > b) Bigdesk > > > > c) Paramedic > > > > d) ElasticHQ > > > > > > > > Not sure which of the above are free and recommend by others? > > > > > > > > Backup & Restore - > > > > a) Can i do backup and restore of a Node in a Cluster? Are there any > > > > free/cost based tools (UI or Script based) that can do this and are > > > > frequently recommended by others? > > > > b) Are there any free/cost based tools (UI or Script based) that can do > > > > backup & restore of ACTIVE AND NON-ACTIVE cluster? > > > > > > > > Thanks for the help and advice. > > > > Much appreciated. > > > > > > > > Thanks > > > > Pranav. > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pranav\_amin](https://avatars.discourse-cdn.com/v4/letter/p/a183cd/32.png) [@pranav\_amin](https://discuss.elastic.co/u/pranav_amin)\ **Post date:** [August 30, 2013, 4:13pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/7 "2013-08-30T16:13:29Z")
hey guys, sorry false alarm. i needed to restart Elasticsearch after the plug in was installed. now it is working. thanks a lot for the help. and sorry about my mistake. On Fri, Aug 30, 2013 at 12:06 PM, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com)wrote: > Thanks. > I installed it like this, it looked successful. but i don't know how to > make it detect by ES. > > ocal@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$sudo ../plugin -i elasticsearch-inout-plugin -url > file:///$PWD/target/elasticsearch-inout-plugin-0.5.0.jar > > -\> Installing elasticsearch-inout-plugin... > > Trying > file:////var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master/target/elasticsearch-inout-plugin-0.5.0.jar > ... > > Downloading ..DONE > > Installed elasticsearch-inout-plugin into > /var/elasticsearch/elasticsearch-0.90.3/plugins/elasticsearch-inout-plugin > > local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$ > > On Fri, Aug 30, 2013 at 11:58 AM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > > > Your plugin is not installed nor detected by elasticsearch. > > > > HTH > > > > -- > > David 馃槈 > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > Le 30 ao没t 2013 脿 17:51, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) a 茅crit : > > > > Hi bernd, > > > > when i try to use the plug in for export it gives me the following issue, > > any idea what am i missing? > > > > local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ > > curl -X POST '[http://localhost:9200/\_reindex](http://localhost:9200/_reindex)' > > {"error":"InvalidIndexNameException[[\_reindex] Invalid index name > > [_reindex], must not start with '_']","status":400} > > > > local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ > > curl -X POST '[http://localhost:9200/abc/\_export](http://localhost:9200/abc/_export)' -d '{ > > "fields": ["\_id", "\_source", "\_version", "\_index", "\_type"], > > "output\_file": "/tmp/es-data/dump-abc-${shard}.gz", > > "compression": "gzip" > > } > > ' > > {"error":"InvalidTypeNameException[mapping type name [_export] can't > > start with '_']","status":400} > > > > advance thanks for the help > > > > On Thu, Aug 1, 2013 at 10:10 AM, pranav amin [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com)wrote: > > > > > Thank br Bernd. > > > Much appreciated. I will play around with this tool and let you know if > > > i come across any questions. > > > Thanks again. > > > > > > If you have time and have used any Monitoring tools for ES which you > > > would recommend, please let me know. I had played around with some and > > > mentioned below. > > > > > > Thanks again > > > Pranav. > > > > > > On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote: > > > > > > > Hi Pranav > > > > > > > > You might want to take a look at [Crate.io 路 GitHub](https://github.com/crate/)\*\* > > > > elasticsearch-inout-plugin[https://github.com/crate/elasticsearch-inout-plugin](https://github.com/crate/elasticsearch-inout-plugin)we use it for backup restore. The plugin allows you to export/import json > > > > data distributed - without having to download the data to the client. > > > > However there is no UI for this plugin. > > > > > > > > br Bernd > > > > > > > > On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote: > > > > > > > > > Hi There, > > > > > > > > > > I'm a new user for Elastic search and plan to use it in Production > > > > > environment. > > > > > > > > > > Can you please help me with some information - > > > > > > > > > > Monitoring tools - > > > > > I looked at the following. > > > > > a) Elasticsearch head > > > > > b) Bigdesk > > > > > c) Paramedic > > > > > d) ElasticHQ > > > > > > > > > > Not sure which of the above are free and recommend by others? > > > > > > > > > > Backup & Restore - > > > > > a) Can i do backup and restore of a Node in a Cluster? Are there any > > > > > free/cost based tools (UI or Script based) that can do this and are > > > > > frequently recommended by others? > > > > > b) Are there any free/cost based tools (UI or Script based) that can > > > > > do backup & restore of ACTIVE AND NON-ACTIVE cluster? > > > > > > > > > > Thanks for the help and advice. > > > > > Much appreciated. > > > > > > > > > > Thanks > > > > > Pranav. > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pranav\_amin](https://avatars.discourse-cdn.com/v4/letter/p/a183cd/32.png) [@pranav\_amin](https://discuss.elastic.co/u/pranav_amin)\ **Post date:** [August 30, 2013, 6:30pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/8 "2013-08-30T18:30:11Z")
Hi, I tried to use the elasticsearch-inout plug in for export. It worked fine. Now when i try to import it - it fails with local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$ curl -X POST ' [http://localhost:9200/abc\_newindex/abc/\_import](http://localhost:9200/abc_newindex/abc/_import)' -d '{ ``` "directory": "/tmp/es-data", "compression": "gzip" ``` } ' [2013-08-30 18:02:08,818][INFO][cluster.metadata] [Eric] [abc\_newindex] creating index, cause [auto(bulk api)], shards [5]/[1], mappings [2013-08-30 18:02:20,943][WARN][index.engine.robin] [Eric] [abc\_newindex][0] failed engine [2013-08-30 18:02:21,227][DEBUG][action.bulk] [Eric] [abc\_newindex][0], node[wHAaefkjSre\_x6Lj8XS\_6A], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.bulk.BulkShardRequest@591bc6d9] java.lang.OutOfMemoryError: Java heap space {"imports":,"failures":[{"node\_id":"Iow0kmoGRwOLSqZ5jEivAQ","reason":"org.elasticsearch.action.FailedNodeException: Failed node [Iow0kmoGRwOLSqZ5jEivAQ]; org.elasticsearch.transport.RemoteTransportException: [Druig][inet[/10.3.176.23:9300]][el-crate-import/n]; org.elasticsearch.transport.ActionNotFoundTransportException: No handler for action [el-crate-import/n]"},{"node\_id":"wHAaefkjSre\_x6Lj8XS\_6A","reason":"org.elasticsearch.action.FailedNodeException: Failed node [wHAaefkjSre\_x6Lj8XS\_6A]; java.lang.OutOfMemoryError: Java heap space"}]}local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$Exception in thread "elasticsearch[Eric][bulk][T#2]" java.lang.OutOfMemoryError: Java heap space [2013-08-30 18:02:24,845][WARN][index.engine.robin] [Eric] [abc\_newindex][0] failed engine Any idea what is wrong? I can increase the heap size but not sure of when i will see the error again since i have limited RAM on the box. On Fri, Aug 30, 2013 at 12:13 PM, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com)wrote: > hey guys, > > sorry false alarm. i needed to restart Elasticsearch after the plug in > was installed. > now it is working. > > thanks a lot for the help. and sorry about my mistake. > > On Fri, Aug 30, 2013 at 12:06 PM, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com)wrote: > > > Thanks. > > I installed it like this, it looked successful. but i don't know how to > > make it detect by ES. > > > > ocal@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$sudo ../plugin -i elasticsearch-inout-plugin -url > > file:///$PWD/target/elasticsearch-inout-plugin-0.5.0.jar > > > > -\> Installing elasticsearch-inout-plugin... > > > > Trying > > file:////var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master/target/elasticsearch-inout-plugin-0.5.0.jar > > ... > > > > Downloading ..DONE > > > > Installed elasticsearch-inout-plugin into > > /var/elasticsearch/elasticsearch-0.90.3/plugins/elasticsearch-inout-plugin > > > > local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$ > > > > On Fri, Aug 30, 2013 at 11:58 AM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > > > > > Your plugin is not installed nor detected by elasticsearch. > > > > > > HTH > > > > > > -- > > > David 馃槈 > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > Le 30 ao没t 2013 脿 17:51, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) a 茅crit : > > > > > > Hi bernd, > > > > > > when i try to use the plug in for export it gives me the following > > > issue, any idea what am i missing? > > > > > > local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ > > > curl -X POST '[http://localhost:9200/\_reindex](http://localhost:9200/_reindex)' > > > {"error":"InvalidIndexNameException[[\_reindex] Invalid index name > > > [_reindex], must not start with '_']","status":400} > > > > > > local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ > > > curl -X POST '[http://localhost:9200/abc/\_export](http://localhost:9200/abc/_export)' -d '{ > > > "fields": ["\_id", "\_source", "\_version", "\_index", "\_type"], > > > "output\_file": "/tmp/es-data/dump-abc-${shard}.gz", > > > "compression": "gzip" > > > } > > > ' > > > {"error":"InvalidTypeNameException[mapping type name [_export] can't > > > start with '_']","status":400} > > > > > > advance thanks for the help > > > > > > On Thu, Aug 1, 2013 at 10:10 AM, pranav amin [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com)wrote: > > > > > > > Thank br Bernd. > > > > Much appreciated. I will play around with this tool and let you know if > > > > i come across any questions. > > > > Thanks again. > > > > > > > > If you have time and have used any Monitoring tools for ES which you > > > > would recommend, please let me know. I had played around with some and > > > > mentioned below. > > > > > > > > Thanks again > > > > Pranav. > > > > > > > > On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote: > > > > > > > > > Hi Pranav > > > > > > > > > > You might want to take a look at [Crate.io 路 GitHub](https://github.com/crate/)\*\* > > > > > elasticsearch-inout-plugin[https://github.com/crate/elasticsearch-inout-plugin](https://github.com/crate/elasticsearch-inout-plugin)we use it for backup restore. The plugin allows you to export/import json > > > > > data distributed - without having to download the data to the client. > > > > > However there is no UI for this plugin. > > > > > > > > > > br Bernd > > > > > > > > > > On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote: > > > > > > > > > > > Hi There, > > > > > > > > > > > > I'm a new user for Elastic search and plan to use it in Production > > > > > > environment. > > > > > > > > > > > > Can you please help me with some information - > > > > > > > > > > > > Monitoring tools - > > > > > > I looked at the following. > > > > > > a) Elasticsearch head > > > > > > b) Bigdesk > > > > > > c) Paramedic > > > > > > d) ElasticHQ > > > > > > > > > > > > Not sure which of the above are free and recommend by others? > > > > > > > > > > > > Backup & Restore - > > > > > > a) Can i do backup and restore of a Node in a Cluster? Are there any > > > > > > free/cost based tools (UI or Script based) that can do this and are > > > > > > frequently recommended by others? > > > > > > b) Are there any free/cost based tools (UI or Script based) that can > > > > > > do backup & restore of ACTIVE AND NON-ACTIVE cluster? > > > > > > > > > > > > Thanks for the help and advice. > > > > > > Much appreciated. > > > > > > > > > > > > Thanks > > > > > > Pranav. > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to a topic in the > > > > > > Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this topic, visit > > > > > > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe) > > > > > > . > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pranav\_amin](https://avatars.discourse-cdn.com/v4/letter/p/a183cd/32.png) [@pranav\_amin](https://discuss.elastic.co/u/pranav_amin)\ **Post date:** [September 3, 2013, 1:55pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/9 "2013-09-03T13:55:06Z")
hey guys, much appreciated if someone can shed light on the above error here? i can increase the heap size, but that doesn't seem to be like a fix. since the overall data size is just 46 MB, if i want to do an import of 1Gb or 10Gb, i don't know how much heap will be needed. may be i'm missing something. thanks On Fri, Aug 30, 2013 at 2:30 PM, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) wrote: > Hi, > > I tried to use the elasticsearch-inout plug in for export. It worked fine. > Now when i try to import it - it fails with > > local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$ curl -X POST > '[http://localhost:9200/abc\_newindex/abc/\_import](http://localhost:9200/abc_newindex/abc/_import)' -d '{ > > ``` > "directory": "/tmp/es-data", > > "compression": "gzip" > > ``` > > } > > ' > > [2013-08-30 18:02:08,818][INFO][cluster.metadata] [Eric] > [abc\_newindex] creating index, cause [auto(bulk api)], shards [5]/[1], > mappings > > [2013-08-30 18:02:20,943][WARN][index.engine.robin] [Eric] > [abc\_newindex][0] failed engine > > [2013-08-30 18:02:21,227][DEBUG][action.bulk] [Eric] > [abc\_newindex][0], node[wHAaefkjSre\_x6Lj8XS\_6A], [P], s[STARTED]: Failed to > execute [org.elasticsearch.action.bulk.BulkShardRequest@591bc6d9] > > java.lang.OutOfMemoryError: Java heap space > > {"imports":,"failures":[{"node\_id":"Iow0kmoGRwOLSqZ5jEivAQ","reason":"org.elasticsearch.action.FailedNodeException: > Failed node [Iow0kmoGRwOLSqZ5jEivAQ]; > org.elasticsearch.transport.RemoteTransportException: > [Druig][inet[/10.3.176.23:9300]][el-crate-import/n]; > org.elasticsearch.transport.ActionNotFoundTransportException: No handler > for action > [el-crate-import/n]"},{"node\_id":"wHAaefkjSre\_x6Lj8XS\_6A","reason":"org.elasticsearch.action.FailedNodeException: > Failed node [wHAaefkjSre\_x6Lj8XS\_6A]; java.lang.OutOfMemoryError: Java heap > space"}]}local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$Exception in thread "elasticsearch[Eric][bulk][T#2]" > java.lang.OutOfMemoryError: Java heap space > > [2013-08-30 18:02:24,845][WARN][index.engine.robin] [Eric] > [abc\_newindex][0] failed engine > > Any idea what is wrong? I can increase the heap size but not sure of when > i will see the error again since i have limited RAM on the box. > > On Fri, Aug 30, 2013 at 12:13 PM, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com)wrote: > > > hey guys, > > > > sorry false alarm. i needed to restart Elasticsearch after the plug in > > was installed. > > now it is working. > > > > thanks a lot for the help. and sorry about my mistake. > > > > On Fri, Aug 30, 2013 at 12:06 PM, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com)wrote: > > > > > Thanks. > > > I installed it like this, it looked successful. but i don't know how to > > > make it detect by ES. > > > > > > ocal@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$sudo ../plugin -i elasticsearch-inout-plugin -url > > > file:///$PWD/target/elasticsearch-inout-plugin-0.5.0.jar > > > > > > -\> Installing elasticsearch-inout-plugin... > > > > > > Trying > > > file:////var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master/target/elasticsearch-inout-plugin-0.5.0.jar > > > ... > > > > > > Downloading ..DONE > > > > > > Installed elasticsearch-inout-plugin into > > > /var/elasticsearch/elasticsearch-0.90.3/plugins/elasticsearch-inout-plugin > > > > > > local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$ > > > > > > On Fri, Aug 30, 2013 at 11:58 AM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > > > > > > > Your plugin is not installed nor detected by elasticsearch. > > > > > > > > HTH > > > > > > > > -- > > > > David 馃槈 > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > Le 30 ao没t 2013 脿 17:51, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) a 茅crit : > > > > > > > > Hi bernd, > > > > > > > > when i try to use the plug in for export it gives me the following > > > > issue, any idea what am i missing? > > > > > > > > local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ > > > > curl -X POST '[http://localhost:9200/\_reindex](http://localhost:9200/_reindex)' > > > > {"error":"InvalidIndexNameException[[\_reindex] Invalid index name > > > > [_reindex], must not start with '_']","status":400} > > > > > > > > local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ > > > > curl -X POST '[http://localhost:9200/abc/\_export](http://localhost:9200/abc/_export)' -d '{ > > > > "fields": ["\_id", "\_source", "\_version", "\_index", "\_type"], > > > > "output\_file": "/tmp/es-data/dump-abc-${shard}.gz", > > > > "compression": "gzip" > > > > } > > > > ' > > > > {"error":"InvalidTypeNameException[mapping type name [_export] can't > > > > start with '_']","status":400} > > > > > > > > advance thanks for the help > > > > > > > > On Thu, Aug 1, 2013 at 10:10 AM, pranav amin [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com)wrote: > > > > > > > > > Thank br Bernd. > > > > > Much appreciated. I will play around with this tool and let you know > > > > > if i come across any questions. > > > > > Thanks again. > > > > > > > > > > If you have time and have used any Monitoring tools for ES which you > > > > > would recommend, please let me know. I had played around with some and > > > > > mentioned below. > > > > > > > > > > Thanks again > > > > > Pranav. > > > > > > > > > > On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote: > > > > > > > > > > > Hi Pranav > > > > > > > > > > > > You might want to take a look at [Crate.io 路 GitHub](https://github.com/crate/)\*\* > > > > > > elasticsearch-inout-plugin[https://github.com/crate/elasticsearch-inout-plugin](https://github.com/crate/elasticsearch-inout-plugin)we use it for backup restore. The plugin allows you to export/import json > > > > > > data distributed - without having to download the data to the client. > > > > > > However there is no UI for this plugin. > > > > > > > > > > > > br Bernd > > > > > > > > > > > > On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote: > > > > > > > > > > > > > Hi There, > > > > > > > > > > > > > > I'm a new user for Elastic search and plan to use it in Production > > > > > > > environment. > > > > > > > > > > > > > > Can you please help me with some information - > > > > > > > > > > > > > > Monitoring tools - > > > > > > > I looked at the following. > > > > > > > a) Elasticsearch head > > > > > > > b) Bigdesk > > > > > > > c) Paramedic > > > > > > > d) ElasticHQ > > > > > > > > > > > > > > Not sure which of the above are free and recommend by others? > > > > > > > > > > > > > > Backup & Restore - > > > > > > > a) Can i do backup and restore of a Node in a Cluster? Are there any > > > > > > > free/cost based tools (UI or Script based) that can do this and are > > > > > > > frequently recommended by others? > > > > > > > b) Are there any free/cost based tools (UI or Script based) that can > > > > > > > do backup & restore of ACTIVE AND NON-ACTIVE cluster? > > > > > > > > > > > > > > Thanks for the help and advice. > > > > > > > Much appreciated. > > > > > > > > > > > > > > Thanks > > > > > > > Pranav. > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to a topic in the > > > > > > > Google Groups "elasticsearch" group. > > > > > > > To unsubscribe from this topic, visit > > > > > > > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe) > > > > > > > . > > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\ **Post date:** [September 3, 2013, 3:05pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/10 "2013-09-03T15:05:28Z")
I have not used this plug-in: I currently use my own tools: 1. A scan-based query-all that iterates over a scanned snapshot, and emits each document together with its appropriate action\_and\_meta\_data line for ease of import. Based on your posts, I imagine that the plug-in is doing something similar, except that the plug-in is also adding compression (my tool emits to stdout so I can also add compression by piping through gzip or other compressor). So far, so good. 2. You try to import using curl. No!!!!!! The curl command is to import as your fingernails are to digging the Chunnel. So I wrote my own importer that can read the action\_and\_meta\_data line along with the subsequent \_source line, use the BulkRequestBuilder Java object to collect up to 4096 of them (configurable, but this works well), and I can import hundreds of millions of documents this way with no problems. But I left curl behind long ago, using it only for loading a small handful of documents during initial testing. The curl command has too many limitations. You can fire up your C compiler, get the source, and try to force its behavior to better fit with what you want. Or you can use the BulkRequestBuilder and do it in Java. But either way, you have to feed the import in chunks and not as one huge chunk. And I would strongly recommend against curl for imports into production databases and for imports of huge numbers of documents in one bulk increment in any environment. Brian On Tuesday, September 3, 2013 9:55:06 AM UTC-4, pranav amin wrote: > hey guys, > > much appreciated if someone can shed light on the above error here? > i can increase the heap size, but that doesn't seem to be like a fix. > since the overall data size is just 46 MB, if i want to do an import of 1Gb > or 10Gb, i don't know how much heap will be needed. > > may be i'm missing something. > > thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pranav\_amin](https://avatars.discourse-cdn.com/v4/letter/p/a183cd/32.png) [@pranav\_amin](https://discuss.elastic.co/u/pranav_amin)\ **Post date:** [September 4, 2013, 2:25pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/11 "2013-09-04T14:25:18Z")
Thanks brian. Much appreciated. This means i will need to customize or come up with my own tool to do this. Any idea if there is any other plugin which works well and i can just reuse rather than - reinventing the wheel? thanks On Tue, Sep 3, 2013 at 11:05 AM, InquiringMind [brian.from.fl@gmail.com](mailto:brian.from.fl@gmail.com)wrote: > I have not used this plug-in: I currently use my own tools: > > 1. A scan-based query-all that iterates over a scanned snapshot, and emits > each document together with its appropriate action\_and\_meta\_data line for > ease of import. Based on your posts, I imagine that the plug-in is doing > something similar, except that the plug-in is also adding compression (my > tool emits to stdout so I can also add compression by piping through gzip > or other compressor). So far, so good. > > 2. You try to import using curl. No!!!!!! The curl command is to import as > your fingernails are to digging the Chunnel. So I wrote my own importer > that can read the action\_and\_meta\_data line along with the subsequent > \_source line, use the BulkRequestBuilder Java object to collect up to 4096 > of them (configurable, but this works well), and I can import hundreds of > millions of documents this way with no problems. But I left curl behind > long ago, using it only for loading a small handful of documents during > initial testing. > > The curl command has too many limitations. You can fire up your C > compiler, get the source, and try to force its behavior to better fit with > what you want. Or you can use the BulkRequestBuilder and do it in Java. But > either way, you have to feed the import in chunks and not as one huge > chunk. And I would strongly recommend against curl for imports into > production databases and for imports of huge numbers of documents in one > bulk increment in any environment. > > Brian > > On Tuesday, September 3, 2013 9:55:06 AM UTC-4, pranav amin wrote: > > > hey guys, > > > > much appreciated if someone can shed light on the above error here? > > i can increase the heap size, but that doesn't seem to be like a fix. > > since the overall data size is just 46 MB, if i want to do an import of 1Gb > > or 10Gb, i don't know how much heap will be needed. > > > > may be i'm missing something. > > > > thanks > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\ **Post date:** [September 4, 2013, 3:17pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/12 "2013-09-04T15:17:58Z")
The plug-in that was mentioned earlier looks to be able to do both the back-up and the restore (export and import): > **[crate/elasticsearch-inout-plugin](https://github.com/crate/elasticsearch-inout-plugin)** > > elasticsearch-inout-plugin - An Elasticsearch plugin which provides the ability to export data by query on server side. Looks pretty cool! I haven't used it, but after reading the post that mentioned it, it seems to be just what you're looking for. Brian -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pranav\_amin](https://avatars.discourse-cdn.com/v4/letter/p/a183cd/32.png) [@pranav\_amin](https://discuss.elastic.co/u/pranav_amin)\ **Post date:** [September 4, 2013, 3:27pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/13 "2013-09-04T15:27:17Z")
Thanks. I had the same opinion. However the curl commands what it has mentioned in the document/link gave me the out of heap memory error. And there is no alternative mentioned of how to run that command with Bulk size configurable to avoid the out of memory error. Thanks On Wed, Sep 4, 2013 at 11:17 AM, InquiringMind [brian.from.fl@gmail.com](mailto:brian.from.fl@gmail.com)wrote: > The plug-in that was mentioned earlier looks to be able to do both the > back-up and the restore (export and import): [Crate.io 路 GitHub](https://github.com/crate/)\*\* > elasticsearch-inout-plugin[https://github.com/crate/elasticsearch-inout-plugin](https://github.com/crate/elasticsearch-inout-plugin) > > Looks pretty cool! I haven't used it, but after reading the post that > mentioned it, it seems to be just what you're looking for. > > Brian > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 4, 2013, 3:42pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/14 "2013-09-04T15:42:38Z")
You can try my knapsack plugin which has a "size" URL parameter to control the bulk export. > **[jprante/elasticsearch-knapsack](https://github.com/jprante/elasticsearch-knapsack)** > > elasticsearch-knapsack - Knapsack plugin is an import/export tool for Elasticsearch J枚rg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pranav\_amin](https://avatars.discourse-cdn.com/v4/letter/p/a183cd/32.png) [@pranav\_amin](https://discuss.elastic.co/u/pranav_amin)\ **Post date:** [September 4, 2013, 8:14pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/15 "2013-09-04T20:14:25Z")
thanks jorg. is the export/import same as backup/restore or it is different? does the script only export data from the node it currently runs on or does the backup on every node in the cluster? thanks On Wed, Sep 4, 2013 at 11:42 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > You can try my knapsack plugin which has a "size" URL parameter to control > the bulk export. > > [GitHub - jprante/elasticsearch-knapsack: Knapsack plugin is an import/export tool for Elasticsearch](https://github.com/jprante/elasticsearch-knapsack) > > J枚rg > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dobe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dobe/32/2152_2.png) [@dobe](https://discuss.elastic.co/u/dobe)\ **Post date:** [September 5, 2013, 5:06am UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/16 "2013-09-05T05:06:40Z")
Hi Parul We actually did not see any memory issues till now on our installations. So could you please set the log level to debug and send the logs after forcing the memory error. It would also help to have the following information to find the problem: 1. elasticsearch version, plugin version (would be best to use 0.90.3 and the latest plugin version) 2. heap space size 3. data size, number of data files, number of nodes 4. the index settings and status output thx, bernd On Aug 30, 2013, at 20:30 , parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) wrote: > Hi, > > I tried to use the elasticsearch-inout plug in for export. It worked fine. Now when i try to import it - it fails with > > local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$ curl -X POST '[http://localhost:9200/abc\_newindex/abc/\_import](http://localhost:9200/abc_newindex/abc/_import)' -d '{ > > ``` > "directory": "/tmp/es-data", > > "compression": "gzip" > > ``` > > } > > ' > > [2013-08-30 18:02:08,818][INFO][cluster.metadata] [Eric] [abc\_newindex] creating index, cause [auto(bulk api)], shards [5]/[1], mappings > > [2013-08-30 18:02:20,943][WARN][index.engine.robin] [Eric] [abc\_newindex][0] failed engine > > [2013-08-30 18:02:21,227][DEBUG][action.bulk] [Eric] [abc\_newindex][0], node[wHAaefkjSre\_x6Lj8XS\_6A], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.bulk.BulkShardRequest@591bc6d9] > > java.lang.OutOfMemoryError: Java heap space > > {"imports":,"failures":[{"node\_id":"Iow0kmoGRwOLSqZ5jEivAQ","reason":"org.elasticsearch.action.FailedNodeException: Failed node [Iow0kmoGRwOLSqZ5jEivAQ]; org.elasticsearch.transport.RemoteTransportException: [Druig][inet[/10.3.176.23:9300]][el-crate-import/n]; org.elasticsearch.transport.ActionNotFoundTransportException: No handler for action [el-crate-import/n]"},{"node\_id":"wHAaefkjSre\_x6Lj8XS\_6A","reason":"org.elasticsearch.action.FailedNodeException: Failed node [wHAaefkjSre\_x6Lj8XS\_6A]; java.lang.OutOfMemoryError: Java heap space"}]}local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$ Exception in thread "elasticsearch[Eric][bulk][T#2]" java.lang.OutOfMemoryError: Java heap space > > [2013-08-30 18:02:24,845][WARN][index.engine.robin] [Eric] [abc\_newindex][0] failed engine > > Any idea what is wrong? I can increase the heap size but not sure of when i will see the error again since i have limited RAM on the box. > > On Fri, Aug 30, 2013 at 12:13 PM, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) wrote: > hey guys, > > sorry false alarm. i needed to restart Elasticsearch after the plug in was installed. > now it is working. > > thanks a lot for the help. and sorry about my mistake. > > On Fri, Aug 30, 2013 at 12:06 PM, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) wrote: > Thanks. > I installed it like this, it looked successful. but i don't know how to make it detect by ES. > > ocal@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$ sudo ../plugin -i elasticsearch-inout-plugin -url file:///$PWD/target/elasticsearch-inout-plugin-0.5.0.jar > > -\> Installing elasticsearch-inout-plugin... > > Trying file:////var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master/target/elasticsearch-inout-plugin-0.5.0.jar... > > Downloading ..DONE > > Installed elasticsearch-inout-plugin into /var/elasticsearch/elasticsearch-0.90.3/plugins/elasticsearch-inout-plugin > > local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$ > > On Fri, Aug 30, 2013 at 11:58 AM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > Your plugin is not installed nor detected by elasticsearch. > > HTH > > -- > David 馃槈 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 30 ao没t 2013 脿 17:51, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) a 茅crit : > > > Hi bernd, > > > > when i try to use the plug in for export it gives me the following issue, any idea what am i missing? > > > > local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ > > curl -X POST '[http://localhost:9200/\_reindex](http://localhost:9200/_reindex)' > > {"error":"InvalidIndexNameException[[\_reindex] Invalid index name [_reindex], must not start with '_']","status":400} > > > > local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ > > curl -X POST '[http://localhost:9200/abc/\_export](http://localhost:9200/abc/_export)' -d '{ > > "fields": ["\_id", "\_source", "\_version", "\_index", "\_type"], > > "output\_file": "/tmp/es-data/dump-abc-${shard}.gz", > > "compression": "gzip" > > } > > ' > > {"error":"InvalidTypeNameException[mapping type name [_export] can't start with '_']","status":400} > > > > advance thanks for the help > > > > On Thu, Aug 1, 2013 at 10:10 AM, pranav amin [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) wrote: > > Thank br Bernd. > > Much appreciated. I will play around with this tool and let you know if i come across any questions. > > Thanks again. > > > > If you have time and have used any Monitoring tools for ES which you would recommend, please let me know. I had played around with some and mentioned below. > > > > Thanks again > > Pranav. > > > > On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote: > > Hi Pranav > > > > You might want to take a look at [GitHub - crate/elasticsearch-inout-plugin: An Elasticsearch plugin which provides the ability to export data by query on server side.](https://github.com/crate/elasticsearch-inout-plugin) we use it for backup restore. The plugin allows you to export/import json data distributed - without having to download the data to the client. However there is no UI for this plugin. > > > > br Bernd > > > > On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote: > > Hi There, > > > > I'm a new user for Elastic search and plan to use it in Production environment. > > > > Can you please help me with some information - > > > > Monitoring tools - > > I looked at the following. > > a) Elasticsearch head > > b) Bigdesk > > c) Paramedic > > d) ElasticHQ > > > > Not sure which of the above are free and recommend by others? > > > > Backup & Restore - > > a) Can i do backup and restore of a Node in a Cluster? Are there any free/cost based tools (UI or Script based) that can do this and are frequently recommended by others? > > b) Are there any free/cost based tools (UI or Script based) that can do backup & restore of ACTIVE AND NON-ACTIVE cluster? > > > > Thanks for the help and advice. > > Much appreciated. > > > > Thanks > > Pranav. > > > > -- > > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 5, 2013, 7:05am UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/17 "2013-09-05T07:05:48Z")
Knapsack is working on index/type level and scans/scrolls over the \_source fields with variable bulk size. It is enough to install it on one node to dump/restore the whole cluster. Knapsack uses tar.gz archives for dump archives by default (also handy for external post processing since the ES JSON docs can be unpacked to the file system) J枚rg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![lebowitz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lebowitz/32/1814_2.png) [@lebowitz](https://discuss.elastic.co/u/lebowitz)\ **Post date:** [September 6, 2013, 1:54pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/18 "2013-09-06T13:54:37Z")
I can recommend Jorg's knapsack plugin. It efficiently compresses each document as a file and dames your settings and mapping. When you are using it in import mode, you can specify a new settings and mapping. This makes it handy for manipulation and indexing. Jorg one question for you: if I extract the knapsack archive, will the file system have a problem with 100m files? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pranav\_amin](https://avatars.discourse-cdn.com/v4/letter/p/a183cd/32.png) [@pranav\_amin](https://discuss.elastic.co/u/pranav_amin)\ **Post date:** [September 6, 2013, 3:18pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/19 "2013-09-06T15:18:36Z")
thanks bernd. We use - 1. Elasticsearch version 0.90.3, 2. Plugin version was picked up from website. 3. Heap space size is default i.e. 256M 4. Data size is 46MB 5. Number of data files - Not sure how do i check this? 6. Number of nodes - 3 7. Index settings - Settings - {"abc":{"settings":{"index.number\_of\_shards":"5","index.number\_of\_replicas":"1","index.version.created":"900399"}}} Mappings - {"abc":{"export":{"properties":{"compression":{"type":"string"},"fields":{"type":"string"},"output\_file":{"type":"string"}}},"device":{"properties":{"deviceId":{"type":"string"},"packages":{"properties":{"dependencies":{"properties":{"id":{"type":"string"},"version":{"type":"string"}}},"id":{"type":"string"},"source":{"type":"string"},"version":{"type":"string"}}},"properties":{"properties":{"pin":{"properties":{"source":{"type":"string"},"value":{"type":"string"}}}}},"supportedAPIs":{"properties":{"id":{"type":"string"},"source":{"type":"string"},"type":{"type":"string"},"version":{"type":"string"}}},"ts":{"type":"long"},"vendorId":{"type":"string"}}}}} Thanks On Thu, Sep 5, 2013 at 1:06 AM, Bernd Dorn [bernddorn@gmail.com](mailto:bernddorn@gmail.com) wrote: > Hi Parul > > We actually did not see any memory issues till now on our installations. > > So could you please set the log level to debug and send the logs after > forcing the memory error. > > It would also help to have the following information to find the problem: > > 1. elasticsearch version, plugin version (would be best to use 0.90.3 and > the latest plugin version) > 2. heap space size > 3. data size, number of data files, number of nodes > 4. the index settings and status output > > thx, bernd > > On Aug 30, 2013, at 20:30 , parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) wrote: > > Hi, > > I tried to use the elasticsearch-inout plug in for export. It worked fine. > Now when i try to import it - it fails with > > local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$ curl -X POST > '[http://localhost:9200/abc\_newindex/abc/\_import](http://localhost:9200/abc_newindex/abc/_import)' -d '{ > > ``` > "directory": "/tmp/es-data", > > "compression": "gzip" > > ``` > > } > > ' > > [2013-08-30 18:02:08,818][INFO][cluster.metadata] [Eric] > [abc\_newindex] creating index, cause [auto(bulk api)], shards [5]/[1], > mappings > > [2013-08-30 18:02:20,943][WARN][index.engine.robin] [Eric] > [abc\_newindex][0] failed engine > > [2013-08-30 18:02:21,227][DEBUG][action.bulk] [Eric] > [abc\_newindex][0], node[wHAaefkjSre\_x6Lj8XS\_6A], [P], s[STARTED]: Failed to > execute [org.elasticsearch.action.bulk.BulkShardRequest@591bc6d9] > > java.lang.OutOfMemoryError: Java heap space > > {"imports":,"failures":[{"node\_id":"Iow0kmoGRwOLSqZ5jEivAQ","reason":"org.elasticsearch.action.FailedNodeException: > Failed node [Iow0kmoGRwOLSqZ5jEivAQ]; > org.elasticsearch.transport.RemoteTransportException: > [Druig][inet[/10.3.176.23:9300]][el-crate-import/n]; > org.elasticsearch.transport.ActionNotFoundTransportException: No handler > for action > [el-crate-import/n]"},{"node\_id":"wHAaefkjSre\_x6Lj8XS\_6A","reason":"org.elasticsearch.action.FailedNodeException: > Failed node [wHAaefkjSre\_x6Lj8XS\_6A]; java.lang.OutOfMemoryError: Java heap > space"}]}local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$Exception in thread "elasticsearch[Eric][bulk][T#2]" > java.lang.OutOfMemoryError: Java heap space > > [2013-08-30 18:02:24,845][WARN][index.engine.robin] [Eric] > [abc\_newindex][0] failed engine > > Any idea what is wrong? I can increase the heap size but not sure of when > i will see the error again since i have limited RAM on the box. > > On Fri, Aug 30, 2013 at 12:13 PM, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com)wrote: > > > hey guys, > > > > sorry false alarm. i needed to restart Elasticsearch after the plug in > > was installed. > > now it is working. > > > > thanks a lot for the help. and sorry about my mistake. > > > > On Fri, Aug 30, 2013 at 12:06 PM, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com)wrote: > > > > > Thanks. > > > I installed it like this, it looked successful. but i don't know how to > > > make it detect by ES. > > > > > > ocal@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$sudo ../plugin -i elasticsearch-inout-plugin -url > > > file:///$PWD/target/elasticsearch-inout-plugin-0.5.0.jar > > > > > > -\> Installing elasticsearch-inout-plugin... > > > > > > Trying > > > file:////var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master/target/elasticsearch-inout-plugin-0.5.0.jar > > > ... > > > > > > Downloading ..DONE > > > > > > Installed elasticsearch-inout-plugin into > > > /var/elasticsearch/elasticsearch-0.90.3/plugins/elasticsearch-inout-plugin > > > > > > local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$ > > > > > > On Fri, Aug 30, 2013 at 11:58 AM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > > > > > > > Your plugin is not installed nor detected by elasticsearch. > > > > > > > > HTH > > > > > > > > -- > > > > David 馃槈 > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > Le 30 ao没t 2013 脿 17:51, parul patel [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com) a 茅crit : > > > > > > > > Hi bernd, > > > > > > > > when i try to use the plug in for export it gives me the following > > > > issue, any idea what am i missing? > > > > > > > > local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ > > > > curl -X POST '[http://localhost:9200/\_reindex](http://localhost:9200/_reindex)' > > > > {"error":"InvalidIndexNameException[[\_reindex] Invalid index name > > > > [_reindex], must not start with '_']","status":400} > > > > > > > > local@:/var/elasticsearch/elasticsearch-0.90.3/bin$ > > > > curl -X POST '[http://localhost:9200/abc/\_export](http://localhost:9200/abc/_export)' -d '{ > > > > "fields": ["\_id", "\_source", "\_version", "\_index", "\_type"], > > > > "output\_file": "/tmp/es-data/dump-abc-${shard}.gz", > > > > "compression": "gzip" > > > > } > > > > ' > > > > {"error":"InvalidTypeNameException[mapping type name [_export] can't > > > > start with '_']","status":400} > > > > > > > > advance thanks for the help > > > > > > > > On Thu, Aug 1, 2013 at 10:10 AM, pranav amin [parulpatel25@gmail.com](mailto:parulpatel25@gmail.com)wrote: > > > > > > > > > Thank br Bernd. > > > > > Much appreciated. I will play around with this tool and let you know > > > > > if i come across any questions. > > > > > Thanks again. > > > > > > > > > > If you have time and have used any Monitoring tools for ES which you > > > > > would recommend, please let me know. I had played around with some and > > > > > mentioned below. > > > > > > > > > > Thanks again > > > > > Pranav. > > > > > > > > > > On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote: > > > > > > > > > > > Hi Pranav > > > > > > > > > > > > You might want to take a look at [Crate.io 路 GitHub](https://github.com/crate/)\*\* > > > > > > elasticsearch-inout-plugin[https://github.com/crate/elasticsearch-inout-plugin](https://github.com/crate/elasticsearch-inout-plugin)we use it for backup restore. The plugin allows you to export/import json > > > > > > data distributed - without having to download the data to the client. > > > > > > However there is no UI for this plugin. > > > > > > > > > > > > br Bernd > > > > > > > > > > > > On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote: > > > > > > > > > > > > > Hi There, > > > > > > > > > > > > > > I'm a new user for Elastic search and plan to use it in Production > > > > > > > environment. > > > > > > > > > > > > > > Can you please help me with some information - > > > > > > > > > > > > > > Monitoring tools - > > > > > > > I looked at the following. > > > > > > > a) Elasticsearch head > > > > > > > b) Bigdesk > > > > > > > c) Paramedic > > > > > > > d) ElasticHQ > > > > > > > > > > > > > > Not sure which of the above are free and recommend by others? > > > > > > > > > > > > > > Backup & Restore - > > > > > > > a) Can i do backup and restore of a Node in a Cluster? Are there any > > > > > > > free/cost based tools (UI or Script based) that can do this and are > > > > > > > frequently recommended by others? > > > > > > > b) Are there any free/cost based tools (UI or Script based) that can > > > > > > > do backup & restore of ACTIVE AND NON-ACTIVE cluster? > > > > > > > > > > > > > > Thanks for the help and advice. > > > > > > > Much appreciated. > > > > > > > > > > > > > > Thanks > > > > > > > Pranav. > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe) > > > > . > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe](https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 6, 2013, 8:59pm UTC](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015/20 "2013-09-06T20:59:27Z")
Good question, I think Linux ext4 can handle 100m dir entries quite well if it was configured to do so - but, many file based tools will not work properly with 100m dir entries, for example ls or rm -rf. Either your main memory will blow up, or wait times will become excessively long. Because of this annoying side effects, I recommend using a tar archive reader routine for processing 100m files in a while loop from the (compressed) single archive file. It will save a lot of resources. J枚rg On Fri, Sep 6, 2013 at 3:54 PM, Craig Lebowitz [craiglebowitz@gmail.com](mailto:craiglebowitz@gmail.com)wrote: > Jorg one question for you: if I extract the knapsack archive, will the > file system have a problem with 100m files? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). [Next page](https://discuss.elastic.co/t/monitoring-backup-restore-tools-for-elastic-search/13015.md?page=2)