Monitoring, Backup & Restore Tools for Elastic search


(pranav amin) #1

Hi There,

I'm a new user for Elastic search and plan to use it in Production
environment.

Can you please help me with some information -

Monitoring tools -
I looked at the following.
a) Elasticsearch head
b) Bigdesk
c) Paramedic
d) ElasticHQ

Not sure which of the above are free and recommend by others?

Backup & Restore -
a) Can i do backup and restore of a Node in a Cluster? Are there any
free/cost based tools (UI or Script based) that can do this and are
frequently recommended by others?
b) Are there any free/cost based tools (UI or Script based) that can do
backup & restore of ACTIVE AND NON-ACTIVE cluster?

Thanks for the help and advice.
Much appreciated.

Thanks
Pranav.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(dobe) #2

Hi Pranav

You might want to take a look
at https://github.com/crate/elasticsearch-inout-plugin we use it for backup
restore. The plugin allows you to export/import json data distributed -
without having to download the data to the client. However there is no UI
for this plugin.

br Bernd

On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote:

Hi There,

I'm a new user for Elastic search and plan to use it in Production
environment.

Can you please help me with some information -

Monitoring tools -
I looked at the following.
a) Elasticsearch head
b) Bigdesk
c) Paramedic
d) ElasticHQ

Not sure which of the above are free and recommend by others?

Backup & Restore -
a) Can i do backup and restore of a Node in a Cluster? Are there any
free/cost based tools (UI or Script based) that can do this and are
frequently recommended by others?
b) Are there any free/cost based tools (UI or Script based) that can do
backup & restore of ACTIVE AND NON-ACTIVE cluster?

Thanks for the help and advice.
Much appreciated.

Thanks
Pranav.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(pranav amin) #3

Thank br Bernd.
Much appreciated. I will play around with this tool and let you know if i
come across any questions.
Thanks again.

If you have time and have used any Monitoring tools for ES which you would
recommend, please let me know. I had played around with some and mentioned
below.

Thanks again
Pranav.

On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote:

Hi Pranav

You might want to take a look at
https://github.com/crate/elasticsearch-inout-plugin we use it for backup
restore. The plugin allows you to export/import json data distributed -
without having to download the data to the client. However there is no UI
for this plugin.

br Bernd

On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote:

Hi There,

I'm a new user for Elastic search and plan to use it in Production
environment.

Can you please help me with some information -

Monitoring tools -
I looked at the following.
a) Elasticsearch head
b) Bigdesk
c) Paramedic
d) ElasticHQ

Not sure which of the above are free and recommend by others?

Backup & Restore -
a) Can i do backup and restore of a Node in a Cluster? Are there any
free/cost based tools (UI or Script based) that can do this and are
frequently recommended by others?
b) Are there any free/cost based tools (UI or Script based) that can do
backup & restore of ACTIVE AND NON-ACTIVE cluster?

Thanks for the help and advice.
Much appreciated.

Thanks
Pranav.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(pranav amin) #4

Hi bernd,

when i try to use the plug in for export it gives me the following issue,
any idea what am i missing?

local@:/var/elasticsearch/elasticsearch-0.90.3/bin$
curl -X POST 'http://localhost:9200/_reindex'
{"error":"InvalidIndexNameException[[_reindex] Invalid index name
[reindex], must not start with '']","status":400}

local@:/var/elasticsearch/elasticsearch-0.90.3/bin$
curl -X POST 'http://localhost:9200/abc/_export' -d '{
"fields": ["_id", "_source", "_version", "_index", "_type"],
"output_file": "/tmp/es-data/dump-abc-${shard}.gz",
"compression": "gzip"
}
'
{"error":"InvalidTypeNameException[mapping type name [export] can't start
with '
']","status":400}

advance thanks for the help

On Thu, Aug 1, 2013 at 10:10 AM, pranav amin parulpatel25@gmail.com wrote:

Thank br Bernd.
Much appreciated. I will play around with this tool and let you know if i
come across any questions.
Thanks again.

If you have time and have used any Monitoring tools for ES which you would
recommend, please let me know. I had played around with some and mentioned
below.

Thanks again
Pranav.

On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote:

Hi Pranav

You might want to take a look at https://github.com/crate/**
elasticsearch-inout-pluginhttps://github.com/crate/elasticsearch-inout-pluginwe use it for backup restore. The plugin allows you to export/import json
data distributed - without having to download the data to the client.
However there is no UI for this plugin.

br Bernd

On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote:

Hi There,

I'm a new user for Elastic search and plan to use it in Production
environment.

Can you please help me with some information -

Monitoring tools -
I looked at the following.
a) Elasticsearch head
b) Bigdesk
c) Paramedic
d) ElasticHQ

Not sure which of the above are free and recommend by others?

Backup & Restore -
a) Can i do backup and restore of a Node in a Cluster? Are there any
free/cost based tools (UI or Script based) that can do this and are
frequently recommended by others?
b) Are there any free/cost based tools (UI or Script based) that can do
backup & restore of ACTIVE AND NON-ACTIVE cluster?

Thanks for the help and advice.
Much appreciated.

Thanks
Pranav.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #5

Your plugin is not installed nor detected by elasticsearch.

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2013 à 17:51, parul patel parulpatel25@gmail.com a écrit :

Hi bernd,

when i try to use the plug in for export it gives me the following issue, any idea what am i missing?

local@:/var/elasticsearch/elasticsearch-0.90.3/bin$
curl -X POST 'http://localhost:9200/_reindex'
{"error":"InvalidIndexNameException[[_reindex] Invalid index name [reindex], must not start with '']","status":400}

local@:/var/elasticsearch/elasticsearch-0.90.3/bin$
curl -X POST 'http://localhost:9200/abc/_export' -d '{
"fields": ["_id", "_source", "_version", "_index", "_type"],
"output_file": "/tmp/es-data/dump-abc-${shard}.gz",
"compression": "gzip"
}
'
{"error":"InvalidTypeNameException[mapping type name [export] can't start with '']","status":400}

advance thanks for the help

On Thu, Aug 1, 2013 at 10:10 AM, pranav amin parulpatel25@gmail.com wrote:

Thank br Bernd.
Much appreciated. I will play around with this tool and let you know if i come across any questions.
Thanks again.

If you have time and have used any Monitoring tools for ES which you would recommend, please let me know. I had played around with some and mentioned below.

Thanks again
Pranav.

On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote:

Hi Pranav

You might want to take a look at https://github.com/crate/elasticsearch-inout-plugin we use it for backup restore. The plugin allows you to export/import json data distributed - without having to download the data to the client. However there is no UI for this plugin.

br Bernd

On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote:

Hi There,

I'm a new user for Elastic search and plan to use it in Production environment.

Can you please help me with some information -

Monitoring tools -
I looked at the following.
a) Elasticsearch head
b) Bigdesk
c) Paramedic
d) ElasticHQ

Not sure which of the above are free and recommend by others?

Backup & Restore -
a) Can i do backup and restore of a Node in a Cluster? Are there any free/cost based tools (UI or Script based) that can do this and are frequently recommended by others?
b) Are there any free/cost based tools (UI or Script based) that can do backup & restore of ACTIVE AND NON-ACTIVE cluster?

Thanks for the help and advice.
Much appreciated.

Thanks
Pranav.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(pranav amin) #6

Thanks.
I installed it like this, it looked successful. but i don't know how to
make it detect by ES.

ocal@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$sudo
../plugin -i elasticsearch-inout-plugin -url
file:///$PWD/target/elasticsearch-inout-plugin-0.5.0.jar<file:///\$PWD\target\elasticsearch-inout-plugin-0.5.0.jar>

-> Installing elasticsearch-inout-plugin...

Trying
file:////var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master/target/elasticsearch-inout-plugin-0.5.0.jar<file:///\var\elasticsearch\elasticsearch-0.90.3\bin\elasticsearch-inout-plugin-master\target\elasticsearch-inout-plugin-0.5.0.jar>
...

Downloading ..DONE

Installed elasticsearch-inout-plugin into
/var/elasticsearch/elasticsearch-0.90.3/plugins/elasticsearch-inout-plugin

local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$

On Fri, Aug 30, 2013 at 11:58 AM, David Pilato david@pilato.fr wrote:

Your plugin is not installed nor detected by elasticsearch.

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2013 à 17:51, parul patel parulpatel25@gmail.com a écrit :

Hi bernd,

when i try to use the plug in for export it gives me the following issue,
any idea what am i missing?

local@:/var/elasticsearch/elasticsearch-0.90.3/bin$
curl -X POST 'http://localhost:9200/_reindex'
{"error":"InvalidIndexNameException[[_reindex] Invalid index name
[reindex], must not start with '']","status":400}

local@:/var/elasticsearch/elasticsearch-0.90.3/bin$
curl -X POST 'http://localhost:9200/abc/_export' -d '{
"fields": ["_id", "_source", "_version", "_index", "_type"],
"output_file": "/tmp/es-data/dump-abc-${shard}.gz",
"compression": "gzip"
}
'
{"error":"InvalidTypeNameException[mapping type name [export] can't start
with '
']","status":400}

advance thanks for the help

On Thu, Aug 1, 2013 at 10:10 AM, pranav amin parulpatel25@gmail.comwrote:

Thank br Bernd.
Much appreciated. I will play around with this tool and let you know if i
come across any questions.
Thanks again.

If you have time and have used any Monitoring tools for ES which you
would recommend, please let me know. I had played around with some and
mentioned below.

Thanks again
Pranav.

On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote:

Hi Pranav

You might want to take a look at https://github.com/crate/**
elasticsearch-inout-pluginhttps://github.com/crate/elasticsearch-inout-pluginwe use it for backup restore. The plugin allows you to export/import json
data distributed - without having to download the data to the client.
However there is no UI for this plugin.

br Bernd

On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote:

Hi There,

I'm a new user for Elastic search and plan to use it in Production
environment.

Can you please help me with some information -

Monitoring tools -
I looked at the following.
a) Elasticsearch head
b) Bigdesk
c) Paramedic
d) ElasticHQ

Not sure which of the above are free and recommend by others?

Backup & Restore -
a) Can i do backup and restore of a Node in a Cluster? Are there any
free/cost based tools (UI or Script based) that can do this and are
frequently recommended by others?
b) Are there any free/cost based tools (UI or Script based) that can do
backup & restore of ACTIVE AND NON-ACTIVE cluster?

Thanks for the help and advice.
Much appreciated.

Thanks
Pranav.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(pranav amin) #7

hey guys,

sorry false alarm. i needed to restart elastic search after the plug in was
installed.
now it is working.

thanks a lot for the help. and sorry about my mistake.

On Fri, Aug 30, 2013 at 12:06 PM, parul patel parulpatel25@gmail.comwrote:

Thanks.
I installed it like this, it looked successful. but i don't know how to
make it detect by ES.

ocal@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$sudo ../plugin -i elasticsearch-inout-plugin -url
file:///$PWD/target/elasticsearch-inout-plugin-0.5.0.jar

-> Installing elasticsearch-inout-plugin...

Trying
file:////var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master/target/elasticsearch-inout-plugin-0.5.0.jar
...

Downloading ..DONE

Installed elasticsearch-inout-plugin into
/var/elasticsearch/elasticsearch-0.90.3/plugins/elasticsearch-inout-plugin

local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$

On Fri, Aug 30, 2013 at 11:58 AM, David Pilato david@pilato.fr wrote:

Your plugin is not installed nor detected by elasticsearch.

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2013 à 17:51, parul patel parulpatel25@gmail.com a écrit :

Hi bernd,

when i try to use the plug in for export it gives me the following issue,
any idea what am i missing?

local@:/var/elasticsearch/elasticsearch-0.90.3/bin$
curl -X POST 'http://localhost:9200/_reindex'
{"error":"InvalidIndexNameException[[_reindex] Invalid index name
[reindex], must not start with '']","status":400}

local@:/var/elasticsearch/elasticsearch-0.90.3/bin$
curl -X POST 'http://localhost:9200/abc/_export' -d '{
"fields": ["_id", "_source", "_version", "_index", "_type"],
"output_file": "/tmp/es-data/dump-abc-${shard}.gz",
"compression": "gzip"
}
'
{"error":"InvalidTypeNameException[mapping type name [export] can't
start with '
']","status":400}

advance thanks for the help

On Thu, Aug 1, 2013 at 10:10 AM, pranav amin parulpatel25@gmail.comwrote:

Thank br Bernd.
Much appreciated. I will play around with this tool and let you know if
i come across any questions.
Thanks again.

If you have time and have used any Monitoring tools for ES which you
would recommend, please let me know. I had played around with some and
mentioned below.

Thanks again
Pranav.

On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote:

Hi Pranav

You might want to take a look at https://github.com/crate/**
elasticsearch-inout-pluginhttps://github.com/crate/elasticsearch-inout-pluginwe use it for backup restore. The plugin allows you to export/import json
data distributed - without having to download the data to the client.
However there is no UI for this plugin.

br Bernd

On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote:

Hi There,

I'm a new user for Elastic search and plan to use it in Production
environment.

Can you please help me with some information -

Monitoring tools -
I looked at the following.
a) Elasticsearch head
b) Bigdesk
c) Paramedic
d) ElasticHQ

Not sure which of the above are free and recommend by others?

Backup & Restore -
a) Can i do backup and restore of a Node in a Cluster? Are there any
free/cost based tools (UI or Script based) that can do this and are
frequently recommended by others?
b) Are there any free/cost based tools (UI or Script based) that can
do backup & restore of ACTIVE AND NON-ACTIVE cluster?

Thanks for the help and advice.
Much appreciated.

Thanks
Pranav.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(pranav amin) #8

Hi,

I tried to use the elasticsearch-inout plug in for export. It worked fine.
Now when i try to import it - it fails with

local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$ curl -X POST '
http://localhost:9200/abc_newindex/abc/_import' -d '{

"directory": "/tmp/es-data",

"compression": "gzip"

}

'

[2013-08-30 18:02:08,818][INFO ][cluster.metadata ] [Eric]
[abc_newindex] creating index, cause [auto(bulk api)], shards [5]/[1],
mappings []

[2013-08-30 18:02:20,943][WARN ][index.engine.robin ] [Eric]
[abc_newindex][0] failed engine

[2013-08-30 18:02:21,227][DEBUG][action.bulk ] [Eric]
[abc_newindex][0], node[wHAaefkjSre_x6Lj8XS_6A], [P], s[STARTED]: Failed to
execute [org.elasticsearch.action.bulk.BulkShardRequest@591bc6d9]

java.lang.OutOfMemoryError: Java heap space

{"imports":[],"failures":[{"node_id":"Iow0kmoGRwOLSqZ5jEivAQ","reason":"org.elasticsearch.action.FailedNodeException:
Failed node [Iow0kmoGRwOLSqZ5jEivAQ];
org.elasticsearch.transport.RemoteTransportException:
[Druig][inet[/10.3.176.23:9300]][el-crate-import/n];
org.elasticsearch.transport.ActionNotFoundTransportException: No handler
for action
[el-crate-import/n]"},{"node_id":"wHAaefkjSre_x6Lj8XS_6A","reason":"org.elasticsearch.action.FailedNodeException:
Failed node [wHAaefkjSre_x6Lj8XS_6A]; java.lang.OutOfMemoryError: Java heap
space"}]}local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$Exception
in thread "elasticsearch[Eric][bulk][T#2]"
java.lang.OutOfMemoryError: Java heap space

[2013-08-30 18:02:24,845][WARN ][index.engine.robin ] [Eric]
[abc_newindex][0] failed engine

Any idea what is wrong? I can increase the heap size but not sure of when i
will see the error again since i have limited RAM on the box.

On Fri, Aug 30, 2013 at 12:13 PM, parul patel parulpatel25@gmail.comwrote:

hey guys,

sorry false alarm. i needed to restart elastic search after the plug in
was installed.
now it is working.

thanks a lot for the help. and sorry about my mistake.

On Fri, Aug 30, 2013 at 12:06 PM, parul patel parulpatel25@gmail.comwrote:

Thanks.
I installed it like this, it looked successful. but i don't know how to
make it detect by ES.

ocal@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$sudo ../plugin -i elasticsearch-inout-plugin -url
file:///$PWD/target/elasticsearch-inout-plugin-0.5.0.jar

-> Installing elasticsearch-inout-plugin...

Trying
file:////var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master/target/elasticsearch-inout-plugin-0.5.0.jar
...

Downloading ..DONE

Installed elasticsearch-inout-plugin into
/var/elasticsearch/elasticsearch-0.90.3/plugins/elasticsearch-inout-plugin

local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$

On Fri, Aug 30, 2013 at 11:58 AM, David Pilato david@pilato.fr wrote:

Your plugin is not installed nor detected by elasticsearch.

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2013 à 17:51, parul patel parulpatel25@gmail.com a écrit :

Hi bernd,

when i try to use the plug in for export it gives me the following
issue, any idea what am i missing?

local@:/var/elasticsearch/elasticsearch-0.90.3/bin$
curl -X POST 'http://localhost:9200/_reindex'
{"error":"InvalidIndexNameException[[_reindex] Invalid index name
[reindex], must not start with '']","status":400}

local@:/var/elasticsearch/elasticsearch-0.90.3/bin$
curl -X POST 'http://localhost:9200/abc/_export' -d '{
"fields": ["_id", "_source", "_version", "_index", "_type"],
"output_file": "/tmp/es-data/dump-abc-${shard}.gz",
"compression": "gzip"
}
'
{"error":"InvalidTypeNameException[mapping type name [export] can't
start with '
']","status":400}

advance thanks for the help

On Thu, Aug 1, 2013 at 10:10 AM, pranav amin parulpatel25@gmail.comwrote:

Thank br Bernd.
Much appreciated. I will play around with this tool and let you know if
i come across any questions.
Thanks again.

If you have time and have used any Monitoring tools for ES which you
would recommend, please let me know. I had played around with some and
mentioned below.

Thanks again
Pranav.

On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote:

Hi Pranav

You might want to take a look at https://github.com/crate/**
elasticsearch-inout-pluginhttps://github.com/crate/elasticsearch-inout-pluginwe use it for backup restore. The plugin allows you to export/import json
data distributed - without having to download the data to the client.
However there is no UI for this plugin.

br Bernd

On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote:

Hi There,

I'm a new user for Elastic search and plan to use it in Production
environment.

Can you please help me with some information -

Monitoring tools -
I looked at the following.
a) Elasticsearch head
b) Bigdesk
c) Paramedic
d) ElasticHQ

Not sure which of the above are free and recommend by others?

Backup & Restore -
a) Can i do backup and restore of a Node in a Cluster? Are there any
free/cost based tools (UI or Script based) that can do this and are
frequently recommended by others?
b) Are there any free/cost based tools (UI or Script based) that can
do backup & restore of ACTIVE AND NON-ACTIVE cluster?

Thanks for the help and advice.
Much appreciated.

Thanks
Pranav.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(pranav amin) #9

hey guys,

much appreciated if someone can shed light on the above error here?
i can increase the heap size, but that doesn't seem to be like a fix. since
the overall data size is just 46 MB, if i want to do an import of 1Gb or
10Gb, i don't know how much heap will be needed.

may be i'm missing something.

thanks

On Fri, Aug 30, 2013 at 2:30 PM, parul patel parulpatel25@gmail.com wrote:

Hi,

I tried to use the elasticsearch-inout plug in for export. It worked fine.
Now when i try to import it - it fails with

local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$ curl -X POST
'http://localhost:9200/abc_newindex/abc/_import' -d '{

"directory": "/tmp/es-data",

"compression": "gzip"

}

'

[2013-08-30 18:02:08,818][INFO ][cluster.metadata ] [Eric]
[abc_newindex] creating index, cause [auto(bulk api)], shards [5]/[1],
mappings []

[2013-08-30 18:02:20,943][WARN ][index.engine.robin ] [Eric]
[abc_newindex][0] failed engine

[2013-08-30 18:02:21,227][DEBUG][action.bulk ] [Eric]
[abc_newindex][0], node[wHAaefkjSre_x6Lj8XS_6A], [P], s[STARTED]: Failed to
execute [org.elasticsearch.action.bulk.BulkShardRequest@591bc6d9]

java.lang.OutOfMemoryError: Java heap space

{"imports":[],"failures":[{"node_id":"Iow0kmoGRwOLSqZ5jEivAQ","reason":"org.elasticsearch.action.FailedNodeException:
Failed node [Iow0kmoGRwOLSqZ5jEivAQ];
org.elasticsearch.transport.RemoteTransportException:
[Druig][inet[/10.3.176.23:9300]][el-crate-import/n];
org.elasticsearch.transport.ActionNotFoundTransportException: No handler
for action
[el-crate-import/n]"},{"node_id":"wHAaefkjSre_x6Lj8XS_6A","reason":"org.elasticsearch.action.FailedNodeException:
Failed node [wHAaefkjSre_x6Lj8XS_6A]; java.lang.OutOfMemoryError: Java heap
space"}]}local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$Exception in thread "elasticsearch[Eric][bulk][T#2]"
java.lang.OutOfMemoryError: Java heap space

[2013-08-30 18:02:24,845][WARN ][index.engine.robin ] [Eric]
[abc_newindex][0] failed engine

Any idea what is wrong? I can increase the heap size but not sure of when
i will see the error again since i have limited RAM on the box.

On Fri, Aug 30, 2013 at 12:13 PM, parul patel parulpatel25@gmail.comwrote:

hey guys,

sorry false alarm. i needed to restart elastic search after the plug in
was installed.
now it is working.

thanks a lot for the help. and sorry about my mistake.

On Fri, Aug 30, 2013 at 12:06 PM, parul patel parulpatel25@gmail.comwrote:

Thanks.
I installed it like this, it looked successful. but i don't know how to
make it detect by ES.

ocal@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$sudo ../plugin -i elasticsearch-inout-plugin -url
file:///$PWD/target/elasticsearch-inout-plugin-0.5.0.jar

-> Installing elasticsearch-inout-plugin...

Trying
file:////var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master/target/elasticsearch-inout-plugin-0.5.0.jar
...

Downloading ..DONE

Installed elasticsearch-inout-plugin into
/var/elasticsearch/elasticsearch-0.90.3/plugins/elasticsearch-inout-plugin

local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$

On Fri, Aug 30, 2013 at 11:58 AM, David Pilato david@pilato.fr wrote:

Your plugin is not installed nor detected by elasticsearch.

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2013 à 17:51, parul patel parulpatel25@gmail.com a écrit :

Hi bernd,

when i try to use the plug in for export it gives me the following
issue, any idea what am i missing?

local@:/var/elasticsearch/elasticsearch-0.90.3/bin$
curl -X POST 'http://localhost:9200/_reindex'
{"error":"InvalidIndexNameException[[_reindex] Invalid index name
[reindex], must not start with '']","status":400}

local@:/var/elasticsearch/elasticsearch-0.90.3/bin$
curl -X POST 'http://localhost:9200/abc/_export' -d '{
"fields": ["_id", "_source", "_version", "_index", "_type"],
"output_file": "/tmp/es-data/dump-abc-${shard}.gz",
"compression": "gzip"
}
'
{"error":"InvalidTypeNameException[mapping type name [export] can't
start with '
']","status":400}

advance thanks for the help

On Thu, Aug 1, 2013 at 10:10 AM, pranav amin parulpatel25@gmail.comwrote:

Thank br Bernd.
Much appreciated. I will play around with this tool and let you know
if i come across any questions.
Thanks again.

If you have time and have used any Monitoring tools for ES which you
would recommend, please let me know. I had played around with some and
mentioned below.

Thanks again
Pranav.

On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote:

Hi Pranav

You might want to take a look at https://github.com/crate/**
elasticsearch-inout-pluginhttps://github.com/crate/elasticsearch-inout-pluginwe use it for backup restore. The plugin allows you to export/import json
data distributed - without having to download the data to the client.
However there is no UI for this plugin.

br Bernd

On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote:

Hi There,

I'm a new user for Elastic search and plan to use it in Production
environment.

Can you please help me with some information -

Monitoring tools -
I looked at the following.
a) Elasticsearch head
b) Bigdesk
c) Paramedic
d) ElasticHQ

Not sure which of the above are free and recommend by others?

Backup & Restore -
a) Can i do backup and restore of a Node in a Cluster? Are there any
free/cost based tools (UI or Script based) that can do this and are
frequently recommended by others?
b) Are there any free/cost based tools (UI or Script based) that can
do backup & restore of ACTIVE AND NON-ACTIVE cluster?

Thanks for the help and advice.
Much appreciated.

Thanks
Pranav.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Brian Yoder) #10

I have not used this plug-in: I currently use my own tools:

  1. A scan-based query-all that iterates over a scanned snapshot, and emits
    each document together with its appropriate action_and_meta_data line for
    ease of import. Based on your posts, I imagine that the plug-in is doing
    something similar, except that the plug-in is also adding compression (my
    tool emits to stdout so I can also add compression by piping through gzip
    or other compressor). So far, so good.

  2. You try to import using curl. No!!!!!! The curl command is to import as
    your fingernails are to digging the Chunnel. So I wrote my own importer
    that can read the action_and_meta_data line along with the subsequent
    _source line, use the BulkRequestBuilder Java object to collect up to 4096
    of them (configurable, but this works well), and I can import hundreds of
    millions of documents this way with no problems. But I left curl behind
    long ago, using it only for loading a small handful of documents during
    initial testing.

The curl command has too many limitations. You can fire up your C compiler,
get the source, and try to force its behavior to better fit with what you
want. Or you can use the BulkRequestBuilder and do it in Java. But either
way, you have to feed the import in chunks and not as one huge chunk. And I
would strongly recommend against curl for imports into production databases
and for imports of huge numbers of documents in one bulk increment in any
environment.

Brian

On Tuesday, September 3, 2013 9:55:06 AM UTC-4, pranav amin wrote:

hey guys,

much appreciated if someone can shed light on the above error here?
i can increase the heap size, but that doesn't seem to be like a fix.
since the overall data size is just 46 MB, if i want to do an import of 1Gb
or 10Gb, i don't know how much heap will be needed.

may be i'm missing something.

thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(pranav amin) #11

Thanks brian.

Much appreciated. This means i will need to customize or come up with my
own tool to do this.
Any idea if there is any other plugin which works well and i can just reuse
rather than - reinventing the wheel?

thanks

On Tue, Sep 3, 2013 at 11:05 AM, InquiringMind brian.from.fl@gmail.comwrote:

I have not used this plug-in: I currently use my own tools:

  1. A scan-based query-all that iterates over a scanned snapshot, and emits
    each document together with its appropriate action_and_meta_data line for
    ease of import. Based on your posts, I imagine that the plug-in is doing
    something similar, except that the plug-in is also adding compression (my
    tool emits to stdout so I can also add compression by piping through gzip
    or other compressor). So far, so good.

  2. You try to import using curl. No!!!!!! The curl command is to import as
    your fingernails are to digging the Chunnel. So I wrote my own importer
    that can read the action_and_meta_data line along with the subsequent
    _source line, use the BulkRequestBuilder Java object to collect up to 4096
    of them (configurable, but this works well), and I can import hundreds of
    millions of documents this way with no problems. But I left curl behind
    long ago, using it only for loading a small handful of documents during
    initial testing.

The curl command has too many limitations. You can fire up your C
compiler, get the source, and try to force its behavior to better fit with
what you want. Or you can use the BulkRequestBuilder and do it in Java. But
either way, you have to feed the import in chunks and not as one huge
chunk. And I would strongly recommend against curl for imports into
production databases and for imports of huge numbers of documents in one
bulk increment in any environment.

Brian

On Tuesday, September 3, 2013 9:55:06 AM UTC-4, pranav amin wrote:

hey guys,

much appreciated if someone can shed light on the above error here?
i can increase the heap size, but that doesn't seem to be like a fix.
since the overall data size is just 46 MB, if i want to do an import of 1Gb
or 10Gb, i don't know how much heap will be needed.

may be i'm missing something.

thanks

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Brian Yoder) #12

The plug-in that was mentioned earlier looks to be able to do both the
back-up and the restore (export and import):

Looks pretty cool! I haven't used it, but after reading the post that
mentioned it, it seems to be just what you're looking for.

Brian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(pranav amin) #13

Thanks. I had the same opinion. However the curl commands what it has
mentioned in the document/link gave me the out of heap memory error. And
there is no alternative mentioned of how to run that command with Bulk size
configurable to avoid the out of memory error.

Thanks

On Wed, Sep 4, 2013 at 11:17 AM, InquiringMind brian.from.fl@gmail.comwrote:

The plug-in that was mentioned earlier looks to be able to do both the
back-up and the restore (export and import): https://github.com/crate/**
elasticsearch-inout-pluginhttps://github.com/crate/elasticsearch-inout-plugin

Looks pretty cool! I haven't used it, but after reading the post that
mentioned it, it seems to be just what you're looking for.

Brian

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #14

You can try my knapsack plugin which has a "size" URL parameter to control
the bulk export.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(pranav amin) #15

thanks jorg.

is the export/import same as backup/restore or it is different?
does the script only export data from the node it currently runs on or does
the backup on every node in the cluster?

thanks

On Wed, Sep 4, 2013 at 11:42 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

You can try my knapsack plugin which has a "size" URL parameter to control
the bulk export.

https://github.com/jprante/elasticsearch-knapsack

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(dobe) #16

Hi Parul

We actually did not see any memory issues till now on our installations.

So could you please set the log level to debug and send the logs after forcing the memory error.

It would also help to have the following information to find the problem:

  1. elasticsearch version, plugin version (would be best to use 0.90.3 and the latest plugin version)
  2. heap space size
  3. data size, number of data files, number of nodes
  4. the index settings and status output

thx, bernd

On Aug 30, 2013, at 20:30 , parul patel parulpatel25@gmail.com wrote:

Hi,

I tried to use the elasticsearch-inout plug in for export. It worked fine. Now when i try to import it - it fails with

local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$ curl -X POST 'http://localhost:9200/abc_newindex/abc/_import' -d '{

"directory": "/tmp/es-data",

"compression": "gzip"

}

'

[2013-08-30 18:02:08,818][INFO ][cluster.metadata ] [Eric] [abc_newindex] creating index, cause [auto(bulk api)], shards [5]/[1], mappings []

[2013-08-30 18:02:20,943][WARN ][index.engine.robin ] [Eric] [abc_newindex][0] failed engine

[2013-08-30 18:02:21,227][DEBUG][action.bulk ] [Eric] [abc_newindex][0], node[wHAaefkjSre_x6Lj8XS_6A], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.bulk.BulkShardRequest@591bc6d9]

java.lang.OutOfMemoryError: Java heap space

{"imports":[],"failures":[{"node_id":"Iow0kmoGRwOLSqZ5jEivAQ","reason":"org.elasticsearch.action.FailedNodeException: Failed node [Iow0kmoGRwOLSqZ5jEivAQ]; org.elasticsearch.transport.RemoteTransportException: [Druig][inet[/10.3.176.23:9300]][el-crate-import/n]; org.elasticsearch.transport.ActionNotFoundTransportException: No handler for action [el-crate-import/n]"},{"node_id":"wHAaefkjSre_x6Lj8XS_6A","reason":"org.elasticsearch.action.FailedNodeException: Failed node [wHAaefkjSre_x6Lj8XS_6A]; java.lang.OutOfMemoryError: Java heap space"}]}local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$ Exception in thread "elasticsearch[Eric][bulk][T#2]" java.lang.OutOfMemoryError: Java heap space

[2013-08-30 18:02:24,845][WARN ][index.engine.robin ] [Eric] [abc_newindex][0] failed engine

Any idea what is wrong? I can increase the heap size but not sure of when i will see the error again since i have limited RAM on the box.

On Fri, Aug 30, 2013 at 12:13 PM, parul patel parulpatel25@gmail.com wrote:
hey guys,

sorry false alarm. i needed to restart elastic search after the plug in was installed.
now it is working.

thanks a lot for the help. and sorry about my mistake.

On Fri, Aug 30, 2013 at 12:06 PM, parul patel parulpatel25@gmail.com wrote:
Thanks.
I installed it like this, it looked successful. but i don't know how to make it detect by ES.

ocal@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$ sudo ../plugin -i elasticsearch-inout-plugin -url file:///$PWD/target/elasticsearch-inout-plugin-0.5.0.jar

-> Installing elasticsearch-inout-plugin...

Trying file:////var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master/target/elasticsearch-inout-plugin-0.5.0.jar...

Downloading ..DONE

Installed elasticsearch-inout-plugin into /var/elasticsearch/elasticsearch-0.90.3/plugins/elasticsearch-inout-plugin

local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$

On Fri, Aug 30, 2013 at 11:58 AM, David Pilato david@pilato.fr wrote:
Your plugin is not installed nor detected by elasticsearch.

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2013 à 17:51, parul patel parulpatel25@gmail.com a écrit :

Hi bernd,

when i try to use the plug in for export it gives me the following issue, any idea what am i missing?

local@:/var/elasticsearch/elasticsearch-0.90.3/bin$
curl -X POST 'http://localhost:9200/_reindex'
{"error":"InvalidIndexNameException[[_reindex] Invalid index name [reindex], must not start with '']","status":400}

local@:/var/elasticsearch/elasticsearch-0.90.3/bin$
curl -X POST 'http://localhost:9200/abc/_export' -d '{
"fields": ["_id", "_source", "_version", "_index", "_type"],
"output_file": "/tmp/es-data/dump-abc-${shard}.gz",
"compression": "gzip"
}
'
{"error":"InvalidTypeNameException[mapping type name [export] can't start with '']","status":400}

advance thanks for the help

On Thu, Aug 1, 2013 at 10:10 AM, pranav amin parulpatel25@gmail.com wrote:
Thank br Bernd.
Much appreciated. I will play around with this tool and let you know if i come across any questions.
Thanks again.

If you have time and have used any Monitoring tools for ES which you would recommend, please let me know. I had played around with some and mentioned below.

Thanks again
Pranav.

On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote:
Hi Pranav

You might want to take a look at https://github.com/crate/elasticsearch-inout-plugin we use it for backup restore. The plugin allows you to export/import json data distributed - without having to download the data to the client. However there is no UI for this plugin.

br Bernd

On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote:
Hi There,

I'm a new user for Elastic search and plan to use it in Production environment.

Can you please help me with some information -

Monitoring tools -
I looked at the following.
a) Elasticsearch head
b) Bigdesk
c) Paramedic
d) ElasticHQ

Not sure which of the above are free and recommend by others?

Backup & Restore -
a) Can i do backup and restore of a Node in a Cluster? Are there any free/cost based tools (UI or Script based) that can do this and are frequently recommended by others?
b) Are there any free/cost based tools (UI or Script based) that can do backup & restore of ACTIVE AND NON-ACTIVE cluster?

Thanks for the help and advice.
Much appreciated.

Thanks
Pranav.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #17

Knapsack is working on index/type level and scans/scrolls over the _source
fields with variable bulk size. It is enough to install it on one node to
dump/restore the whole cluster.

Knapsack uses tar.gz archives for dump archives by default (also handy for
external post processing since the ES JSON docs can be unpacked to the file
system)

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(lebowitz) #18

I can recommend Jorg's knapsack plugin. It efficiently compresses each document as a file and dames your settings and mapping. When you are using it in import mode, you can specify a new settings and mapping. This makes it handy for manipulation and indexing.

Jorg one question for you: if I extract the knapsack archive, will the file system have a problem with 100m files?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(pranav amin) #19

thanks bernd.

We use -

  1. elastic search version 0.90.3,
  2. Plugin version was picked up from website.
  3. Heap space size is default i.e. 256M
  4. Data size is 46MB
  5. Number of data files - Not sure how do i check this?
  6. Number of nodes - 3
  7. Index settings -
    Settings -
    {"abc":{"settings":{"index.number_of_shards":"5","index.number_of_replicas":"1","index.version.created":"900399"}}}
    Mappings
  • {"abc":{"export":{"properties":{"compression":{"type":"string"},"fields":{"type":"string"},"output_file":{"type":"string"}}},"device":{"properties":{"deviceId":{"type":"string"},"packages":{"properties":{"dependencies":{"properties":{"id":{"type":"string"},"version":{"type":"string"}}},"id":{"type":"string"},"source":{"type":"string"},"version":{"type":"string"}}},"properties":{"properties":{"pin":{"properties":{"source":{"type":"string"},"value":{"type":"string"}}}}},"supportedAPIs":{"properties":{"id":{"type":"string"},"source":{"type":"string"},"type":{"type":"string"},"version":{"type":"string"}}},"ts":{"type":"long"},"vendorId":{"type":"string"}}}}}

Thanks

On Thu, Sep 5, 2013 at 1:06 AM, Bernd Dorn bernddorn@gmail.com wrote:

Hi Parul

We actually did not see any memory issues till now on our installations.

So could you please set the log level to debug and send the logs after
forcing the memory error.

It would also help to have the following information to find the problem:

  1. elasticsearch version, plugin version (would be best to use 0.90.3 and
    the latest plugin version)
  2. heap space size
  3. data size, number of data files, number of nodes
  4. the index settings and status output

thx, bernd

On Aug 30, 2013, at 20:30 , parul patel parulpatel25@gmail.com wrote:

Hi,

I tried to use the elasticsearch-inout plug in for export. It worked fine.
Now when i try to import it - it fails with

local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$ curl -X POST
'http://localhost:9200/abc_newindex/abc/_import' -d '{

"directory": "/tmp/es-data",

"compression": "gzip"

}

'

[2013-08-30 18:02:08,818][INFO ][cluster.metadata ] [Eric]
[abc_newindex] creating index, cause [auto(bulk api)], shards [5]/[1],
mappings []

[2013-08-30 18:02:20,943][WARN ][index.engine.robin ] [Eric]
[abc_newindex][0] failed engine

[2013-08-30 18:02:21,227][DEBUG][action.bulk ] [Eric]
[abc_newindex][0], node[wHAaefkjSre_x6Lj8XS_6A], [P], s[STARTED]: Failed to
execute [org.elasticsearch.action.bulk.BulkShardRequest@591bc6d9]

java.lang.OutOfMemoryError: Java heap space

{"imports":[],"failures":[{"node_id":"Iow0kmoGRwOLSqZ5jEivAQ","reason":"org.elasticsearch.action.FailedNodeException:
Failed node [Iow0kmoGRwOLSqZ5jEivAQ];
org.elasticsearch.transport.RemoteTransportException:
[Druig][inet[/10.3.176.23:9300]][el-crate-import/n];
org.elasticsearch.transport.ActionNotFoundTransportException: No handler
for action
[el-crate-import/n]"},{"node_id":"wHAaefkjSre_x6Lj8XS_6A","reason":"org.elasticsearch.action.FailedNodeException:
Failed node [wHAaefkjSre_x6Lj8XS_6A]; java.lang.OutOfMemoryError: Java heap
space"}]}local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin$Exception in thread "elasticsearch[Eric][bulk][T#2]"
java.lang.OutOfMemoryError: Java heap space

[2013-08-30 18:02:24,845][WARN ][index.engine.robin ] [Eric]
[abc_newindex][0] failed engine

Any idea what is wrong? I can increase the heap size but not sure of when
i will see the error again since i have limited RAM on the box.

On Fri, Aug 30, 2013 at 12:13 PM, parul patel parulpatel25@gmail.comwrote:

hey guys,

sorry false alarm. i needed to restart elastic search after the plug in
was installed.
now it is working.

thanks a lot for the help. and sorry about my mistake.

On Fri, Aug 30, 2013 at 12:06 PM, parul patel parulpatel25@gmail.comwrote:

Thanks.
I installed it like this, it looked successful. but i don't know how to
make it detect by ES.

ocal@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$sudo ../plugin -i elasticsearch-inout-plugin -url
file:///$PWD/target/elasticsearch-inout-plugin-0.5.0.jar

-> Installing elasticsearch-inout-plugin...

Trying
file:////var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master/target/elasticsearch-inout-plugin-0.5.0.jar
...

Downloading ..DONE

Installed elasticsearch-inout-plugin into
/var/elasticsearch/elasticsearch-0.90.3/plugins/elasticsearch-inout-plugin

local@pldsvv-07:/var/elasticsearch/elasticsearch-0.90.3/bin/elasticsearch-inout-plugin-master$

On Fri, Aug 30, 2013 at 11:58 AM, David Pilato david@pilato.fr wrote:

Your plugin is not installed nor detected by elasticsearch.

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 30 août 2013 à 17:51, parul patel parulpatel25@gmail.com a écrit :

Hi bernd,

when i try to use the plug in for export it gives me the following
issue, any idea what am i missing?

local@:/var/elasticsearch/elasticsearch-0.90.3/bin$
curl -X POST 'http://localhost:9200/_reindex'
{"error":"InvalidIndexNameException[[_reindex] Invalid index name
[reindex], must not start with '']","status":400}

local@:/var/elasticsearch/elasticsearch-0.90.3/bin$
curl -X POST 'http://localhost:9200/abc/_export' -d '{
"fields": ["_id", "_source", "_version", "_index", "_type"],
"output_file": "/tmp/es-data/dump-abc-${shard}.gz",
"compression": "gzip"
}
'
{"error":"InvalidTypeNameException[mapping type name [export] can't
start with '
']","status":400}

advance thanks for the help

On Thu, Aug 1, 2013 at 10:10 AM, pranav amin parulpatel25@gmail.comwrote:

Thank br Bernd.
Much appreciated. I will play around with this tool and let you know
if i come across any questions.
Thanks again.

If you have time and have used any Monitoring tools for ES which you
would recommend, please let me know. I had played around with some and
mentioned below.

Thanks again
Pranav.

On Thursday, August 1, 2013 12:57:23 AM UTC-4, dobe wrote:

Hi Pranav

You might want to take a look at https://github.com/crate/**
elasticsearch-inout-pluginhttps://github.com/crate/elasticsearch-inout-pluginwe use it for backup restore. The plugin allows you to export/import json
data distributed - without having to download the data to the client.
However there is no UI for this plugin.

br Bernd

On Wednesday, July 31, 2013 4:50:56 PM UTC+2, pranav amin wrote:

Hi There,

I'm a new user for Elastic search and plan to use it in Production
environment.

Can you please help me with some information -

Monitoring tools -
I looked at the following.
a) Elasticsearch head
b) Bigdesk
c) Paramedic
d) ElasticHQ

Not sure which of the above are free and recommend by others?

Backup & Restore -
a) Can i do backup and restore of a Node in a Cluster? Are there any
free/cost based tools (UI or Script based) that can do this and are
frequently recommended by others?
b) Are there any free/cost based tools (UI or Script based) that can
do backup & restore of ACTIVE AND NON-ACTIVE cluster?

Thanks for the help and advice.
Much appreciated.

Thanks
Pranav.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Pf1tLdAtIp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #20

Good question, I think Linux ext4 can handle 100m dir entries quite well if
it was configured to do so - but, many file based tools will not work
properly with 100m dir entries, for example ls or rm -rf. Either your main
memory will blow up, or wait times will become excessively long.

Because of this annoying side effects, I recommend using a tar archive
reader routine for processing 100m files in a while loop from the
(compressed) single archive file. It will save a lot of resources.

Jörg

On Fri, Sep 6, 2013 at 3:54 PM, Craig Lebowitz craiglebowitz@gmail.comwrote:

Jorg one question for you: if I extract the knapsack archive, will the
file system have a problem with 100m files?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.