Elasticsearch Backup/Archive Option


(Puneet Garg) #1

Hi,
I am new to elasticsearch. I am not able to find good read on elasticsearch
data backup/archive.
I am setting up graylog2 with elasticsearch. I want to check can we archive
data in elasticsearch which is non active. It may be monthly basis. So my
elasticsearch don't have stale data which will help elasticsearch to be
fast. In future If I need that archive data can I import that as well?

Thanks for your help in advance.

  • Puneet

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(dobe) #2

hi

we just wrote a plugin for exporting data based on a query directly on the
data nodes, and we are currently working on the re-import . i think this is
exactly what you need. take look at the examples in the readme

br, bernd

On Monday, April 29, 2013 9:38:21 AM UTC+2, Puneet Garg wrote:

Hi,
I am new to elasticsearch. I am not able to find good read on
elasticsearch data backup/archive.
I am setting up graylog2 with elasticsearch. I want to check can we
archive data in elasticsearch which is non active. It may be monthly
basis. So my elasticsearch don't have stale data which will help
elasticsearch to be fast. In future If I need that archive data can I
import that as well?

Thanks for your help in advance.

  • Puneet

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Puneet Garg) #3

Thanks will check the same.

-Puneet

On Mon, Apr 29, 2013 at 3:19 PM, dobe bernddorn@gmail.com wrote:

hi

we just wrote a plugin for exporting data based on a query directly on the
data nodes, and we are currently working on the re-import . i think this is
exactly what you need. take look at the examples in the readme
https://github.com/crate/elasticsearch-inout-plugin

br, bernd

On Monday, April 29, 2013 9:38:21 AM UTC+2, Puneet Garg wrote:

Hi,
I am new to elasticsearch. I am not able to find good read on
elasticsearch data backup/archive.
I am setting up graylog2 with elasticsearch. I want to check can we
archive data in elasticsearch which is non active. It may be monthly
basis. So my elasticsearch don't have stale data which will help
elasticsearch to be fast. In future If I need that archive data can I
import that as well?

Thanks for your help in advance.

  • Puneet

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/GVozCDew3Q0/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Cheers!!,
Puneet Garg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Puneet Garg) #4

Hi,
I installed below mentioned plugin. When I try to run below command:
curl -X POST 'http://localhost:9200/_export' -d '{
"fields": ["_id", "_source"],
"output_file":
"/home/pgarg/puneet/study_projects/logstash/archive/dump-${index}-${shard}"
}
'
I am getting "curl: (52) Empty reply from server". Do you have any idea?

My understanding is it will create new index called export and put all
data which needs to be exported under that and in mentioned file as json.
Am I correct? In readme.txt it's mentioned we can look for all or selected
indexes. Can you please give me an example? Basically I am looking for
indexes those starts with graylog2
i.e. prefix. Is there a way where in we
can mention indexes prefix to run query in elasticsearch?

-Puneet

On Mon, Apr 29, 2013 at 3:26 PM, Puneet Garg puneet.garg0109@gmail.comwrote:

Thanks will check the same.

-Puneet

On Mon, Apr 29, 2013 at 3:19 PM, dobe bernddorn@gmail.com wrote:

hi

we just wrote a plugin for exporting data based on a query directly on
the data nodes, and we are currently working on the re-import . i think
this is exactly what you need. take look at the examples in the readme
https://github.com/crate/elasticsearch-inout-plugin

br, bernd

On Monday, April 29, 2013 9:38:21 AM UTC+2, Puneet Garg wrote:

Hi,
I am new to elasticsearch. I am not able to find good read on
elasticsearch data backup/archive.
I am setting up graylog2 with elasticsearch. I want to check can we
archive data in elasticsearch which is non active. It may be monthly
basis. So my elasticsearch don't have stale data which will help
elasticsearch to be fast. In future If I need that archive data can I
import that as well?

Thanks for your help in advance.

  • Puneet

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/GVozCDew3Q0/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Cheers!!,
Puneet Garg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(dobe) #5

On May 1, 2013, at 12:36 , Puneet Garg puneet.garg0109@gmail.com wrote:

Hi,
I installed below mentioned plugin. When I try to run below command:
curl -X POST 'http://localhost:9200/_export' -d '{
"fields": ["_id", "_source"],
"output_file": "/home/pgarg/puneet/study_projects/logstash/archive/dump-${index}-${shard}"
}
'
I am getting "curl: (52) Empty reply from server". Do you have any idea?

hi

could you please tell me which version of elasticsearch you use. Or you could also investigate the elasticsearch log files, maybe there is more information about this failure.

My understanding is it will create new index called _export and put all data which needs to be exported under that and in mentioned file as json. Am I correct?

No this is not the case. It directly exports the data in json format to the local filesystem of each node to the given path. There should be one file for each shard. It does not create any index.

In readme.txt it's mentioned we can look for all or selected indexes. Can you please give me an example? Basically I am looking for indexes those starts with graylog2_ i.e. prefix. Is there a way where in we can mention indexes prefix to run query in elasticsearch?

This works the same way the _search endpoint works in elasticsearch. see http://www.elasticsearch.org/guide/reference/api/multi-index/

So your request should look like this (with gzip compression on):
curl -vvX POST 'http://localhost:9200/graylog2_*/_export' -d '{
"fields": ["_id", "_source"],
"output_file": "/home/pgarg/puneet/study_projects/logstash/archive/dump-${index}-${shard}.gz",
"compression": "gzip"
}

Note that this export all matching indexes, which might not be what you want. If you do this on a daily basis you might want to export only the newest index because any other would have been already exported. You might also want to take a look at the force_overwrite option described on https://github.com/crate/elasticsearch-inout-plugin which forces to overwrite files that have been generated in a previous run.

Hope this helps, Bernd

-Puneet

On Mon, Apr 29, 2013 at 3:26 PM, Puneet Garg puneet.garg0109@gmail.com wrote:
Thanks will check the same.

-Puneet

On Mon, Apr 29, 2013 at 3:19 PM, dobe bernddorn@gmail.com wrote:
hi

we just wrote a plugin for exporting data based on a query directly on the data nodes, and we are currently working on the re-import . i think this is exactly what you need. take look at the examples in the readme https://github.com/crate/elasticsearch-inout-plugin

br, bernd

On Monday, April 29, 2013 9:38:21 AM UTC+2, Puneet Garg wrote:
Hi,
I am new to elasticsearch. I am not able to find good read on elasticsearch data backup/archive.
I am setting up graylog2 with elasticsearch. I want to check can we archive data in elasticsearch which is non active. It may be monthly basis. So my elasticsearch don't have stale data which will help elasticsearch to be fast. In future If I need that archive data can I import that as well?

Thanks for your help in advance.

  • Puneet

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/GVozCDew3Q0/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Cheers!!,
Puneet Garg

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/GVozCDew3Q0/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Puneet Garg) #6

Hi Bernd,
I was using elasticsearch-0.20.6 and it was breaking with below error:
java.lang.NoSuchMethodError:
org.elasticsearch.cluster.routing.operation.OperationRouting.searchShards(Lorg/elasticsearch/cluster/ClusterState;[Ljava/lang/String;[Ljava/lang/String;Ljava/util/Map;Ljava/lang/String;)Lorg/elasticsearch/cluster/routing/GroupShardsIterator;

I checked APIs and found method signature changed. I upgraded
to elasticsearch-0.90.0 and tried plugin it's working!!
Thanks a lot for you help.

I agree with your point that I would not like to export already exported
data. I will be deleting data after exporting that will make elasticsearch
quick by not having stale data. So already exported data will not be
available next time. As mentioned, I will be waiting for importing option,
so in future I want to analyze patterns, I will import all exported data
and that should do the job.

-Puneet

On Thu, May 2, 2013 at 11:59 AM, Bernd Dorn bernddorn@gmail.com wrote:

On May 1, 2013, at 12:36 , Puneet Garg puneet.garg0109@gmail.com wrote:

Hi,
I installed below mentioned plugin. When I try to run below command:
curl -X POST 'http://localhost:9200/_export' -d '{
"fields": ["_id", "_source"],
"output_file":
"/home/pgarg/puneet/study_projects/logstash/archive/dump-${index}-${shard}"
}
'
I am getting "curl: (52) Empty reply from server". Do you have any idea?

hi

could you please tell me which version of elasticsearch you use. Or you
could also investigate the elasticsearch log files, maybe there is more
information about this failure.

My understanding is it will create new index called _export and put all
data which needs to be exported under that and in mentioned file as json.
Am I correct?

No this is not the case. It directly exports the data in json format to
the local filesystem of each node to the given path. There should be one
file for each shard. It does not create any index.

In readme.txt it's mentioned we can look for all or selected indexes. Can
you please give me an example? Basically I am looking for indexes those
starts with graylog2_ i.e. prefix. Is there a way where in we can mention
indexes prefix to run query in elasticsearch?

This works the same way the _search endpoint works in elasticsearch. see
http://www.elasticsearch.org/guide/reference/api/multi-index/

So your request should look like this (with gzip compression on):
curl -vvX POST 'http://localhost:9200/graylog2_*/_export' -d '{
"fields": ["_id", "_source"],
"output_file":
"/home/pgarg/puneet/study_projects/logstash/archive/dump-${index}-${shard}.gz",
"compression": "gzip"
}

Note that this export all matching indexes, which might not be what you
want. If you do this on a daily basis you might want to export only the
newest index because any other would have been already exported. You might
also want to take a look at the force_overwrite option described on
https://github.com/crate/elasticsearch-inout-plugin which forces to
overwrite files that have been generated in a previous run.

Hope this helps, Bernd

-Puneet

On Mon, Apr 29, 2013 at 3:26 PM, Puneet Garg puneet.garg0109@gmail.comwrote:

Thanks will check the same.

-Puneet

On Mon, Apr 29, 2013 at 3:19 PM, dobe bernddorn@gmail.com wrote:

hi

we just wrote a plugin for exporting data based on a query directly on
the data nodes, and we are currently working on the re-import . i think
this is exactly what you need. take look at the examples in the readme
https://github.com/crate/elasticsearch-inout-plugin

br, bernd

On Monday, April 29, 2013 9:38:21 AM UTC+2, Puneet Garg wrote:

Hi,
I am new to elasticsearch. I am not able to find good read on
elasticsearch data backup/archive.
I am setting up graylog2 with elasticsearch. I want to check can we
archive data in elasticsearch which is non active. It may be monthly
basis. So my elasticsearch don't have stale data which will help
elasticsearch to be fast. In future If I need that archive data can I
import that as well?

Thanks for your help in advance.

  • Puneet

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/GVozCDew3Q0/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Cheers!!,
Puneet Garg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/GVozCDew3Q0/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/GVozCDew3Q0/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Andrzej Bremer) #7

Hi,

I'm trying to use elasticsearch-inout plugin but I get error:
curl: (52) Empty reply from server

In elasticsearch logs I found
java.lang.NoSuchMethodError:
org.elasticsearch.rest.action.support.RestActions.splitIndices(Ljava/lang/String;)[Ljava/lang/String;
at
crate.elasticsearch.rest.action.admin.export.RestExportAction.handleRequest(RestExportAction.java:55)
...

Have you any idea how to handle this error?

Andrzej

W dniu poniedziałek, 29 kwietnia 2013 09:38:21 UTC+2 użytkownik Puneet Garg
napisał:

Hi,
I am new to elasticsearch. I am not able to find good read on
elasticsearch data backup/archive.
I am setting up graylog2 with elasticsearch. I want to check can we
archive data in elasticsearch which is non active. It may be monthly
basis. So my elasticsearch don't have stale data which will help
elasticsearch to be fast. In future If I need that archive data can I
import that as well?

Thanks for your help in advance.

  • Puneet

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #8

Are you using with Elasticsearch > 0.90.3?
I'm asking because it sounds like the plugin has not been updated for later versions.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 8 nov. 2013 à 13:55, Andrzej Bremer bandister@gmail.com a écrit :

Hi,

I'm trying to use elasticsearch-inout plugin but I get error:
curl: (52) Empty reply from server

In elasticsearch logs I found
java.lang.NoSuchMethodError: org.elasticsearch.rest.action.support.RestActions.splitIndices(Ljava/lang/String;)[Ljava/lang/String;
at crate.elasticsearch.rest.action.admin.export.RestExportAction.handleRequest(RestExportAction.java:55)
...

Have you any idea how to handle this error?

Andrzej

W dniu poniedziałek, 29 kwietnia 2013 09:38:21 UTC+2 użytkownik Puneet Garg napisał:

Hi,
I am new to elasticsearch. I am not able to find good read on elasticsearch data backup/archive.
I am setting up graylog2 with elasticsearch. I want to check can we archive data in elasticsearch which is non active. It may be monthly basis. So my elasticsearch don't have stale data which will help elasticsearch to be fast. In future If I need that archive data can I import that as well?

Thanks for your help in advance.

  • Puneet

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Andrzej Bremer) #9

Hi,

I'm using elasticsearch 0.90.6 for testing purposes. So I will try older
version of elasticsearch.

Andrzej

W dniu poniedziałek, 29 kwietnia 2013 09:38:21 UTC+2 użytkownik Puneet Garg
napisał:

Hi,
I am new to elasticsearch. I am not able to find good read on
elasticsearch data backup/archive.
I am setting up graylog2 with elasticsearch. I want to check can we
archive data in elasticsearch which is non active. It may be monthly
basis. So my elasticsearch don't have stale data which will help
elasticsearch to be fast. In future If I need that archive data can I
import that as well?

Thanks for your help in advance.

  • Puneet

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Andrzej Bremer) #10

Hi,

With elastichsearch 0.90.3 version elasticsearch-inout-plugin works fine.
So, as David said, it's a compatibility problem with elasticserarch
versions > 0.90.3.

Andrzej

W dniu poniedziałek, 29 kwietnia 2013 09:38:21 UTC+2 użytkownik Puneet Garg
napisał:

Hi,
I am new to elasticsearch. I am not able to find good read on
elasticsearch data backup/archive.
I am setting up graylog2 with elasticsearch. I want to check can we
archive data in elasticsearch which is non active. It may be monthly
basis. So my elasticsearch don't have stale data which will help
elasticsearch to be fast. In future If I need that archive data can I
import that as well?

Thanks for your help in advance.

  • Puneet

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Ivan Brusic) #11

I am also interested in creating some type of archiving tool, so I updated
the code and sent a pull request

0.90.4: https://github.com/brusic/elasticsearch-inout-plugin/tree/0.90.4
0.90.6: https://github.com/brusic/elasticsearch-inout-plugin/tree/0.90.6

The initial phase of the elasticsearch snapshot/restore API has just been
committed: https://github.com/elasticsearch/elasticsearch/issues/3826

I will probably hold off until 1.0 is released.

Cheers,

Ivan

On Tue, Nov 12, 2013 at 5:00 AM, Andrzej Bremer bandister@gmail.com wrote:

Hi,

With elastichsearch 0.90.3 version elasticsearch-inout-plugin works fine.
So, as David said, it's a compatibility problem with elasticserarch
versions > 0.90.3.

Andrzej

W dniu poniedziałek, 29 kwietnia 2013 09:38:21 UTC+2 użytkownik Puneet
Garg napisał:

Hi,
I am new to elasticsearch. I am not able to find good read on
elasticsearch data backup/archive.
I am setting up graylog2 with elasticsearch. I want to check can we
archive data in elasticsearch which is non active. It may be monthly
basis. So my elasticsearch don't have stale data which will help
elasticsearch to be fast. In future If I need that archive data can I
import that as well?

Thanks for your help in advance.

  • Puneet

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #12