Combine 2 indexes into one

Elastic Stack Elasticsearch
1

Hi,

ES newbie here. Running an ELK stack on all current versions. Have also got Graylog2 installed.

I asked this on the Graylog list but go no response, but in short during learning the ropes I have ended up with two Graylog indexes, an "active" one managed by GL and an "orphan", which was created when I first started learning, but which GL no longer knows about.

The data structures are the same, so is there a way to copy the "orphaned" data into the current index?

I have searched the net, but have yet to find a solution, well one which I understand!

Thanks again.

--
Roland

2

You can use logstash to reindex from one index to another.
Read this: http://david.pilato.fr/blog/2015/05/20/reindex-elasticsearch-with-logstash/

HTH

3

Thanks David,

I'll give this a go. I did try this method last week but logstash (v1.5) errored. I'll follow your approach next and see if I get a successful result.

--
Roland

4

Hi again - got busy with my day job so couldn't get back onto this as soon as I would like.

Anyway, using Logstash I can successfully re-index if the output is the creation of a new index.

If I try to re-index into and existing index (which is what I want to do), I get the following error in my Logstash logs:

{:timestamp=>"2015-06-21T08:38:53.102000+1200", :message=>"failed action with response of 403, dropping action: [\"index\", {:_id=>\"51519f01-c850-11e4-9bed-00249b0be5cb\", :_index=>\"hill_log2_1\", :_type=>\"message\", :_routing=>nil}, #<LogStash::Event:0x7b2d159d @metadata_accessors=#<LogStash::Util::Accessors:0x5071820e @store={\"_index\"=>\"hill_log_0\", \"_type\"=>\"message\", \"_id\"=>\"51519f01-c850-11e4-9bed-00249b0be5cb\", \"retry_count\"=>0}, @lut={\"[_type]\"=>[{\"_index\"=>\"hill_log_0\", \"_type\"=>\"message\", \"_id\"=>\"51519f01-c850-11e4-9bed-00249b0be5cb\", \"retry_count\"=>0}, \"_type\"], \"[_id]\"=>[{\"_index\"=>\"hill_log_0\", \"_type\"=>\"message\", \"_id\"=>\"51519f01-c850-11e4-9bed-00249b0be5cb\", \"retry_count\"=>0}, \"_id\"]}>, @cancelled=false, @data={\"syslog_severity_code\"=>5,.....etc

This is snipped....sorry if it's too much.

I'll research the 403 error but if anyone knows more about this I'd appreciate feedback.

Thanks,

--
Roland