Daily indices into monthly

[niraj_kumar]

I read @warkolm post about moving daily indices into monthly and i thank him for the great example. ( Example )

I have couple hundred individual indexes which i managed to create the logstash.conf file using a ruby program. My question here is, if i try to merge daily indices into monthly with the process below, will it duplicate the data if i re-run after the logstash crashes somehow and if yes, is there a way to avoid it.?

input {
  elasticsearch {
    hosts => [ "xx.xx.xx.xx" ]
    index => "cloudtrail-test1-2017.08.*"
    size => 1000
    scroll => "5m"
    docinfo => true
  }
 elasticsearch {
    hosts => [ "xx.xx.xx.xx" ]
    index => "cloudtrail-test2-2017.08.*"
    size => 1000
    scroll => "5m"
    docinfo => true
  }
}

output {
  elasticsearch {
    hosts => [ "xx.xx.xx.xx" ]
    index => "%{[@metadata][_type]}-%{+YYYY-MM}"
    document_type => "%{[@metadata][_type]}"
    document_id => "%{[@metadata][_id]}"
  }
  stdout {
    codec => "dots"
  }
}

--
Niraj

[warkolm (Mark Walkom)]

You should use the _reindex API for this, then there is no reliance on Logstash

Here's an example on how to do it with your data -https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docs-reindex.html#_reindex_daily_indices

[theuntergeek (Aaron Mildenstein)]

Elasticsearch Curator can also help with this.

[niraj_kumar]

@theuntergeek

Can you give me an short example how to accomplish this?

[theuntergeek (Aaron Mildenstein)]

The first example in the Filter-Selected Indices section of the request_body documentation has a very good example of what you're seeking.