Daily indices into monthly

Elastic Stack Logstash
1

I read @warkolm post about moving daily indices into monthly and i thank him for the great example. ( Example )

I have couple hundred individual indexes which i managed to create the logstash.conf file using a ruby program. My question here is, if i try to merge daily indices into monthly with the process below, will it duplicate the data if i re-run after the logstash crashes somehow and if yes, is there a way to avoid it.?

input {
  elasticsearch {
    hosts => [ "xx.xx.xx.xx" ]
    index => "cloudtrail-test1-2017.08.*"
    size => 1000
    scroll => "5m"
    docinfo => true
  }
 elasticsearch {
    hosts => [ "xx.xx.xx.xx" ]
    index => "cloudtrail-test2-2017.08.*"
    size => 1000
    scroll => "5m"
    docinfo => true
  }
}

output {
  elasticsearch {
    hosts => [ "xx.xx.xx.xx" ]
    index => "%{[@metadata][_type]}-%{+YYYY-MM}"
    document_type => "%{[@metadata][_type]}"
    document_id => "%{[@metadata][_id]}"
  }
  stdout {
    codec => "dots"
  }
}

--
Niraj

2

You should use the _reindex API for this, then there is no reliance on Logstash :slight_smile:

Here's an example on how to do it with your data -https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docs-reindex.html#_reindex_daily_indices

3

Elasticsearch Curator can also help with this.

1 Like
4

@theuntergeek

Can you give me an short example how to accomplish this?

5

The first example in the Filter-Selected Indices section of the request_body documentation has a very good example of what you're seeking.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.