Daily indices into monthly


(Niraj Kumar) #1

I read @warkolm post about moving daily indices into monthly and i thank him for the great example. ( Example )

I have couple hundred individual indexes which i managed to create the logstash.conf file using a ruby program. My question here is, if i try to merge daily indices into monthly with the process below, will it duplicate the data if i re-run after the logstash crashes somehow and if yes, is there a way to avoid it.?

input {
  elasticsearch {
    hosts => [ "xx.xx.xx.xx" ]
    index => "cloudtrail-test1-2017.08.*"
    size => 1000
    scroll => "5m"
    docinfo => true
  }
 elasticsearch {
    hosts => [ "xx.xx.xx.xx" ]
    index => "cloudtrail-test2-2017.08.*"
    size => 1000
    scroll => "5m"
    docinfo => true
  }
}

output {
  elasticsearch {
    hosts => [ "xx.xx.xx.xx" ]
    index => "%{[@metadata][_type]}-%{+YYYY-MM}"
    document_type => "%{[@metadata][_type]}"
    document_id => "%{[@metadata][_id]}"
  }
  stdout {
    codec => "dots"
  }
}

--
Niraj


(Mark Walkom) #2

You should use the _reindex API for this, then there is no reliance on Logstash :slight_smile:

Here's an example on how to do it with your data -https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docs-reindex.html#_reindex_daily_indices


(Aaron Mildenstein) #3

Elasticsearch Curator can also help with this.


(Niraj Kumar) #4

@theuntergeek

Can you give me an short example how to accomplish this?


(Aaron Mildenstein) #5

The first example in the Filter-Selected Indices section of the request_body documentation has a very good example of what you're seeking.


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.