Index rollover using curator

[Ihjaz (Ihjaz Mohamed)]

Hi,

I'm trying to get the indices rolled over once it reaches a given number of documents. I have the curator action as shown below:

  2:
    action: rollover
    description: >-
      Rollover the index associated with index 'name', which should be in the
      form of prefix-000001 (or similar), or prefix-YYYY.MM.DD-1.
    options:
      disable_action: False
      name: filebeat_logs
      conditions:
        max_docs: 500

I have added an alias named filebeat_logs using the curator action.

1:
    action: alias
    description: >-
      Alias indices from last week, with a prefix of logstash- to 'last_week',
      remove indices from the previous week.
    options:
      name: filebeat_logs
      warn_if_no_indices: False
      disable_action: False
    add:
      filters:
      - filtertype: pattern
        kind: prefix
        value: filebeat-

The index is created by the logstash as shown below :

input {
  beats {
    host => "10.10.132.40"
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "10.10.132.44:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}-1"
    document_type => "%{[@metadata][type]}"
  }
} 

When I run the curator, it just creates a new empty index and the logs still gets written to the older index.

health status index                      uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   filebeat-2017.08.25-000002 UyCwyENbT1ah96nT79MyqQ   5   1          0            0       955b           955b
yellow open   filebeat-2017.08.25-1      XNB0rSxJRIizISIGqbMOSA   5   1        906            0    775.8kb        775.8kb

I read that I need to right to an alias and then it will start writing to the new file.
But can someone tell me how I can define and write to an alias from the elasticsearch output plugin so that the logs get written to new files every time I run the curator rollover action.

[theuntergeek (Aaron Mildenstein)]

@Ihjaz

I read that I need to right to an alias and then it will start writing to the new file.

That is correct.

@Ihjaz

But can someone tell me how I can define and write to an alias from the elasticsearch output plugin so that the logs get written to new files every time I run the curator rollover action.

This is currently impossible. You need to manually create the initial index + alias to do rollover. The upside is that once this is done, it never needs to be done again—it's a one-time only thing (unless you accidentally delete the alias). The example below is taken from the official Elasticsearch documentation:

PUT /filebeat-000001 
{
  "aliases": {
    "filebeat_logs": {}
  }
}

This will create index filebeat-000001 and alias filebeat_logs will point to it.

Your logstash output block would then look like:

output {
  elasticsearch {
    hosts => "10.10.132.44:9200"
    manage_template => false
    index => "filebeat_logs"
  }
} 

That's it. You point the index directive to the alias.

Extra credit:
Since you've disabled template management, you should manually maintain an index template that has appropriate mappings for indices following the filebeat-* pattern.

I highly recommend getting away from using document_type => "%{[@metadata][type]}", as mapping types are going away in Elasticsearch 6.0, meaning that each index should have one and only one document type. If you were planning on having multiple types per index, now is a good time to adjust that to multiple indices with one type per index, before you get a bunch of indices which won't be upgradable.

[Ihjaz (Ihjaz Mohamed)]

Thanks @theuntergeek.

This is exactly what I wanted.