Winlogbeat ILM and logstash

[tomrade (Thomas)]

Hey everyone ive been having issues getting ILM to work with winlogbeat and logstash

I don't have ES access for winlogbeat and I dont want to create a seperate windows vm (in my container lab) just to setup ILM

First thing I did was export the winlogbeat template and ilm from winlog. via the export command. I also forced ILM in the winlogbeat.yml to make sure it was on prior to export(i didnt trust auto as i had no elasticsearch output enabled)

Then added them to ES using curl. I can see them in kibana and they look fine. My winlogbeat talks to logstash over beats protocol and my output config is as follows. The top "if" is for filebeat modules (which works I used a temp container to setup).

if [@metadata][pipeline] {
        elasticsearch {
            hosts => ["01","02"]
            manage_template => false
            index => "%{[@metadata][beat]}-%{[@metadata][version]}"
            pipeline => "%{[@metadata][pipeline]}"
            user     => 'logstash_ingest'
            password => "${LOGSTASH_PASSWORD}"  # read password from logstash.keystore
            ssl => true
            cacert => '/usr/share/logstash/config/certs/ca/ca.crt'
        }
    # If no pipeline exists but beats meta data is present use it
    } else if [@metadata] {
        elasticsearch {
            hosts => ["01","02"]
            index => "%{[@metadata][beat]}-%{[@metadata][version]}"
            manage_template => false
            user     => 'logstash_ingest'
            password => "${LOGSTASH_PASSWORD}"  # read password from logstash.keystore
            ssl => true
            cacert => '/usr/share/logstash/config/certs/ca/ca.crt'
        }
}

I get an index called winlogbeat-7.4.0 created (which matches the index template so gets the mappings) but logstash should be creating winlogbeat-7.4.0-etc "with the rollover alias". My winlogbeat index templates looks the same as my filebeat one

winlogbeat template snippet

    "lifecycle": {
      "name": "winlogbeat-7.4.0",
      "rollover_alias": "winlogbeat-7.4.0"
    },

filebeat template snippet

    "lifecycle": {
      "name": "filebeat-7.4.0",
      "rollover_alias": "filebeat-7.4.0"
    },

when I look at the indexes themselves I can see the "provided_name" by logstash is incorrect and does not contain the rollover alias.

[tomrade (Thomas)]

Hey ive now tried setting the index as

index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}-000001"

to see if this works I will update when I hit my ILM limit. im guessing this might be question for logstash on if they want to be able to auto add on "-%{+YYYY.MM.dd}-000001" when manage_template is set to false

[tomrade (Thomas)]

that didnt work I get

illegal_argument_exception: index.lifecycle.rollover_alias [winlogbeat-7.4.0] does not point to index [winlogbeat-7.4.0-2019.12.03-000001]