ELK 7.12 - ILM Rollover - After rollover data keeps being writing old index 000001

Elastic Stack Kibana
1

I am using logstash to centralize all logs from beats then send them to Elastisearch. I create all ILM policies, index templates, bootstrap each index, and everything goes perfect until rollover, the second index packetbeat-7.12.0-000002 is created and set to is_write_index = true and the old index packetbeat-7.12.0-000001 is set to is_write_index = false automatically, but the data keeps writing on the old index packetbeat-7.12.0-000001.

No error logs - nothing! It is happening for all beats and other plugins like Elastiflow that has its own Netflow collector and does not pass through Logstash.

image
image958×167 3.89 KB

GET _ilm/policy/packetbeat_policy

{
  "packetbeat_policy" : {
    "version" : 7,
    "modified_date" : "2021-04-27T16:22:04.629Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "30mb", (just to force rollover)
              "max_age" : "2d"
            }
          }

GET _template/packetbeat_template

{
  "packetbeat_template" : {
    "order" : 1,
    "index_patterns" : [
      "packetbeat-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "packetbeat_policy",
          "rollover_alias" : "packetbeat"
        },
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },

GET packetbeat-7.12.0-000001

{
  "packetbeat-7.12.0-000001" : {
    "aliases" : {
      "packetbeat" : {
        "is_write_index" : false
      }
    },
    "mappings" : {
      "_meta" : {
        "beat" : "packetbeat",
        "version" : "7.12.0"
      },

GET packetbeat-7.12.0-000002

{
  "packetbeat-7.12.0-000002" : {
    "aliases" : {
      "packetbeat" : {
        "is_write_index" : true
      }
    },
    "mappings" : {
      "_meta" : {
        "beat" : "packetbeat",
        "version" : "7.12.0"
      },

Packetbeat 7.12

image
image953×123 2.82 KB

Winlogbet 7.11

image
image959×118 2.77 KB

2

Hello Alberto,

Can you please also share the logstash elasticsearch output config?

So far I have not seen anything unusual...

Best regards
Wolfram

Update:
Could you please try the explain ILM API and post the output for both indices?

3

Hello @Wolfram_Haussig

/etc/logstash/conf.d/beats.conf

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["https://my_elastic_server:9200"] 
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-000001"
    user => "user"
    password => "password"
    ssl => true
    cacert => "/etc/logstash/certs/elasticsearch-ca.pem"
  }
}

I also tried with version and without version the index name, the result is the same.

index => "%{[@metadata][beat]}-%{[@metadata][version]}-000001" == filebeat-7.12.0-000001
index => "%{[@metadata][beat]}-000001" == packetbeat-000001

GET packetbeat-7.12.0-*/_ilm/explain

{
  "indices" : {
    "packetbeat-7.12.0-000001" : {
      "index" : "packetbeat-7.12.0-000001",
      "managed" : true,
      "policy" : "packetbeat_policy",
      "lifecycle_date_millis" : 1619541215086,
      "age" : "3.28h",
      "phase" : "hot",
      "phase_time_millis" : 1619539628328,
      "action" : "complete",
      "action_time_millis" : 1619541215593,
      "step" : "complete",
      "step_time_millis" : 1619541215593,
      "phase_execution" : {
        "policy" : "packetbeat_policy",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "30mb",
              "max_age" : "2d"
            }
          }
        },
        "version" : 7,
        "modified_date_in_millis" : 1619540524629
      }
    },
    "packetbeat-7.12.0-000002" : {
      "index" : "packetbeat-7.12.0-000002",
      "managed" : true,
      "policy" : "packetbeat_policy",
      "lifecycle_date_millis" : 1619541215222,
      "age" : "3.28h",
      "phase" : "hot",
      "phase_time_millis" : 1619541215474,
      "action" : "rollover",
      "action_time_millis" : 1619541215572,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1619541215572,
      "phase_execution" : {
        "policy" : "packetbeat_policy",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "30mb",
              "max_age" : "2d"
            }
          }
        },
        "version" : 7,
        "modified_date_in_millis" : 1619540524629
      }
    }
  }
}

As you can see, packetbeat-7.12.0-000001 is complete and packetbeat-7.12.0-000002 rollover.

4

if you want to pass through
beats->logstash->es
and have all the templates, ILM, pipelines etc work your logstash conf should look like this.
Configure the correct Elasticsearch endpoint and creads.
Do not change the other settings, ILM and Pipelines etc

################################################
# beats->logstash->es default config.
################################################
input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      user => "elastic"
      password => "secret"
    }
  }
}

The general process for this is :slight_smile: if you want to use all the beats functionality but pass through logstash.

  1. You may need to clean up.
  2. Configure your beats / modules how you want.
  3. Point the beats output to elasticsearch and run
    packetbeat setup -e
  4. Then go back into packetbeat.yml and configure the output to logstash, comment out the elasticsearch output.
  5. Start logstash with the conf above
  6. Start packetbeat

When you name the index with the -00001 etc it will never write to the new index etc.

5

Hello @stephenb

I followed what you said and it worked! So for the beats, everything is working pretty well using the default templates (XXXXbeat setup -e). Could you explain the output parameter pipeline => "%{[@metadata][pipeline]}"?

image
image1185×310 7.16 KB

I also have other solutions/plugins that I need to do the same thing (use ILM). I was able to do it for my Syslog events as well but in a different way. As you mentioned, if I set the index name to 00001 at the end, even after the rollover it will keep writing data at 00001 index.

Analyzing the configuration samples and logs during the startup process of the beats I have noticed I line with ilm_pattern => "{now/d}-000001" after searched about it I found good documentation related to it. maybe it is not the correct way to use it, but it worked.

input {
   syslog {
     port => 514
   }
}

output {
  elasticsearch {
    hosts => ["https://my-elk-server:9200"]
    index => "syslog-%{+yyyy.MM.dd}"    
    ilm_rollover_alias => "syslog"
    ilm_pattern => "{now/d}-000001"
    ilm_policy => "syslog_policy"
    user => "elastic"
    password => "secret"
    ssl => true
    cacert => "/etc/logstash/certs/elasticsearch-ca.pem"
  }
}

So, I have to do it with Elastiflow, but I will check with the community of Elastiflow if it supports ILM or another way to rotate the logs (retention policy).

6

Sure some/many modules include an ingest pipeline that parses the data on the elasticsearch side. The name of that pipeline is is stored in that field. If you just use a beat that has pipeline as part of the module that is automatically defined.

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.