ILM failing for rollover for Wazuh module / data

Atul_Chadha · December 14, 2021, 11:10am

I have setup a rollover policy for wazuh index with the below policy defination

{
  "policy": "Wazuh",
  "phase_definition": {
    "min_age": "0ms",
    "actions": {
      "rollover": {
        "max_size": "20gb",
        "max_age": "1d"
      }
    }
  },
  "version": 4,
  "modified_date_in_millis": 1639189543104
}

Per my understanding there are 3 basic requirements as mentioned in below link

I have create a IML policy as mentioned below

	PUT _ilm/policy/Wazuh
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "20gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "actions": {}
      },
      "delete": {
        "min_age": "365d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

Created a template to associate with index pattern

{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "Wazuh",
          "rollover_alias": "wazuh-alerts"
        },
        "number_of_shards": "3",
        "number_of_replicas": "1"
      }
    },
    "aliases": {},
    "mappings": {}
  }
}

Created a inital write index with name " wazuh-alerts-000001"

I am seeing index getting as large as 90G for single day and the below error in rollover

java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [wazuh-alerts] does not point to index [wazuh-alerts-4.x-2021.12.13]
	at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:114)
	at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:174)
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:327)
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:265)
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183)
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:216)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)

What am i missing ?

Atul_Chadha · December 15, 2021, 2:28am

Checking if anyone has any ideas?

stephenb · December 15, 2021, 3:44am

Perhaps You forgot to create the initial managed index correctly with the write alias

You should have done this...

PUT wazuh-alerts-4.x-2021.12.13
{
  "aliases": {
    "wazuh-alerts":{
      "is_write_index": true 
    }
  }
}

Then what you are using to index should point to the write alias wazuh-alerts instead looks like you are writing directly to the concrete index wazuh-alerts-4.x-2021.12.13

Atul_Chadha · December 15, 2021, 7:46am

Thanks @stephenb for the response. I tried the suggested approach however didnt had much luck.

Here are some other changes i tried

Filebeat.yml

setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/wazuh-template.json'
setup.template.json.name: 'wazuh'
#setup.ilm.overwrite: true
setup.ilm.rollover_alias: "wazuh-alerts-4.x"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.enabled: true
setup.ilm.policy_name: 'Wazuh'
setup.ilm.policy_file: '/etc/filebeat/policy.json'

I see a write index was created when i restarted filebeat on wazuh manager

I see an alias on the write index however none on the other index created for the day

Do we need the index to be in format wazuh-alerts-4.x-2021.12.15-000001 where 000001 will rollover to 000002 and so on for today date ?

If so, how can i force creation of the index with such naming convention ?

Atul_Chadha · December 15, 2021, 1:45pm

I have tried creating a write index again with name "wazuh-alerts-4.x-2021.12.13-000001" and on rollover i see its trying to rollover the write index itself rather than my main index in question wazuh-alerts-4.x-2021.12.15.

Seems like some misconfiguration to me.

stephenb · December 15, 2021, 2:41pm

Can you show the output section in your filebeat.yml.

The index you're writing to should be wazuh-alerts-4.x the alias name.

Also that should be the alias in your first managed index at all has to line up.

PUT wazuh-alerts-4.x-2021.12.13-000001
{
  "aliases": {
    "wazuh-alerts-4.x":{
      "is_write_index": true 
    }
  }
}

It all has to line up.

Also I just noticed this.
Get rid of all this.. You're putting the template in by hand in the dev tools I don't know what's in all these but you're probably overriding and that's causing those weird templates.

Atul_Chadha · December 15, 2021, 3:38pm

@stephenb Thank you for the update, i tried to update the alias and recreated the index to match todays date.

I am getting a different error this time " "reason" : "index name [wazuh-alerts-4.x-2021.12.15] does not match pattern '^.*-\\d+$'"

I see similar issue raised in the below post.

Do we need the increment number after the index to rollover ? I am fine to clear out the settings and start fresh if that helps.

Atul_Chadha · December 15, 2021, 3:41pm

My current filebeat.yml is

output.elasticsearch:
  hosts: [00000]
  protocol: http
  username: "00000"
  password: "00000"
  ssl.certificate_authorities:
    - /etc/filebeat/certs/root-ca.pem
  ssl.certificate: "/etc/filebeat/certs/filebeat.pem"
  ssl.key: "/etc/filebeat/certs/filebeat-key.pem"
setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/wazuh-template.json'
setup.template.json.name: 'wazuh'
setup.ilm.overwrite: true
setup.ilm.enabled: false
setup.ilm.rollover_alias: "wazuh-alerts-4.x"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.enabled: true
setup.ilm.policy_name: 'Wazuh'
setup.ilm.policy_file: '/etc/filebeat/policy.json'
filebeat.modules:
  - module: wazuh
    alerts:
      enabled: true
    archives:
      enabled: false

stephenb · December 15, 2021, 3:55pm

I fixed above... for initial managed

PUT wazuh-alerts-4.x-2021.12.13-000001
{
  "aliases": {
    "wazuh-alerts-4.x":{
      "is_write_index": true 
    }
  }
}

Think you need...

output.elasticsearch:
  hosts: [00000]
  index: wazuh-alerts-4.x

Not sure I quite understand ... No you should not increment that should happen automatically.
If you are saying the you need to go to ....000002 then you need to set the old index to "is_write_index": false and create a new managed index.

Not sure why you are doing all this...

setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/wazuh-template.json'
setup.template.json.name: 'wazuh'

setup.ilm.policy_name: 'Wazuh'
setup.ilm.policy_file: '/etc/filebeat/policy.json'

Once you set it via Dev Tools or UI.. you don't need this.

I would set.

setup.ilm.overwrite: false

Also ... I would think it would be

setup.ilm.enabled: false

If the policy is set in the template you don't need to set it here.

Seems like you are setting all sorts of settings.....

BTW Also this is why we really have people just start with the defaults (filebeat index) get used to it all .. get it working with the defaults and THEN change to custom index names etc.. etc..

If you are following some other tutorial...perhaps take another read...

Atul_Chadha · December 16, 2021, 4:27pm

I think there may be a bug in what we are trying to achieve ([Docs] Note that output.elasticsearch.index setting is ignored with ILM enabled · Issue #11866 · elastic/beats · GitHub)

@stephenb Appreciate your efforts and expertise. Will try to see what other option i can try.

stephenb · December 16, 2021, 4:34pm

@Atul_Chadha

Perhaps but I don't think so ... that is against and old version.

I configure custom indexes with ILM with filebeat every day...

The Problem is ... It is hard from me to follow everything you are doing (or not doing)... we just set up 8 custom indices with ILM yesterday for a customer using filebeat it is running today and rolling over just fine.

The basic Steps:

Create and ILM Policy
Create a Template with the index patterns, the ILM policy and the rollover alias.
Create the initial managed index
Put the minimal required config in filebeat (This is where people struggle)
Run Filebeat.

AND you can always just use the defaults and get it running

Example here is a complete filebeat where we setup the ILM manually notice how little is there

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.inputs:
- type: aws-s3
  enabled: true
  queue_url: 
  tags: [forwarded, custom-logs]
  index: "custom-logs-writer-alias" <!---- KEY as we set it up all manually with the template, ILM etc 
  api_timeout: 6m
  visibility_timeout: 12m
  pipeline: custom-logs

setup.ilm.enabled: false <!---- KEY as we set it up all manually with the template, ILM etc 

output.elasticsearch:
  hosts: ["http://localhost:9200"]

Atul_Chadha · December 16, 2021, 5:00pm

I think part of problem is i am forced to use wazuh template rather than stock filebeat. Will try to investigate more and if i find something will post it here

stephenb · December 16, 2021, 5:03pm

The above I just posted used a custom template... you are close... don't give up... you are over configuring filebeat and it is conflicting with what you setup with the Template, ILM, and Initial Managed Index with the Writer Alias

Create and ILM policy
Just load the Wazuh Template... and add that policy to it.
Create the initial managed index
Then use minimal filebeat like above.

the
setup.ilm.enabled: false

Seems counter intuitive.. what this says is filebeat don't manage the ILM I have set it up manually

Atul_Chadha · December 16, 2021, 5:15pm

Sure, Let me try again with fresh eyes tomorrow , has been 4 days struggling with this now

stephenb · December 16, 2021, 5:33pm

Sounds like a good idea...

problem with the other tutorials they often age out... or are not the most simple way...
You will get it!

Look carefully at that minimal... that is all you should need really need ... 2 settings

setup.ilm.enabled: false
and set the index to the write alias (Elasticsearch side will manage the rest) either on the inputs section or output.elasticsearch section

Atul_Chadha · December 16, 2021, 6:08pm

I have recreated a ILM policy and index template.

Here is the current filebeat.yml

output.elasticsearch:
  hosts: ["XXXXXXXXXXXXXXX"]
  protocol: http
  username: "elastic"
  password: "DDDDDDDD"
  index: 'wazuh-alerts-4.x-%{+dd.MM.yyyy}'
  ssl.certificate_authorities:
    - /etc/filebeat/certs/root-ca.pem
  ssl.certificate: "/etc/filebeat/certs/filebeat.pem"
  ssl.key: "/etc/filebeat/certs/filebeat-key.pem"
setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/wazuh-template.json'
setup.template.json.name: 'wazuh'
setup.ilm.enabled: false
filebeat.modules:
  - module: wazuh
    alerts:
      enabled: true
    archives:
      enabled: false
logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

Created a inital index

curl -X PUT "localhost:9200/wazuh-alerts-4.x-000001?pretty" -H 'Content-Type: application/json' -d'
{
  "aliases": {
    "wazuh-alerts-4.x": {
      "is_write_index": true
    }
  }
}
'

I am getting the below error now in filebeat

2021-12-16T12:59:56.801-0500    INFO    instance/beat.go:645    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2021-12-16T12:59:56.802-0500    DEBUG   [beat]  instance/beat.go:697    Beat metadata path: /var/lib/filebeat/meta.json
2021-12-16T12:59:56.802-0500    INFO    instance/beat.go:653    Beat ID: 9e4203d0-9719-48b9-8970-d4f876665e6f
2021-12-16T12:59:56.802-0500    INFO    instance/beat.go:373    filebeat stopped.
2021-12-16T12:59:56.806-0500    ERROR   instance/beat.go:956    Exiting: setup.template.name and setup.template.pattern have to be set if index name is modified

stephenb · December 16, 2021, 6:12pm

We are close...

Clean up that other index / write alias

I moved the index under the module... this does an override...

Try this... give me the errors

curl -X PUT "localhost:9200/wazuh-alerts-4.x-2021.12.16-000001?pretty" -H 'Content-Type: application/json' -d'
{
  "aliases": {
    "wazuh-alerts-4.x": {
      "is_write_index": true
    }
  }
}

output.elasticsearch:
  hosts: ["XXXXXXXXXXXXXXX"]
  protocol: http
  username: "elastic"
  password: "DDDDDDDD"
  ssl.certificate_authorities:
    - /etc/filebeat/certs/root-ca.pem
  ssl.certificate: "/etc/filebeat/certs/filebeat.pem"
  ssl.key: "/etc/filebeat/certs/filebeat-key.pem"
setup.ilm.enabled: false
filebeat.modules:
  - module: wazuh
    alerts:
      enabled: true
      input.index: 'wazuh-alerts-4.x'
    archives:
      enabled: false

logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

Atul_Chadha · December 16, 2021, 6:33pm

We have progress, i see index rolled over. The index "wazuh-alerts-4.x-2021.12.16" seems to be exceeding the policy , is this expected ?

GET /wazuh-alerts-4.x-2021.12.16/_ilm/explain
{
  "indices" : {
    "wazuh-alerts-4.x-2021.12.16" : {
      "index" : "wazuh-alerts-4.x-2021.12.16",
      "managed" : true,
      "policy" : "Wazuh",
      "lifecycle_date_millis" : 1639678706913,
      "age" : "10.08m",
      "phase" : "hot",
      "phase_time_millis" : 1639678709566,
      "action" : "rollover",
      "action_time_millis" : 1639679304073,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1639679304073,
      "phase_execution" : {
        "policy" : "Wazuh",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "20mb",
              "max_age" : "1m"
            }
          }
        },
        "version" : 27,
        "modified_date_in_millis" : 1639676567202
      }
    }
  }
}

stephenb · December 16, 2021, 6:38pm

Excellent almost there...

Yes tiny policies are not going to work the way you think...
1m / 20 mb is not really valid it will roll over but tiny fast / will not be exact.. it will rollover when it gets to it... background task. That will work right at scale of GBs / Day

BUT it looks like you did not create the initial managed index correct it should be pattern

wazuh-alerts-4.x-yyyy.mm.dd-000001

Stop filebeat
Delete all the indices
Go to Kibana Dev Tools and do the following

DELETE waz*

PUT wazuh-alerts-4.x-2021.12.16-000001
{
  "aliases": {
    "wazuh-alerts-4.x": {
      "is_write_index": true
    }
  }
}

Try again

Atul_Chadha · December 16, 2021, 6:48pm

Stopped filebeat and wazuh manager
Deleted all wazuh indices
Updated the ILM to rollover 20G / 1day
Will check the result tomorrow and update you

Appreciate your time @stephenb