Does single node Elasticsearch supports ILM ploicy

Ravi_Pattar · January 9, 2024, 7:03pm

Please find the details about filebeat.yml in the below link.

stephenb · January 9, 2024, 7:24pm

Ohhh you are going through logstash...

You did not mention that... do you need logstash? its adds complexity... and is why your ILM is not working

Its OK but this means you have an issue there...

Share your logstash conf file... particularly the output.

Or take out logstash and go directly to Filebeat -> Elasticsearch

Anyway... your old indices will never roll over and get deleted because they were not set up correctly because you put logstash in the middle and did not configure it correctly ...

You need to run
./filebeat setup -e
When filebeat is pointing to elasticsearch

Then change Filebeat to point to logstash and then use the correct config

AND yes the docs are wrong (I thought I had them fix it)

it should look like (notice the index name)

So you probably followed the docs bu they are not correct.

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      user => "elastic"
      password => "secret"
    }
  }
}

Ravi_Pattar · January 9, 2024, 7:38pm

cat 30-elasticsearch-output.conf

output {
  if [@metadata][pipeline] {
        elasticsearch {
        hosts => ["localhost:9200"]
        manage_template => false
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        pipeline => "%{[@metadata][pipeline]}"
        }
  } else {
        elasticsearch {
        hosts => ["localhost:9200"]
        manage_template => false
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        }
  }
}

stephenb · January 9, 2024, 7:43pm

Should be .... the following... if you want ILM to work.

Your logstash config is wrong it is meant for daily indices not ILM.

This should work going forward... your other indices will never roll over because they are "Daily Indices" and are not using the Write Alias which is

filebeat-7.17.13
hence
index => "%{[@metadata][beat]}-%{[@metadata][version]}"

Correct

output {
  if [@metadata][pipeline] {
        elasticsearch {
        hosts => ["localhost:9200"]
        manage_template => false
        index => "%{[@metadata][beat]}-%{[@metadata][version]}"
        pipeline => "%{[@metadata][pipeline]}"
        }
  } else {
        elasticsearch {
        hosts => ["localhost:9200"]
        manage_template => false
        index => "%{[@metadata][beat]}-%{[@metadata][version]}"
        }
  }
}

You will need to manually delete the old indices as they will never rollover and thus never move to the next step and never be deleted

To be direct the Guide you are following was intended to use Daily Indices not ILM... until now .. we did not know that... and that is the source of most of these issues.

You still need to run
./filebeat setup -e
At least once while filebeat is pointing to Elasticsearch BEFORE you start sending data through logstash.

See it should look like this you are missing all the write alias ....

GET /filebeat-7.17.3-2024.01.09-000001
{
  "filebeat-7.17.3-2024.01.09-000001" : {
    "aliases" : { <!--- THIS
      "filebeat-7.17.3" : { <!--- THIS
        "is_write_index" : true <!--- THIS
      }
    },

Ravi_Pattar · January 10, 2024, 4:59am

Thanks, Stephen.

I will check and update on my findings.

Ravi_Pattar · January 11, 2024, 6:14pm

Hi @stephenb

I have a query on the below and this is specific to production environment.

Does ILM supported on the basic version of ELK (7.17.15)?
Does the attached in the below URL is also having issues? Because I am seeing exceptions when I run GET /filebeat-7.17.15-2024.01.10/_ilm/explain.

The existing production setup has implementation as filebeat > Logstash > Elasticsearch
And taking out logstash to make ILM work may not be feasible at the moment.
Since the indices are piling up on daily basis result in huge disk utilization. do we have a solution on how this can sorted out? Please suggest.

Thanks,

stephenb · January 11, 2024, 6:27pm

Yes Absolutely

That is EXACTLY what I am trying to explain ... you have Setup ILM vs Daily Indices and are mixing the concept ... and this is causing the errors.... you following a 3rd party doc / config which does NOT support what you want to do...

But It can be easily fixed if you follow my instructions.

That is fine I gave you the solution above... with the output section that will fix your ILM issue for indices going forward it will NOT fix the old broken indices.

Replace your output section with what I gave you AND ILM should start working

BUT you need to run

filebeat setup -e

FIRST before you send the data through logstash

Then your ILM will work as you want.

Can you run

GET _alias/filebeat*

And show the result please

This can be fixed but you actually need to do what I am suggesting otherwise it will never work

You will need to delete them manually... which is easy to do using

DELETE indexname

WARNING that will permanently delete the indices and the data will be lost

Ravi_Pattar · January 11, 2024, 6:35pm

Thanks, Stephen for your quick response. Please find the details below.

{
  "filebeat-7.17.15-2024.01.05-000003" : {
    "aliases" : {
      "filebeat-7.17.15" : {
        "is_write_index" : true
      }
    }
  },
  "filebeat-7.17.15-2023.12.29-000002" : {
    "aliases" : {
      "filebeat-7.17.15" : {
        "is_write_index" : false
      }
    }
  }
}

Ravi_Pattar · January 11, 2024, 7:13pm

Hi Stephen

Another thing that I have prepared the procedural doc on applying the changes as per your suggestions on the test instance.

Please review the doc from the link below and please let me know if its correct so that I can start applying changes.

After running the filebeat setup -e. do I need to revert the changes in filebeat.yml again i.e. pointing filbeat -> Logstash

Thanks,

stephenb · January 11, 2024, 7:22pm

This is good it means you have a viable write alias...

Hi Ravi... quick look ... not quite right...

Steps Should be

Fix Logstash output
Point filebeat output to elasticsearch
./filebeat setup -e (your command is not correct)
Point filebeat output back to logstash
Start Filebeat

You will need to test and try... I am not really have time to review all your procedures...

Ravi_Pattar · January 11, 2024, 7:50pm

Hi Stephen

I have run the procedure and I could see as below when compared to yours in previous post.

stephenb:

See it should look like this you are missing all the write alias ....

GET /filebeat-7.17.3-2024.01.09-000001
{
  "filebeat-7.17.3-2024.01.09-000001" : {
    "aliases" : { <!--- THIS
      "filebeat-7.17.3" : { <!--- THIS
        "is_write_index" : true <!--- THIS
      }
    },

Latest from my test instance as below.

GET filebeat-7.17.13-2024.01.09-000006

{
  "filebeat-7.17.13-2024.01.09-000006" : {
    "aliases" : {
      "filebeat-7.17.13" : {
        "is_write_index" : true
      }
    },

stephenb · January 11, 2024, 8:15pm

So now if you configure logstash output as I suggested...

Logstash will write data to the write alias filebeat-7.17.13 (which points to the concrete index), and your ILM, rollover and delete should work per the policy...

In order for ILM to work... the data needs to be written to the write alias...

I would also suggest to run this.

POST filebeat-7.17.3/_rollover

That will force the write alias to to rollover the index, and the new index will follow your new ILM Policy, and you should be good to go!

Ravi_Pattar · January 11, 2024, 8:25pm

POST filebeat-7.17.13/_rollover

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "old_index" : "filebeat-7.17.13-2024.01.11-000007",
  "new_index" : "filebeat-7.17.13-2024.01.11-000008",
  "rolled_over" : true,
  "dry_run" : false,
  "conditions" : { }
}

stephenb · January 11, 2024, 8:27pm

You should be good to go... Looks like a new index what just created today anyways but that should not matter...

Start Logstash with the Config
Point Filebeat to Logstash
And you should see data flowing into filebeat-7.17.13-2024.01.11-000008

Ravi_Pattar · January 11, 2024, 8:47pm

This was done. reference output below.

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

You mean as below..

root@ELK-test:/usr/share/logstash# bin/logstash -f /etc/logstash/conf.d/30-elasticsearch-output.conf

But it give some errors. file attached in the below URL.

stephenb · January 11, 2024, 8:50pm

You need to start logstash the normal way... probably

systemctl start logstash

I feel like I am helping in the dark.....

I am not following your guide (you have not shared it... but also we recommend following our documentation) so I have no clue how you installed logstash and what you did with logstash etc.etc...

If you have questions on starting / stopping running logstash etc... you should open a separate thread on that...

Ravi_Pattar · January 11, 2024, 8:57pm

yeah, I have restarted the LS service and now its up and running.

# systemctl status logstash
● logstash.service - logstash
     Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-01-11 20:52:15 UTC; 2s ago
   Main PID: 5293 (java)
      Tasks: 14 (limit: 4667)

I have followed the guide from the sources on the installation of ELK stack.

would request you to share your documentation which will be of a great help.

Thanks,

stephenb · January 11, 2024, 9:05pm

So that "HowTo" looks like it was pretty good... but it is ~3.5+ years old... Elastic moves fast so that is an issue... and why the new ILM did not work with the old config etc..

There are many many resources ...

https://www.elastic.co/learn

I am sure you can look around you are doing fine... just keep working and asking questions

Good Luck!

Ravi_Pattar · January 11, 2024, 9:14pm

Thanks, Stephen for your guidance and support on this thread.

I will keep posted on my observations on post changes today.

Ravi_Pattar · January 12, 2024, 4:59pm

stephenb:

See it should look like this you are missing all the write alias ....

GET /filebeat-7.17.3-2024.01.09-000001
{
  "filebeat-7.17.3-2024.01.09-000001" : {
    "aliases" : { <!--- THIS
      "filebeat-7.17.3" : { <!--- THIS
        "is_write_index" : true <!--- THIS
      }
    },

Hi Stephen,

ILM is working as expected in the test environment and thanks for your guidance on the same.

I am planning to implement the same in the production system today. Currently I came across with the below observations.

GET /filebeat-8.10.4-2024.01.12

{
  "filebeat-8.10.4-2024.01.12" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },

GET /filebeat-8.11.2-2024.01.12

{
  "filebeat-8.11.2-2024.01.12" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },

Also w.r.t 7.7.15 for which I am planning to run the steps for ILM the GET request shows as below. And I can handle this case as I have did this in test bed yesterday only.

GET /<filebeat-7.17.15>

{
  "filebeat-7.17.15-2024.01.05-000003" : {
    "aliases" : {
      "filebeat-7.17.15" : {
        "is_write_index" : false
      }

so, in these conditions mentioned above for index 8.10 and 8.11. How to handle when it comes to rollover e.g. POST filebeat-7.17.3/_rollover

Thanks,