Examples for setting up Beats (modules) > Logstash > ES with ILM?

Elastic Stack Logstash
1

Are there any basic configuration examples showing how to set up a Beats (with modules and dashboards) > Logstash > ES pipeline using Index Lifecycle Management? Currently I'm familiar with the "classic" setup with Metricbeat and Filebeat pushing logs to logstash. Logstash is configured to create daily indices with %{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd} for the name.

To set this up to use ILM, I need to run the beats setup commands with ilm enabled to create the templates, which creates the policy and index template (something like filebeat-7.0.1-2019-05-06-000001) That is the "real" index, but logstash should just be configured to write to the alias filebeat-7.0.1?

Or is logstash supposed to be configured to use ILM and I need to set up the template and alias myself? Should it still separate indices per beat or just use one logstash index? Would that break the built-in dashboards? If anyone has a basic working example for a setup like this I would really appreciate it!

1 Like
2

I have the same question. I have everything working but ILM is having issues. image

3

Has anyone figured this out yet? I'm facing the same issue.

4

Do you have logstash configured for daily indexes? Something like index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"?

When you set up ILM it creates an alias like filebeat-7.0.1 and ILM handles rolling over and creating real indexes. So if you configure logstash to just push to the alias index => "%{[@metadata][beat]}-%{[@metadata][version]}" it should get rid of those errors.

The problem I ran in to was it all seems to work as long as you never touch it again. If you ever delete indexes manually, I still havent figured out how to configure it again to restore it to a working state.

5

yup I have logstash creating daily indexes. Alright so I should make that change. Now I am wondering if this will be broken for me now though because I just reindexed and deleted an index.

6

Now it's creating the index but no ILM policy or alias. Let me start from scratch again.

7

I started from scratch completely and es is not creating ilm at all using index => "%{[@metadata][beat]}-%{[@metadata][version]}"

8

i think ILM configuration is created when you run the beats setup commands like filebeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'.

I would stop the logstash service before cleaning up indexes so it doesnt create a new one before you have a chance to run setup

9

Thanks for the help Pete. This is driving me nuts. The documentation states this should all be automagic but it's just not quite meshing together correctly. Luckily I have the time to invest right now.

10

That seems to be working, thank you. The initial index is still created with a date stamp in the name, but it appears to be working.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.